feat: add gateway discriminator to exposed service hostnames for multi-gateway clusters #4840

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Oct 22, 2026
2. jhjaggars commented on Oct 22, 2026

   

   jhjaggars

   on Oct 22, 2026

   Last edited by jhjaggars

   ContributorAuthor

   More actions

   🏗️ build-plan

   Implementation Plan

   Issue type: feat
   Complexity: High
   Confidence: Medium-high

   Summary

   Add an opt-in service_routing_gateway_name setting that changes newly generated service URL labels from readable <workspace>--<sandbox>[--<service>] labels to a stable, one-label, 16-character lowercase base32 SHA-256 prefix. Preserve legacy URLs and parsing when unset, retain legacy-host routing during transition when configured, and add an in-memory hash-to-route index hydrated from persisted ServiceEndpoint records.

   Scope

   • crates/openshell-core/src/config.rs → add ServiceRoutingConfig.service_routing_gateway_name: Option<String>, an infallible builder, DNS-label validation, defaults, and core tests.
   • crates/openshell-server/src/service_routing.rs → implement SHA-256/NUL-delimited tuple hashing and fixed 10-byte lowercase RFC-4648 base32 encoding; generate hashed hosts; add hash-route index/rebuild/register/remove/resolve logic; retain legacy parsing; make hash resolution state/store-aware; extend routing tests.
   • crates/openshell-server/src/lib.rs → add the shared service-route index to ServerState, initialize it with the configured routing name, and hydrate it from persisted ServiceEndpoint records before accepting traffic.
   • crates/openshell-server/src/http.rs → update HTTP middleware to await state-aware service-host resolution so configured hash hosts route and unknown hashes fall through to normal routing/404 behavior.
   • crates/openshell-server/src/grpc/service.rs → reserve/check the route hash before persistence, update the index after successful expose/update, remove it after successful service deletion, and return a clear collision status without persisting the conflicting endpoint.
   • crates/openshell-server/src/compute/mod.rs → share the route index with ComputeRuntime and remove entries when sandbox cleanup deletes its owned ServiceEndpoint records.
   • crates/openshell-server/src/config_file.rs → add optional [openshell.gateway] service_routing_gateway_name TOML deserialization/serialization coverage.
   • crates/openshell-server/src/cli.rs → add --service-routing-gateway-name / OPENSHELL_SERVICE_ROUTING_GATEWAY_NAME, merge the TOML value only when CLI/env did not supply it, validate the effective value, and wire it into Config.
   • crates/openshell-cli/src/run.rs → add URL-normalization tests proving CLI output preserves the opaque hashed hostname while replacing only the externally reachable gateway port.
   • deploy/helm/openshell/values.yaml → add documented server.serviceRoutingGatewayName: ""; empty remains backward-compatible and does not emit TOML.
   • deploy/helm/openshell/templates/_toml.tpl → map a non-empty legacy chart value to openshell.gateway.service_routing_gateway_name, while preserving explicit gatewayConfig.openshell.gateway.service_routing_gateway_name precedence and omitting the field when absent/empty.
   • deploy/helm/openshell/tests/gateway_config_test.yaml → verify default omission, non-empty rendering, and explicit schema-v2 gatewayConfig precedence.
   • docs/how-it-works/gateways/configuration.mdx → document the TOML field, validation, default, CLI/env precedence, examples, gateway-name uniqueness guidance, and that it is unrelated to gateway_jwt.gateway_id.
   • docs/how-it-works/gateways/overview.mdx → correct the existing hostname grammar to include the mandatory workspace segment and describe legacy versus configured hashed service labels.
   • docs/how-it-works/sandboxes/overview.mdx → update the service-exposure section to explain that callers should consume printed URLs rather than construct hostnames.
   • docs/kubernetes/ingress.mdx → add operator-managed exact-host HTTPRoute/Ingress guidance for emitted hash labels, wildcard-certificate requirements, and using distinct routing gateway names when multiple gateways share a base domain.
   • docs/kubernetes/setup.mdx → add server.serviceRoutingGatewayName to the chart-values reference.

   Implementation Steps

   1. Add the configuration contract in openshell-core.

      • Extend ServiceRoutingConfig (currently crates/openshell-core/src/config.rs:311-319) with service_routing_gateway_name: Option<String>, defaulting to None.
      • Add Config::with_service_routing_gateway_name(...), keeping with_server_sans limited to deriving base_domains so it preserves this new setting.
      • Add ServiceRoutingConfig::validate() (or an equivalent public validation helper used by the server) that accepts None and otherwise requires an RFC/DNS-1123-label-shaped value: 1–63 ASCII bytes, lowercase a-z, 0-9, or -, and no leading/trailing -. Reuse the existing private is_dns_label behavior at config.rs:1173-1180.
      • Reject empty strings rather than treating them as enabled; the absence of the setting is represented only by None. This keeps the optional field unambiguous and produces ingress-rule-friendly operator identifiers.

   2. Plumb the value through TOML, CLI, environment, and server startup.

      • Add GatewayFileSection.service_routing_gateway_name: Option<String> beside server_sans and enable_loopback_service_http (config_file.rs:152-162).
      • Add RunArgs.service_routing_gateway_name: Option<String> with Clap long = "service-routing-gateway-name" and env = "OPENSHELL_SERVICE_ROUTING_GATEWAY_NAME", adjacent to server_sans (cli.rs:276-301).
      • In merge_file_into_args, copy the TOML field only if args.service_routing_gateway_name.is_none() and arg_defaulted(matches, "service_routing_gateway_name"), matching the existing source-precedence convention: CLI flag > env > TOML > built-in default.
      • In prepare_server_config_with_drivers, apply the setting after with_server_sans/loopback routing setup, then validate the effective config.service_routing before startup continues. This makes invalid TOML fail both preflight and startup; Clap can additionally use the same core validation helper for CLI/env input.
      • Do not derive this setting from Config.name or GatewayJwtConfig.gateway_id; it stays unset unless explicitly configured.

   3. Implement deterministic hash labels without adding a dependency.

      • In service_routing.rs, add a pure helper such as service_route_hash_label(gateway_name, workspace, sandbox, service) -> String.
      • Serialize the exact tuple as gateway_name, workspace, sandbox, service joined by literal NUL bytes (gateway\0workspace\0sandbox\0service), with "" for unnamed services; SHA-256 the bytes; take digest[..10]; encode those 80 bits with the lowercase RFC-4648 alphabet abcdefghijklmnopqrstuvwxyz234567, no padding. Ten bytes encode to exactly 16 characters.
      • Use the workspace sha2 dependency already present in openshell-server; no base32 encoder exists in workspace manifests or Cargo.lock. Hand-roll the fixed-size encoder using a small constant alphabet and bit accumulator rather than adding a new crate for one 10-byte encoding operation.
      • Update endpoint_host so Some(service_routing_gateway_name) emits <hash>.<first-base-domain> and None preserves the current byte-for-byte readable format. endpoint_url remains dynamically generated and requires no proto or persistence change.
      • Retain MAX_ROUTABLE_NAME_LEN at grpc/mod.rs:143-148: legacy labels remain routable during the transition, so the legacy 63-character DNS-label constraint still applies even though hash labels themselves do not need it.

   4. Introduce and hydrate the hash-to-endpoint index.

      • Add a ServiceRouteIndex in service_routing.rs, owned by ServerState as an Arc around a lock-protected HashMap<String, ServiceRouteTarget>. The value must include workspace, sandbox, and service; it is the input used by the existing load_endpoint path, not a persisted copy of ServiceEndpoint.
      • Add index operations to:
        • calculate/register an endpoint’s label and reject a different existing target under the same label;
        • remove only when the stored target matches the deleting endpoint;
        • rebuild from Store::collect_messages::<ServiceEndpoint>(ObjectListQuery::AllWorkspaces);
        • clear/no-op when service_routing_gateway_name is unset.
      • Initialize the index in ServerState::new_with_credentials and call its asynchronous rebuild after ensure_default_workspace but before listener binding in lib.rs (currently between lib.rs:827-850). Treat a persisted ambiguous hash as startup configuration/state failure rather than picking an arbitrary route.
      • The current server has no existing endpoint-routing map: ServerState only has service_upstreams at lib.rs:312-314, while endpoint resolution is currently a direct persistence lookup in service_routing.rs:606-621. The new index must therefore be hydrated at startup; only updating it on future exposes would break all existing endpoints after a gateway restart.

   5. Resolve configured hash hosts while retaining legacy host compatibility.

      • Preserve a legacy parsing path for <workspace>--<sandbox>[--<service>].<base-domain>, including configured gateways. Legacy hosts remain syntactically distinguishable because hash labels are exactly 16 lowercase base32 characters and readable labels contain --.
      • Refactor parse_host/classification into a syntactic host parser plus an asynchronous state-aware resolver. Hash lookup needs the route index and Store; the current parse_host(host, &ServiceRoutingConfig) at service_routing.rs:203-232 cannot resolve an opaque label by configuration alone.
      • For configured hash candidates, first consult the index; on a miss, refresh it from the shared store and retry. This is required for multi-replica Postgres deployments, where the replica receiving an HTTP request may not be the replica that handled ExposeService. If the hash remains absent, return None, allowing the normal HTTP router to handle it and ultimately return its ordinary 404 rather than treating it as a service endpoint.
      • Make is_sandbox_service_request, proxy_sandbox_service_request, and emit_cross_origin_service_http_rejection use this resolver. Update http.rs middleware (http.rs:202-231) to await classification/resolution. Keep request_host behavior unchanged so both Host and HTTP/2 URI authority work.

   6. Keep the index correct across endpoint lifecycle paths and enforce collisions.

      • In grpc::service::expose_service_endpoint (grpc/service.rs:86-185), calculate/reserve the route before put_if. If the same hash maps to a different (workspace, sandbox, service), return Status::already_exists with a clear "service routing hostname hash collision" message and do not write the new record.
      • On persistence failure, roll back the reservation only if it still belongs to the attempted target. On success, commit/update the target and generate the response URL through the existing endpoint_url.
      • In handle_delete_service (grpc/service.rs:295-349), remove the matching index entry only after delete_if reports success.
      • ComputeRuntime::cleanup_sandbox_service_endpoints (compute/mod.rs:5675-5703) also directly deletes service records during sandbox cleanup. Give ComputeRuntime the same shared index (using an initialization method/OnceLock pattern like its existing shared registries) and remove each successfully deleted endpoint entry there. This prevents stale labels after sandbox deletion.
      • Because endpoint data is shared in a database but the index is process-local, retain the resolver’s refresh-on-hash-miss behavior. The local register/remove operations provide immediate consistency on the mutating replica; a store refresh converges other replicas without adding a persistence schema.

   7. Add Helm plumbing without chart-managed ingress objects.

      • Add server.serviceRoutingGatewayName: "" and comments explaining that it is optional, must be unique among gateways sharing a service base domain, and causes opaque service labels only when non-empty.
      • Update templates/_toml.tpl, not the thin templates/gateway-config.yaml wrapper, because gateway-config.yaml merely includes openshell.gatewayConfigToml at line 22 and all legacy-value translation happens in _toml.tpl:100-119.
      • Only set service_routing_gateway_name when the chart value is non-empty and no explicit gatewayConfig.openshell.gateway.service_routing_gateway_name value exists. Do not add ingress, HTTPRoute, or wildcard routing automation to the chart.

   8. Update user-facing documentation and run focused verification.

      • Document the new hostname grammar, hash input and stability properties, migration behavior, exact-host ingress requirements, and the operational consequence that changing the configured routing name changes hash URLs.
      • Correct the existing readable-format documentation to include the workspace segment everywhere it presents the grammar.
      • Run formatter and targeted Rust/Helm/doc checks after the corresponding tests below pass.

   Test Plan

   Unit

   • crates/openshell-core/src/config.rs

     • Add service_routing_gateway_name_defaults_to_none.
     • Add service_routing_gateway_name_accepts_dns_label.
     • Add service_routing_gateway_name_rejects_empty_uppercase_invalid_hyphen_and_overlong_values.
     • Confirm with_server_sans does not overwrite the configured routing name.

   • crates/openshell-server/src/service_routing.rs

     • Add a fixed-vector test for gateway-a\0default\0my-sandbox\0web, asserting SHA-256 prefix label enh47uqyj7bpyttz; assert unnamed service uses the empty fourth component and produces eiud2z5apv3yeucg.
     • Add hash-host URL tests for named and unnamed services, asserting exactly one 16-character [a-z2-7]{16} label before the base domain.
     • Preserve existing legacy URL tests unchanged for unset configuration.
     • Add configured-hash parsing tests for Host and HTTP/2 authority forms, backed by a populated ServiceRouteIndex.
     • Add legacy_host_is_accepted_when_hash_routing_is_configured.
     • Add unknown_hash_host_is_not_classified_as_service_request and assert it resolves to None.
     • Add index rebuild tests using test_store() and persisted endpoints, including restart-style rehydration.
     • Add collision coverage by directly seeding the route-index map with a different ServiceRouteTarget under the attempted endpoint’s generated label, then assert registration is rejected. Do not weaken/truncate production hashes just to make a probabilistic collision test feasible.
     • Add remove-after-delete and remove-after-sandbox-cleanup coverage.

   • crates/openshell-server/src/grpc/service.rs

     • Add expose_service_rejects_route_hash_collision_before_persistence.
     • Extend endpoint lifecycle coverage to assert an exposed configured gateway returns the hash URL, GetService/ListServices regenerate the same URL, and successful deletion removes the lookup.
     • Exercise the same behavior through create-time service_exposures, since grpc/sandbox.rs:729-771 delegates to expose_service_endpoint.

   • crates/openshell-server/src/http.rs

     • Add/extend middleware coverage proving a configured known hash is routed as a service request while a syntactically valid but unknown hash proceeds through normal non-service handling.

   • crates/openshell-server/src/config_file.rs

     • Parse and serialize service_routing_gateway_name = "gateway-west".

   • crates/openshell-server/src/cli.rs

     • Add file_populates_service_routing_gateway_name.
     • Add service_routing_gateway_name_cli_and_env_override_file_value, using the existing parse_with_args, Environment, and arg_defaulted convention.
     • Add preparation coverage that asserts the effective value reaches prepared.config.service_routing and invalid TOML fails preparation/preflight.

   • crates/openshell-cli/src/run.rs

     • Add service_url_for_gateway_preserves_hashed_service_label for a URL such as https://enh47uqyj7bpyttz.apps.example.com:8080/, proving only the port is normalized.

   Integration

   • Start an in-memory ServerState with service_routing_gateway_name = Some("gateway-a"), expose an endpoint, then resolve a request by its emitted hash host to the expected workspace/sandbox/service tuple.
   • Persist an endpoint, build a fresh state/index against the same store, rebuild the index, and verify the emitted hash host still resolves. This covers the required restart path without changing persistence schema.
   • For shared-store behavior, simulate a second state/index against the same test store: expose through one state, resolve the hash through the other, and verify miss-triggered index refresh finds the endpoint.

   E2E

   • No new runtime-driver E2E lane is required: this change does not alter sandbox/supervisor protocols, persistence schemas, or service relay transport.
   • Run focused server tests (cargo test -p openshell-server service_routing, service gRPC tests, and HTTP routing tests), core tests, and CLI tests.
   • Run Helm unit tests for deploy/helm/openshell/tests/gateway_config_test.yaml.
   • Run the repository’s applicable formatting, Helm lint/template, and documentation-link checks for changed files.

   Risks & Open Questions

   • Deviations: docs/reference/gateway-config.mdx, docs/sandboxes/manage-gateways.mdx, and architecture/gateway.md do not exist on current upstream/main (commit 05050a6d6). The canonical current equivalents are docs/how-it-works/gateways/configuration.mdx, docs/how-it-works/gateways/overview.mdx, and docs/how-it-works/sandboxes/overview.mdx; there is no standalone gateway architecture grammar document to update. The plan updates those canonical pages instead of creating duplicate/unlinked documentation.
   • Deviations: deploy/helm/openshell/templates/gateway-config.yaml is only a 22-line ConfigMap wrapper. The actual TOML translation is in deploy/helm/openshell/templates/_toml.tpl, so that is the required Helm-template change.
   • The current gateway configuration is immutable for process lifetime; there is no general gateway TOML hot-reload mechanism. "Rebuilt when config changes" therefore means rebuilt during the next process startup after a configuration change, not a live in-process config update.
   • Hash labels cannot be reversed from configuration alone. This forces parse_host callers to receive routing state/store access through an asynchronous resolver; a synchronous signature limited to &ServiceRoutingConfig is insufficient.
   • In HA, the index is per process while ServiceEndpoint persistence is shared. A hash miss must refresh from store before concluding the host is unknown. This creates a bounded extra lookup/scan for unknown hash-shaped hostnames; consider rate-limiting or negative-cache behavior only if profiling shows it is needed.
   • The 80-bit truncation makes accidental collisions extraordinarily unlikely, but the implementation must still detect collisions locally and during startup hydration. A simultaneous cross-replica collision remains impractical but cannot be made globally atomic without either broader mutation-lock coordination or persisted hash uniqueness, both outside the approved no-schema-change design.
   • Operators must assign distinct service_routing_gateway_name values to gateways sharing a base domain; including the name only produces cross-gateway uniqueness when the configured names are themselves unique.
   • No LSM impact expected: this change only changes configuration, URL construction, HTTP host classification, and in-memory routing state; it does not touch process identity, /proc, exec, or sandbox LSM policy paths.

   Documentation Impact

   • Add the field to the canonical gateway TOML reference with:
     • [openshell.gateway] service_routing_gateway_name = "gateway-west";
     • default unset;
     • validation ([a-z0-9]([-a-z0-9]*[a-z0-9])?, maximum 63 bytes);
     • precedence: --service-routing-gateway-name > OPENSHELL_SERVICE_ROUTING_GATEWAY_NAME > TOML > unset;
     • hash URL example such as https://enh47uqyj7bpyttz.apps.example.com/;
     • explicit statement that this is neither [openshell.gateway].name nor [openshell.gateway.gateway_jwt].gateway_id.
   • Update current gateway/sandbox service documentation to state the legacy grammar accurately: <workspace>--<sandbox>[--<service>].<base-domain>. Current docs/how-it-works/gateways/overview.mdx:45 incorrectly omits <workspace>, while implementation has always included it (service_routing.rs:193-197).
   • Explain compatibility: unset preserves existing readable URLs; configured gateways emit hashes but still accept legacy readable hosts during migration; changing the configured routing gateway name deliberately changes all hash URLs.
   • Add Kubernetes ingress guidance that the chart does not create routes for service hostnames. Operators should use exact emitted hash hosts in HTTPRoute/Ingress rules targeting the corresponding gateway Service, while a *.apps.example.com certificate remains sufficient because every generated service hostname is exactly one label under the base domain.
   • Explain the multi-gateway pattern: share one wildcard certificate/base domain, configure unique routing gateway names, and create exact-host rules per gateway/service endpoint. Add the Helm value comment and setup-table entry.

   Key design decisions verified in code

   • Current URL and persistence behavior: on upstream/main, endpoint_host builds workspace--sandbox[--service].base-domain at crates/openshell-server/src/service_routing.rs:186-198; endpoint_url regenerates a URL at :158-173; parse_host reverses the readable host at :203-232. ServiceEndpoint persistence uses endpoint_key(sandbox, service) (:150-156) and workspace-scoped lookups (:606-621), so URL data is not persisted and no proto/persistence schema change is needed.
   • Configuration threading: ServiceRoutingConfig currently has only base_domains and enable_loopback_service_http (crates/openshell-core/src/config.rs:311-319); Config::with_server_sans writes its base_domains at :1082-1095; startup applies service-routing values at crates/openshell-server/src/cli.rs:556-568. TOML service fields are in GatewayFileSection at config_file.rs:152-162, and file precedence is applied by merge_file_into_args at cli.rs:1191-1235.
   • Validation rule: the existing core DNS-label helper at config.rs:1173-1180 already enforces lowercase ASCII alphanumeric/hyphen labels, no edge hyphens, and 63-byte maximum. Reusing it for the optional routing gateway name is consistent with existing domain/SAN validation and keeps the setting suitable for ingress operations.
   • Route-map location and request path: there is no existing endpoint map. ServerState has only a pooled upstream map (service_upstreams, lib.rs:312-314); HTTP classification currently calls is_sandbox_service_request with only &state.config.service_routing (http.rs:202-212), and service proxying subsequently loads the endpoint from persistence (service_routing.rs:238-255, :337-350). The feature therefore requires a new ServerState route index plus startup hydration, state/store-aware host resolution, and lifecycle updates in both service RPC deletion (grpc/service.rs:295-349) and sandbox-owned endpoint cleanup (compute/mod.rs:5675-5703).
   • Hash specification and dependency choice: sha2 is already a workspace dependency and an openshell-server dependency; no base32/data-encoding crate is present in workspace manifests. A local fixed-size lowercase RFC-4648 encoder is preferable to adding a dependency for a single 10-byte/16-character conversion.
   • Existing name limit: MAX_ROUTABLE_NAME_LEN is 19 at crates/openshell-server/src/grpc/mod.rs:143-148 because legacy three-segment labels must fit DNS’s 63-character label limit. Keep it while legacy host parsing remains supported.

   Complexity / Confidence / Issue type / Estimated files

   • Complexity: High
   • Confidence: Medium-high
   • Issue type: feat
   • Estimated files: 17 modified files; no new proto, persistence migration, Cargo dependency, chart-managed ingress resource, or LSM-policy file.

   ---

   Revision 1 — initial plan, verified against upstream/main @ 05050a6d6
3. jhjaggars commented on Oct 22, 2026

   

   jhjaggars

   on Oct 22, 2026

   ContributorAuthor

   More actions

   🏗️ build-from-issue-agent

   Implementation Complete

   PR: #4842

   What was built

   An optional service_routing_gateway_name gateway setting (TOML/CLI/env/Helm) that, when configured, changes exposed-service hostnames to a deterministic 16-character base32 SHA-256 label of (gateway name, workspace, sandbox, service) under the same wildcard-cert base domain. Unset gateways keep the exact legacy readable format; configured gateways still accept legacy hosts. Hashed hosts resolve via an in-memory, store-hydrated route index with expose-time collision rejection.

   Tests

   • Unit: config validation/precedence, fixed hash vectors, hash URL construction, index register/remove/rebuild/collision, host resolution (Host + HTTP/2 authority, legacy + unknown-hash fallthrough), gRPC expose/delete lifecycle, CLI URL normalization
   • Integration: index rehydration (restart path) and cross-replica miss-triggered refresh
   • E2E: N/A — no protocol, persistence, or relay changes

   Local verification: cargo test -p openshell-core (559), cargo test -p openshell-server (2205), cargo test -p openshell-cli (targeted), mise run helm:test (345), clippy -D warnings, fmt — all green. mise run pre-commit has one pre-existing unrelated failure (examples/supervisor-middleware-content-guard/Cargo.lock references yanked clap 4.6.7), which also fails on unmodified main.

   Docs updated

   • docs/how-it-works/gateways/configuration.mdx, docs/how-it-works/gateways/overview.mdx (also fixes the pre-existing workspace-segment omission), docs/how-it-works/sandboxes/overview.mdx, docs/kubernetes/ingress.mdx, docs/kubernetes/setup.mdx, Helm chart README

   The issue will auto-close when the PR is merged.
4. TaylorMutch

   added

   state:acceptedA maintainer decided OpenShell should pursue this issueA maintainer decided OpenShell should pursue this issue

   and removed

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Oct 22, 2026
5. jhjaggars

   added a commit that references this issue on Oct 23, 2026

   feat(gateway): add hashed service-routing hostnames for multi-gateway…

   ...

   This issue will close once commit 28551bd is merged into the 'main' branch.

   28551bd