nimbus

An open-source cloud-native deployment tool.

Prerequisites

Before setting up Nimbus, ensure you have the following installed:

Kubernetes (either a remote cluster or local installation)
Docker
kubectl
Make
Go

For local development, we recommend using Minikube to set up a local Kubernetes cluster. Ensure your Kubernetes configuration is available at ~/.kube/config.

Server Installation

To deploy Nimbus on an existing Kubernetes cluster:

Create the required namespace:
```
kubectl create namespace nimbus
```
Apply the Kubernetes configurations:
```
kubectl apply -f kubernetes/
```

Production Considerations

In a production environment, the Nimbus server will have elevated admin permissions over your entire Kubernetes cluster. These permissions are specified in the permissions.yaml file through the ServiceAccount configuration. Ensure you review and understand the security implications before deployment.

Additionally, to run Nimbus in production, you must set the environment variable ENVIRONMENT=production. If this variable is not specified, it defaults to development.

Persistent Storage Requirement

Nimbus volumes use a Kubernetes dynamic storage provisioner. Set NIMBUS_STORAGE_CLASS to the production storage class; there is no default. Use reclaimPolicy: Retain when production data must survive application deletion.

Set NIMBUS_PREVIEW_STORAGE_CLASS to a separate class with reclaimPolicy: Delete. Nimbus uses it for new volumes on every branch except main/master. It rejects new preview volumes if this setting is missing or the class does not explicitly use Delete. Existing claims are reused on ordinary redeploys; they are not recreated or silently migrated. Stateless previews do not need this setting.

For Rancher local-path clusters, local-preview.yaml is an optional example. Match its topology and any provisioner path parameters to your existing production class, apply it separately, then configure Nimbus:

NIMBUS_STORAGE_CLASS=local
NIMBUS_PREVIEW_STORAGE_CLASS=local-preview
NIMBUS_CLEANUP_TIMEOUT=3m

Deleting a preview removes its PVCs and namespace. Nimbus waits for the namespace, claims, and all PVs referring to that namespace to disappear before reporting success and removing volume database records. The provisioner owns deletion of PV data; Nimbus never force-deletes PV objects, removes finalizers, or deletes host directories. Verify that your provisioner's teardown actually removes data (some NFS provisioners archive directories even with a Delete policy). Deletion errors and timeouts return an API error and retain volume records for retry. Temporary API deletion errors receive bounded retries. A retry also finds PVs whose PVCs were already deleted using their claim references. Production cleanup preserves its configured PV retention policy.

Existing previews: changing the setting affects new claims only. Preview cleanup refuses to remove claims if a PV in that namespace still uses Retain. An administrator must first verify that each such PV belongs to a disposable preview and explicitly change that PV's reclaim policy to Delete, then retry cleanup. Do not change the production class or bulk-patch all released PVs. Older orphan PVs whose branch database records have already been removed require separate, deliberate cleanup.

Automatically retire branch previews

Nimbus does not receive GitHub branch events itself. Call its branch-delete API when a PR closes (merged or unmerged) and when its branch is deleted. The reusable cleanup workflow retries failed requests, refuses main/master or an empty branch, and waits for the API's cleanup result. Add a caller like this to each application repository, pinning the Nimbus workflow to a reviewed commit or release:

name: Clean up previews
on:
  pull_request:
    types: [closed]
  delete:
permissions: {}
# Use the SAME group in the preview deployment workflow so closing a PR cancels
# an in-progress deployment instead of allowing it to recreate the preview.
concurrency:
  group: nimbus-preview-${{ github.event.pull_request.head.ref || github.event.ref }}
  cancel-in-progress: true
jobs:
  cleanup:
    if: >-
      (github.event_name == 'pull_request' &&
       github.event.pull_request.head.repo.full_name == github.repository) ||
      (github.event_name == 'delete' && github.event.ref_type == 'branch')
    uses: rayman-tech/nimbus/.github/workflows/cleanup-preview.yml@<commit-or-release>
    with:
      project: your-project
      branch: ${{ github.event.pull_request.head.ref || github.event.ref }}
      server: https://nimbus.example.com
    secrets:
      NIMBUS_API_KEY: ${{ secrets.NIMBUS_API_KEY }}

Repeated PR-close and branch-delete events are safe. A failed job must be retried; check for no remaining branch namespace, no PVs referencing its namespace, and successful provisioner teardown. A healthy disposable-preview lifecycle should be tested with a temporary volume before enabling this policy broadly. No live storage migration is performed by installing the new Nimbus binary.

To restrict deployments to only the main or master branches for a project, add allowBranchPreviews: false to your project's nimbus.yaml. When disabled, deploy requests from any other branch will be rejected.

Local Development

For local development, you can run Nimbus either directly or using Docker Compose.

Running with Docker Compose (Recommended)

This method runs the server with hot-reloading enabled and includes a PostgreSQL database:

Copy the .env.example files and set the required variables:

cp .env.example .env
cp .env.database.example .env.database

Ensure Kubernetes is running locally (Minikube is recommended):
```
minikube start
```
Start the development environment:
```
make docker-up
```
To stop the development environment:
```
make docker-down
```
To stop and remove volumes:
```
make docker-down-volumes
```

You may view the logs of the container as such:

docker logs nimbus-api

Running Directly

Alternatively, you can run the server directly without Docker:

Copy the .env.example file to .env and set the required environment variables:
```
cp .env.example .env
```
Ensure Kubernetes is running locally (Minikube is recommended):
```
minikube start
```
Run the Nimbus server locally:
```
make server
```

Development Tools

The following tools are included as development dependencies:

golangci-lint - Go linter aggregator for running multiple linters
pg_format - PostgreSQL SQL syntax formatter
air - Hot-reloading tool for Go applications

These tools are automatically configured in the project's Makefile for linting, formatting, and development tasks.

Deployment

A nimbus.yaml file needs to be present in your repository to deploy to Nimbus. This file defines the services that will be deployed, as well as the networking and environment configurations. There are pre-defined templates for services, such as databases and Redis, to make configuration easier. A sample nimbus.yaml file is available here.

Deployment can be done through our GitHub action, or through the local CLI, which is used for managing project state.

Following published images

Set imageAutoRefresh: true on an individual service to follow its image tag. Nimbus resolves the tag to an immutable digest on deployment, then checks for changes using the global NIMBUS_IMAGE_REFRESH_INTERVAL (default 5s). Services without this flag are not polled. A changed digest starts a rollout; an unchanged digest does not restart the service. The publishing repository only needs to build, test, and publish its image; no deployment action is needed for image updates.

See image refresh setup for registry credentials, configuration, status, and disabling updates or rolling back.

Monitoring

To have Prometheus scrape a service, add a monitoring block to it. Only the port is required; the path defaults to /metrics:

services:
  - name: api
    image: docker.prayujt.com/api:latest
    network:
      ports: [8080]
    monitoring:
      port: 8080      # container port exposing metrics
      path: /metrics  # optional, defaults to /metrics

Nimbus adds the standard prometheus.io/scrape, prometheus.io/port, and prometheus.io/path annotations to the service's pods, which a Prometheus instance using the kubernetes-pods scrape job will discover automatically. Omit the block to disable scraping.

Nimbus also exposes its own Prometheus metrics at GET /metrics (HTTP request counts/latency, deploy outcomes, Kubernetes API operation timings, and Go runtime/process metrics). This endpoint is unauthenticated and bypasses the API validator.

Envoy Gateway routing

Public http services now create Gateway API resources instead of NGINX Ingress objects. The existing ingress: YAML field, API/DB hostname field, and ingress-host environment references retain their names for compatibility. Custom domains on main/master and previously assigned preview hostnames are preserved. The GitHub action continues to submit the same deployment request.

Nimbus creates an HTTPS listener on the configured shared Gateway, an HTTPRoute (or GRPCRoute for features: [grpc]), an HTTP-to-HTTPS redirect, a Certificate, and the required policies/ReferenceGrant. Resource and listener names identify their service; only long names get a short collision-resistant suffix. Other applications' Gateway listeners are preserved during updates and cleanup. Existing preview hostname strings are not renamed.

Before deploying this Nimbus version:

Install Envoy Gateway 1.9.1 with Gateway API 1.6.1, including its extension CRDs. Enable config.envoyGateway.extensionApis.enableLua=true in the controller chart for the request-size policies, then roll out the controller after configuration changes.
Configure ENVOY_GATEWAY_NAME (default edge), ENVOY_GATEWAY_NAMESPACE (default envoy-gateway-system), and ENVOY_HTTP_LISTENER (default http). The Gateway needs a port-80 HTTP listener admitting routes from Nimbus application namespaces. Its public Service/router entrypoint must already exist; Nimbus does not replace it.
Enable Gateway API in cert-manager and configure letsencrypt-prod (or the service's selected ClusterIssuer) for Gateway HTTP-01 on that listener. Certificate issuance must complete before a deployment is reported ready.
For SPA or legacy external-auth services, set NIMBUS_ROUTE_HELPER_IMAGE to a digest-pinned image built from this Nimbus version, available for the application's node architectures. For example, registry.example.com/nimbus@sha256:<actual-manifest-digest>. This is not the separate kubeconfigs authentication helper image.
Nimbus needs access to its application resources and the configured shared Gateway/ClientTrafficPolicies. The existing installation example grants cluster-admin; this migration does not broaden that binding.

features: [grpc] creates a GRPCRoute and marks the backend Service port kubernetes.io/h2c. Ordinary HTTP routes retain the former 1 MiB request limit and 5s connect / 60s idle timeouts. gRPC and WebSocket upgrades are not buffered by the HTTP body-limit filter.

features: [spa] uses a small per-service helper to preserve the former upstream-404 → /index.html fallback. External authentication uses the same helper's separate auth port. Helpers run the nimbus route-helper subcommand, with two replicas, read-only configuration, no service-account token, and no privileges. Hostnames and upstreams are fixed by each service's ConfigMap; requests cannot select an arbitrary upstream. Successful SPA responses stream.

Envoy annotations

Use envoy.nimbus.dev/* settings in nimbus.yaml. These are Nimbus configuration keys: Nimbus validates them and generates Gateway API routes and Envoy policies. They are not native Envoy Gateway annotations, and copying them onto an arbitrary Kubernetes resource does not configure Envoy. Unknown keys under this prefix are rejected.

For example, external authentication:

services:
  - name: web
    template: http
    public: true
    ingress: app.example.com
    image: registry.example.com/web:latest
    network:
      ports: [8080]
    annotations:
      envoy.nimbus.dev/auth-url: "https://idp.example.com/sessions/whoami"
      envoy.nimbus.dev/auth-signin: "https://login.example.com/start?rd=$scheme://$host$request_uri"

The adapter accepts a successful 2xx session check, preserves 403 denial, and turns a 401 into the configured login redirect (or returns 401 when no sign-in URL is supplied). Network errors, missing required identity headers, and unexpected auth responses deny access. Envoy removes incoming identity headers before auth; only configured headers returned by the auth service reach the app. This preserves existing session checks; it does not add user/role management.

All keys below use the envoy.nimbus.dev/ prefix:

Key	Values / behavior
`backend-protocol`	`http` or `h2c`; `h2c` selects a GRPCRoute and HTTP/2 backend. `features: [grpc]` remains shorthand. Backend TLS is not implemented.
`connect-timeout`	Positive duration, default `5s`.
`stream-idle-timeout`	Duration with no stream activity, default `60s`; `0s` disables it.
`request-timeout`	Entire upstream response timeout, default `0s` (disabled).
`max-stream-duration`	Maximum stream duration, default `0s` (unlimited).
`grpc-service`, `grpc-method`	Optional exact protobuf service/method matches on a gRPC route; omitted means all services/methods.
`grpc-retry-count`, `grpc-retry-on`	Explicit opt-in retries; both required. Count 1–10. Triggers listed below.
`grpc-per-retry-timeout`	Optional positive duration; requires retry count and triggers.
`auth-url`, `auth-signin`, `auth-response-headers`	Session check URL, optional login redirect, comma-separated authenticated identity headers.
`enable-cors`	`true` or `false`; default disabled.
`cors-allow-origin`, `cors-allow-methods`, `cors-allow-headers`, `cors-expose-headers`	Comma-separated CORS values.
`cors-allow-credentials`, `cors-max-age`	Boolean and cache lifetime in seconds.
`body-size`	Request limit, default `1m`; integer bytes or `k`/`m`/`g` suffix; `0` unlimited. Not buffered for gRPC.
`spa`	`true` enables the helper fallback; `features: [spa]` remains shorthand. Cannot combine with gRPC.
`cluster-issuer`	Certificate ClusterIssuer, default `letsencrypt-prod`.
`ssl-redirect`	`true`; public routes always redirect HTTP to HTTPS.

Timeouts accept Gateway API duration syntax such as 250ms, 5s, or 5m, up to 24 hours. Retry triggers are cancelled, deadline-exceeded, internal, resource-exhausted, unavailable, connect-failure, refused-stream, and reset-before-request. Retries are absent by default; enable them only for RPCs that are safe to repeat. Envoy's gRPC status retries apply to response headers, not status codes delivered in trailers, and cannot replay an established stream.

services:
  - name: grpc-api
    template: http
    public: true
    image: registry.example.com/grpc-api:latest
    features: [grpc]
    network:
      ports: [50051]
    annotations:
      envoy.nimbus.dev/backend-protocol: "h2c"
      envoy.nimbus.dev/connect-timeout: "5s"
      envoy.nimbus.dev/stream-idle-timeout: "300s"
      envoy.nimbus.dev/request-timeout: "0s"
      # Optional restriction to a single protobuf service / RPC:
      # envoy.nimbus.dev/grpc-service: "example.v1.Catalog"
      # envoy.nimbus.dev/grpc-method: "GetItem"
      # Optional retries for an RPC that is safe to repeat:
      # envoy.nimbus.dev/grpc-retry-count: "2"
      # envoy.nimbus.dev/grpc-retry-on: "unavailable"
      # envoy.nimbus.dev/grpc-per-retry-timeout: "5s"

This annotation interface supports a single service/method match and retry policy per Nimbus service. Multiple match rules and weighted backends would need a future structured configuration interface; arbitrary policy YAML is not accepted.

Deprecated NGINX aliases

Previously supported nginx.ingress.kubernetes.io/* keys remain deprecated input aliases. Prefer the Envoy keys above for new configuration. Most keep the same suffix; proxy-body-size becomes body-size, proxy-connect-timeout becomes connect-timeout, and proxy-read-timeout / proxy-send-timeout become stream-idle-timeout. Old timeout values are integer seconds; new ones include units. backend-protocol: GRPC becomes backend-protocol: h2c, and the exact old SPA configuration-snippet becomes spa: "true". The existing cert-manager.io/cluster-issuer also remains an alias.

Supplying conflicting new and old settings is a validation error. Matching aliases are accepted (duration values are compared by elapsed time). An explicit HTTP backend conflicts with features: [grpc]; spa: "false" conflicts with the SPA feature/snippet. Unknown NGINX annotations and arbitrary snippets are rejected. CORS origin settings alone remain inert unless enable-cors: "true" is set. CORS and authentication share a SecurityPolicy when both apply. Non-NGINX metadata annotations, including the explicit Envoy inputs, remain visible on the HTTPS route.

Redeploy and cleanup

Redeploy each existing service to migrate its stored hostname. Nimbus installs helper/policy resources before attaching the route, waits for fresh route and policy acceptance, certificate readiness, and helper availability, then deletes only that service's old <service>-ingress. Existing TLS Secrets and private-key settings are preserved; old Ingress certificate ownership is detached. A failed readiness check reports deployment failure and retains the legacy Ingress; it does not provide a transactional rollback of all application changes. ENVOY_READY_TIMEOUT_SECONDS defaults to 180 per service. Set the proxy in front of the Nimbus API to allow enough idle time for the deployment request (the sum of sequential service waits); the action does not stream progress.

Making a service private, changing it to a non-HTTP template, or deleting a service/preview removes its owned routes, shared listener, policies, and helper. Cleanup errors retain the DB record/hostname for retry. Certificates/keys are retained for redeployment; deleting the branch namespace removes its namespaced resources. Do not delete namespaces outside Nimbus and expect shared Gateway listeners to be garbage-collected automatically.

The shared Gateway has a 64-listener limit. Capacity and hostname conflicts produce errors; Nimbus does not overwrite another application's listener. Keep the configured Gateway stable while services are deployed, and avoid reapplying a static Gateway manifest that discards Nimbus-added listeners.

This change does not deploy Envoy, switch router ports, or migrate every stored service at Nimbus startup. Test browser login, uploads, streaming, and gRPC in your deployment environment before retiring a still-active old controller.

Routing validation

go test -race ./...
go vet ./...
NIMBUS_ROUTE_FIXTURES=/tmp/nimbus-routes.json go test ./internal/kubernetes -run TestExportRouteFixtures
python3 scripts/validate-routes.py /tmp/nimbus-routes.json /path/to/envoy-v1.9.1-install.yaml --egctl /path/to/egctl

The offline validator uses the pinned Envoy CRD schemas and synthetic certificates/services for xDS translation; it never contacts a cluster. Unit tests cover the helper, hostname compatibility, auth/CORS preservation, gRPC, readiness-gated ingress retirement, listener isolation, and cleanup ownership.

API Documentation

The Nimbus API is documented using the OpenAPI 3.0.1 specification, available in docs/api.yaml. This documentation provides detailed information about all available endpoints, request/response schemas, and authentication requirements.

Viewing the Documentation

A Swagger UI server is included in both the Docker Compose and Kubernetes configurations for interactive API exploration:

Docker Compose: When running make docker-up, Swagger UI is automatically available at http://localhost:8080/docs/
Kubernetes: The Swagger UI is deployed as part of the Kubernetes manifests in kubernetes/ and can be accessed through the configured ingress or service

The Swagger UI provides an interactive interface to explore the API, view schemas, and test endpoints directly from your browser.

CLI Installation

Install the Nimbus CLI locally with:

sudo make install

This command builds the nimbus binary and copies it to /usr/local/bin, allowing you to run nimbus from any directory.

CLI Usage

After installing the CLI, you can deploy your application easily:

nimbus deploy

Flags:

-H, --host – Nimbus server address. Defaults to the NIMBUS_HOST environment variable or http://localhost:8080.
-f, --file – Path to the nimbus.yaml file. Defaults to ./nimbus.yaml.
-a, --apikey – API key used for authentication. Defaults to the NIMBUS_API_KEY environment variable.

The client CLI exposes several subcommands:

nimbus deploy – deploy a project using a nimbus.yaml file.
nimbus projects – manage projects (create, list, delete).
nimbus services – inspect services (list, get, logs).
nimbus secrets – manage project secrets (list, edit).
nimbus branch delete – remove a branch and its resources.

Running nimbus server will start the server locally.

Contributing

We welcome contributions! Feel free to submit issues and pull requests to improve Nimbus.

Native Authentik forward auth

For public HTTP applications, add this structured field to the service in nimbus.yaml:

auth:
  provider: authentik

Nimbus always uses the cluster installation at authentik-server.authentik:80. There are no per-service server overrides. Omit auth to disable this integration. Unknown auth fields/providers and the superseded Authentik annotations are rejected.

Nimbus emits a fail-closed SecurityPolicy calling /outpost.goauthentik.io/auth/envoy and an unauthenticated callback HTTPRoute for /outpost.goauthentik.io only. The normal application route stays protected. The outpost returns login redirects directly; no auth-signin adapter is required. Incoming identity headers are removed before authorization. SPA helpers remain independent; an auth-only application no longer needs a helper image or deployment.

Configure a forward-auth provider in Authentik and assign it to a healthy outpost before deployment. Single-application providers must match the actual hostname, including previews. A domain-level provider can cover previews under a shared parent domain with the same access policy; unknown hosts still fail closed. Nimbus does not configure Authentik users, providers, or access policies.

Nimbus creates a separate ReferenceGrant for each authenticated service in the Authentik namespace. It admits only SecurityPolicy and HTTPRoute references from that service's namespace to the exact authentik-server Service. This works for dynamically created preview namespaces without editing a shared allowlist. Grants carry Nimbus ownership labels and a Namespace owner reference, so namespace deletion also garbage-collects them after interrupted deployments. Nimbus leaves administrator-managed grants unchanged.

The Nimbus service account needs get/create/update/delete access to ReferenceGrants in authentik and read access to application Namespaces (the supplied Kubernetes permissions already include these). Any NetworkPolicy must permit the Envoy proxy to reach the outpost. Route readiness failures report the pending resource and Envoy condition reasons, including denied references. Do not attach auth to the callback route, and do not use a Gateway-wide auth policy that also catches callbacks.

This mode cannot be combined with auth-url/auth-signin (including deprecated NGINX aliases), and browser forward-auth on GRPCRoute is rejected. Existing gRPC routing, legacy auth and SPA behavior are unchanged. Removing the setting or deleting the service also removes its owned callback route and Authentik grant. Other services in the same preview namespace keep their grants. The feature is inactive until a service explicitly selects it.

Container builds

Go images are built with ko v0.19.1 using the Go version in go.mod. CI saves Go modules and compiler outputs between commits. Image repositories, tags, target architectures, and deployment triggers are preserved.

To build locally with ko and Docker installed:

ko build ./cmd --local --platform=linux/amd64

The development Compose workflow continues to use Dockerfile.dev.

Navigation Menu

Repository navigation

About

Repository files navigation

nimbus

Prerequisites

Server Installation

Production Considerations

Persistent Storage Requirement

Automatically retire branch previews

Local Development

Running with Docker Compose (Recommended)

Running Directly

Development Tools

Deployment

Following published images

Monitoring

Envoy Gateway routing

Envoy annotations

Deprecated NGINX aliases

Redeploy and cleanup

Routing validation

API Documentation

Viewing the Documentation

CLI Installation

CLI Usage

Contributing

Native Authentik forward auth

Container builds

About

Releases

Packages

Used by

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
bin		bin
cmd		cmd
docs		docs
internal		internal
kubernetes		kubernetes
scripts		scripts
.air.toml		.air.toml
.dockerignore		.dockerignore
.env.database.example		.env.database.example
.env.example		.env.example
.gitignore		.gitignore
.golangci.yaml		.golangci.yaml
.ko.yaml		.ko.yaml
.pg_format		.pg_format
CLAUDE.md		CLAUDE.md
Dockerfile.dev		Dockerfile.dev
Makefile		Makefile
README.md		README.md
docker-compose.dev.yaml		docker-compose.dev.yaml
fileserver.conf		fileserver.conf
go.mod		go.mod
go.sum		go.sum
sqlc.yaml		sqlc.yaml

Latest commit

History

Folders and files

Repository files navigation

nimbus

Prerequisites

Server Installation

Production Considerations

Persistent Storage Requirement

Automatically retire branch previews

Local Development

Running with Docker Compose (Recommended)

Running Directly

Development Tools

Deployment

Following published images

Monitoring

Envoy Gateway routing

Envoy annotations

Deprecated NGINX aliases

Redeploy and cleanup

Routing validation

API Documentation

Viewing the Documentation

CLI Installation

CLI Usage

Contributing

Native Authentik forward auth

Container builds

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages