Steward Trustd

steward-trustd is a Talos-compatible trustd service that enables Talos Linux worker nodes to operate with Steward-hosted Kubernetes control planes. It signs apid server certificates for Talos workers, allowing full talosctl management of nodes in Steward-managed tenant clusters.

Overview

When Talos workers connect to a non-Talos control plane (like Steward), they cannot get their apid certificates signed by the standard Talos trustd service. This causes the workers to remain in a "Booting" stage with recurring errors from the secrets.APIController.

steward-trustd solves this by implementing the Talos trustd gRPC interface (SecurityService) and signing certificates using a Steward-managed OS CA. This enables:

• Full talosctl management capabilities
• Talos stage shows "Running" instead of "Booting"
• No secrets.APIController errors
• Enterprise-ready Talos worker operations

Architecture

+---------------------------------------------------------------------------+
|                         Management Cluster                                |
|                                                                           |
|  +---------------------------+    +-------------------------------------+ |
|  | Steward TCP               |    | Steward Trustd (sidecar)            | |
|  | - K8s API server           |    | - Listens on :50001                 | |
|  | - Signs kubelet CSRs       |    | - Authenticates via trustd token    | |
|  | - Port 6443                |    | - Signs apid CSRs with OS CA        | |
|  +---------------------------+    +-------------------------------------+ |
|            |                                 |                            |
+---------------------------------------------------------------------------+
             |                                 |
             v                                 v
+---------------------------------------------------------------------------+
|                         Talos Worker Node                                 |
|                                                                           |
|  kubelet ---------> K8s API:6443        apid ---------> trustd:50001     |
|  (TLS bootstrap)   (Steward)            (CSR signing)  (Steward)         |
|                                                                           |
|  Result: Node fully functional with both Kubernetes AND Talos APIs       |
+---------------------------------------------------------------------------+

Features

• Talos-Compatible Protocol: Implements the same gRPC interface as native Talos trustd
• Secure by Default: TLS encryption, token authentication, constant-time comparisons
• Server-Auth Only: Strips client auth permissions from certificates (security measure)
• Kubernetes Native: Designed to run as a sidecar in the TCP pod
• Minimal Footprint: Scratch-based container image (~10MB)
• Multi-Architecture: Supports both amd64 and arm64

Quick Start

Generate Credentials

# Build the credential generation tool
make build-gencreds

# Generate credentials for a tenant cluster
./bin/steward-trustd-gencreds -cluster my-cluster -output creds/

# This creates:
# - creds/os-ca.crt / os-ca.key (OS CA)
# - creds/server.crt / server.key (trustd server cert)
# - creds/token (authentication token)
# - creds/kubernetes-secret.yaml (K8s Secret manifest)

Run Locally (Development)

# Build and run with test credentials
make run-local

Deploy to Kubernetes

In production, steward-trustd is automatically deployed as a sidecar container by Steward when spec.addons.workerBootstrap is configured on a TenantControlPlane.

For standalone deployment:

# Apply the generated secret
kubectl apply -f creds/kubernetes-secret.yaml

# Deploy steward-trustd (edit namespace/cluster name first)
kubectl apply -f deploy/kubernetes/deployment.yaml

Configuration

steward-trustd can be configured via environment variables or mounted files.

Environment Variables

Variable	Description	Default
TRUSTD_PORT	Port to listen on	50001
TRUSTD_TENANT	Tenant cluster name (for logging)	-
TRUSTD_TOKEN	Authentication token (machine.token)	-
TRUSTD_OS_CA_CRT	Base64-encoded OS CA certificate	-
TRUSTD_OS_CA_KEY	Base64-encoded OS CA private key	-
TRUSTD_SERVER_CRT	Base64-encoded server certificate	-
TRUSTD_SERVER_KEY	Base64-encoded server private key	-
TRUSTD_CERT_DIR	Directory containing cert files	/etc/steward-trustd/certs
TRUSTD_DEBUG	Enable debug logging	false

File-Based Configuration

When using TRUSTD_CERT_DIR, place these files in the directory:

/etc/steward-trustd/certs/
+-- os-ca.crt          # OS CA certificate (required)
+-- os-ca.key          # OS CA private key (required)
+-- server.crt         # Server certificate (optional, will generate)
+-- server.key         # Server private key (optional, will generate)
+-- token              # Authentication token (required)
+-- accepted-cas.crt   # Additional accepted CAs (optional)

Integration with Talos Machineconfig

When configuring Talos workers to connect to steward-trustd, the machineconfig must include the trustd endpoint and token. Steward generates this automatically, but for manual setup:

# Talos machineconfig snippet
machine:
  # ... other config
  certSANs:
    - ${TRUSTD_ENDPOINT}  # Add trustd IP if needed

cluster:
  controlPlane:
    endpoint: https://${K8S_API}:6443  # Steward K8s API

  # Talos will connect to trustd at the same IP on port 50001
  # The trustdinfo section is derived from secrets

The trustd endpoint should be reachable from worker nodes. When using LoadBalancer services, this is typically the same IP as the Steward control plane but on port 50001.

Security Considerations

Certificate Signing Policy

steward-trustd follows the same security policy as native Talos trustd:

1. Server Auth Only: Signed certificates only include ExtKeyUsageServerAuth
2. Organization Stripped: Any Organization field in the CSR is removed to prevent client auth
3. CSR Validation: CSR signatures are verified before signing
4. No Client Verification: Clients are not required to present certificates (token auth only)

Token Security

• Tokens are validated using constant-time comparison to prevent timing attacks
• TLS is required for all connections
• Tokens should be securely generated (32+ hex characters recommended)

Network Security

• Deploy with NetworkPolicy to restrict access to worker node CIDRs
• Use LoadBalancer or NodePort with appropriate firewall rules
• Consider running trustd in the same network segment as workers

Development

Prerequisites

• Go 1.23+
• Docker (for container builds)
• golangci-lint (for linting)

Building

# Build binary
make build

# Build all binaries (including gencreds)
make build-all

# Build container image
make docker-build

# Build multi-arch container
make docker-buildx

Testing

# Run tests
make test

# Run tests with coverage
make test-coverage

# Run linter
make lint

Project Structure

steward-trustd/
+-- cmd/
|   +-- steward-trustd/          # Main daemon
|   +-- steward-trustd-gencreds/ # Credential generation tool
+-- internal/
|   +-- auth/                    # Token authentication
|   +-- ca/                      # Certificate authority
|   +-- config/                  # Configuration management
|   +-- reg/                     # gRPC service registration
|   +-- server/                  # Main server
|   +-- tls/                     # TLS configuration
+-- pkg/
|   +-- creds/                   # Credential generation library
+-- deploy/
|   +-- kubernetes/              # K8s deployment manifests
+-- Dockerfile
+-- Makefile
+-- go.mod

Talosctl Functionality Matrix

With steward-trustd properly configured:

Command	Status	Notes
talosctl logs	Works
talosctl dmesg	Works
talosctl services	Works
talosctl upgrade	Works
talosctl reset	Works
talosctl dashboard	Works
talosctl kubeconfig	Works
talosctl get members	Limited	Discovery disabled
talosctl cluster show	Limited	Discovery disabled

License

Apache License 2.0

Contributing

Contributions are welcome! Please see the Steward CONTRIBUTING.md for guidelines.

Related Documentation

• Steward Documentation
• Talos Linux Documentation