Skip to content

Commit ad603db

Browse files
MrGuatoMrGuato
committed
feat: add new postt for grafana
1 parent 4aa377f commit ad603db

2 files changed

Lines changed: 334 additions & 0 deletions

File tree

‎public/posts/2026-03-25/gitops-tls-pipeline.svg‎

Lines changed: 85 additions & 0 deletions
Loading

‎src/content/posts/2026-3-25-26-securing-grafana-with-tls-in-a-fluxcd-homelab.md‎

Lines changed: 249 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,249 @@
1+
---
2+
title: "Securing Grafana with TLS in a FluxCD GitOps Homelab"
3+
description: "How I generated a self-signed TLS cert, SOPS-encrypted the Kubernetes secret, and shipped it through FluxCD GitOps to put HTTPS on my Grafana dashboard — without ever storing a plaintext secret in Git."
4+
pubDate: 2026-03-25
5+
tags: ["kubernetes", "gitops", "flux", "sops", "grafana", "tls", "monitoring", "homelab"]
6+
draft: false
7+
---
8+
9+
## The Goal
10+
11+
My k3s homelab runs kube-prometheus-stack, which deploys Grafana as part of the monitoring stack. I route traffic through a Cloudflare Tunnel, but the ingress controller (Traefik, bundled with k3s) still needs a TLS secret to do in-cluster HTTPS termination. The catch: all my secrets live in Git, SOPS-encrypted with age. I can't just `kubectl apply` a plaintext secret — that defeats the point of GitOps.
12+
13+
Here's exactly what I did.
14+
15+
---
16+
17+
## The Pattern
18+
19+
The full workflow looks like this:
20+
21+
1. Generate a self-signed TLS certificate with `openssl`
22+
2. Use `kubectl --dry-run=client` to render a `Secret` manifest without touching the cluster
23+
3. Encrypt the manifest **in place** with SOPS + age (only the `data` fields get encrypted)
24+
4. Move the encrypted file into the right GitOps path, wire up the kustomizations, and push
25+
5. FluxCD reconciles, the SOPS controller decrypts, and the Secret appears in the cluster
26+
27+
At no point does a plaintext private key touch Git.
28+
29+
---
30+
31+
## Step 1: Generate the Certificate
32+
33+
```bash
34+
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
35+
-keyout ./tls.key \
36+
-out ./tls.crt \
37+
-subj "/C=US/ST=Boston/L=Basement/O=Home Lab Heroes Inc./OU=Department of Monitoring/CN=grafana.yourdomain.net" \
38+
-addext "subjectAltName=DNS:grafana.yourdomain.net"
39+
```
40+
41+
The `-nodes` flag skips the passphrase (required for automated ingress use). The `-addext "subjectAltName=..."` line is important — modern browsers and ingress controllers validate the SAN, not just the CN.
42+
43+
> **Note:** This is a self-signed cert. It will trigger browser warnings if accessed directly. In my case, Cloudflare terminates the public-facing TLS, so this cert only exists inside the cluster between Traefik and the backend. For a direct-exposure setup, use cert-manager + Let's Encrypt instead.
44+
45+
---
46+
47+
## Step 2: Render the Secret Manifest Without Applying It
48+
49+
```bash
50+
kubectl create secret tls grafana-tls-secret \
51+
--cert=tls.crt \
52+
--key=tls.key \
53+
--namespace=monitoring \
54+
--dry-run=client \
55+
-o yaml > grafana-tls-secret.yaml
56+
```
57+
58+
`--dry-run=client` renders the YAML locally — no cluster connection needed. The output is a standard `kubernetes.io/tls` Secret with base64-encoded `tls.crt` and `tls.key` under the `data` key.
59+
60+
---
61+
62+
## Step 3: SOPS-Encrypt the Manifest
63+
64+
```bash
65+
export AGE_PUBLIC=age1esq3qzaqeuwrfzu8fx89u9k8dl3uvmse460l29kt28yj8vefn9wstkgdn7
66+
67+
sops --age=$AGE_PUBLIC \
68+
--encrypt --encrypted-regex '^(data|stringData)$' \
69+
--in-place grafana-tls-secret.yaml
70+
```
71+
72+
Two things worth calling out here:
73+
74+
**`--encrypted-regex '^(data|stringData)$'`** — This tells SOPS to only encrypt the actual secret values. The `kind`, `metadata`, `name`, `namespace`, and `type` fields stay in plaintext. This makes Git diffs readable and keeps the SOPS MAC (message authentication code) covering only what matters.
75+
76+
**`--in-place`** — Overwrites the file with its encrypted version. After this, `tls.crt` and `tls.key` inside the YAML are AES256-GCM ciphertext. The age recipient public key is embedded in the `sops:` block at the bottom.
77+
78+
The raw `tls.key` and `tls.crt` files from Step 1 should be **deleted and gitignored** immediately. They served their purpose.
79+
80+
---
81+
82+
## Step 4: Place Files in the GitOps Tree
83+
84+
My repo uses a `base/overlay` Kustomize pattern, with Flux auto-discovering subfolders that contain a `kustomization.yaml`. For monitoring, I maintain a `monitoring/configs/` tree separate from `monitoring/controllers/` — configs holds environment-specific secrets and patches, controllers holds HelmReleases and HelmRepositories.
85+
86+
I created this structure:
87+
88+
```
89+
monitoring/
90+
└── configs/
91+
└── staging/
92+
├── kustomization.yaml
93+
└── kube-prometheus-stack/
94+
├── kustomization.yaml
95+
└── grafana-tls-secret.yaml ← the SOPS-encrypted file
96+
```
97+
98+
The `kustomization.yaml` at `configs/staging/kube-prometheus-stack/` is straightforward:
99+
100+
```yaml
101+
apiVersion: kustomize.config.k8s.io/v1beta1
102+
kind: Kustomization
103+
resources:
104+
- grafana-tls-secret.yaml
105+
```
106+
107+
And the one at `configs/staging/` lists the subdirectory:
108+
109+
```yaml
110+
apiVersion: kustomize.config.k8s.io/v1beta1
111+
kind: Kustomization
112+
resources:
113+
- kube-prometheus-stack
114+
```
115+
116+
Flux picks this up because `clusters/staging/monitoring.yaml` points at the `monitoring/` directory.
117+
118+
---
119+
120+
## Step 5: Update the HelmRelease Values
121+
122+
In `monitoring/controllers/base/kube-prometheus-stack/release.yaml`, I added the ingress + TLS configuration to the Grafana values block:
123+
124+
```yaml
125+
values:
126+
grafana:
127+
ingress:
128+
enabled: true
129+
ingressClassName: traefik
130+
hosts:
131+
- grafana.yourdomain.net
132+
tls:
133+
- secretName: grafana-tls-secret
134+
hosts:
135+
- grafana.yourdomain.net
136+
```
137+
138+
The `secretName` here must match the `metadata.name` in `grafana-tls-secret.yaml` exactly — a mismatch causes a silent TLS failure where the ingress comes up but serves no certificate.
139+
140+
---
141+
142+
## Step 6: Commit and Push
143+
144+
```bash
145+
git add clusters/ apps/ monitoring/
146+
git commit -m "feat: add grafana TLS secret"
147+
git push
148+
```
149+
150+
Flux reconciles on its interval (or immediately with `flux reconcile`), the SOPS controller decrypts the secret using the age key stored as a cluster secret, and the `grafana-tls-secret` Secret appears in the `monitoring` namespace. Traefik picks it up, and Grafana is now accessible at `https://grafana.yourdomain.net` with in-cluster TLS.
151+
152+
---
153+
154+
## What Actually Broke (And How I Fixed It)
155+
156+
This wasn't a clean one-shot deployment. Two bugs hit in sequence.
157+
158+
### Bug 1: YAML indentation broke the entire kustomize build
159+
160+
After the first push, `flux get kustomizations` showed `monitoring` as `READY: False` with this error:
161+
162+
```
163+
MalformedYAMLError: yaml: line 40: mapping values are not allowed in this context in File: release.yaml
164+
```
165+
166+
The culprit was a subtle indentation mistake in the `tls` block of `release.yaml`. The `hosts` key was indented one level too far, making it a child of `secretName` instead of a sibling:
167+
168+
```yaml
169+
# What I wrote (wrong)
170+
tls:
171+
- secretName: grafana-tls-secret
172+
hosts:
173+
- grafana.yourdomain.net
174+
175+
# What it should be
176+
tls:
177+
- secretName: grafana-tls-secret
178+
hosts:
179+
- grafana.yourdomain.net
180+
```
181+
182+
YAML list item fields (`secretName` and `hosts` are both keys on the same list element) must be at the same indentation level. The extra indent made YAML see `hosts` as a nested mapping inside `secretName`'s value, which is invalid.
183+
184+
Fixed, committed, pushed. The `monitoring` kustomization went green.
185+
186+
### Bug 2: The configs path wasn't registered with Flux
187+
188+
With the YAML fixed, Flux reconciled `monitoring` successfully — but `kubectl get secrets -n monitoring` still showed no `grafana-tls-secret`. Running `flux get kustomizations` showed only three entries: `apps`, `flux-system`, and `monitoring`. The `monitoring-configs` kustomization I expected didn't exist.
189+
190+
The reason: `clusters/staging/monitoring.yaml` only pointed Flux at `./monitoring/controllers/staging`. The `./monitoring/configs/staging` path — where the encrypted secret lives — was never registered. Flux doesn't auto-discover all subdirectories; you have to explicitly tell it where to look.
191+
192+
The fix was adding a second `Kustomization` block to `monitoring.yaml`:
193+
194+
```yaml
195+
---
196+
apiVersion: kustomize.toolkit.fluxcd.io/v1
197+
kind: Kustomization
198+
metadata:
199+
name: monitoring-configs
200+
namespace: flux-system
201+
spec:
202+
interval: 1m0s
203+
retryInterval: 1m
204+
timeout: 5m
205+
sourceRef:
206+
kind: GitRepository
207+
name: flux-system
208+
path: ./monitoring/configs/staging
209+
prune: true
210+
decryption:
211+
provider: sops
212+
secretRef:
213+
name: sops-age
214+
```
215+
216+
After pushing and reconciling the source, `flux get kustomizations` showed four entries including `monitoring-configs: READY True`. A few seconds later, `kubectl get secrets -n monitoring` showed `grafana-tls-secret` with `TYPE: kubernetes.io/tls` and `DATA: 2`. Done.
217+
218+
---
219+
220+
## Confirming It Worked
221+
222+
```bash
223+
kubectl get ingress -n monitoring
224+
# NAME CLASS HOSTS ADDRESS PORTS
225+
# kube-prometheus-stack-grafana traefik grafana.yourdomain.net 192.168.1.21 80, 443
226+
227+
flux reconcile helmrelease kube-prometheus-stack -n monitoring
228+
# ✔ applied revision 82.14.0
229+
```
230+
231+
The ingress shows `PORTS: 80, 443` — TLS is configured. The Helm release reconciled cleanly against the updated values.
232+
233+
---
234+
235+
**GitOps + secrets = SOPS, always.** You cannot push a plaintext Kubernetes Secret to a Git repo and call it GitOps. The SOPS + age workflow gives you an encrypted file that Flux knows how to decrypt at apply time — the best of both worlds.
236+
237+
**Only encrypt what matters.** The `--encrypted-regex` flag is a deliberate choice. Keeping metadata in plaintext makes code review possible. The encrypted fields are protected by AES256-GCM; the unencrypted fields are just labels.
238+
239+
**Secret name consistency is a hidden gotcha.** The name in the SOPS manifest, the name referenced in your HelmRelease values, and the namespace must all align. Check these first if TLS isn't working after reconciliation.
240+
241+
**`kubectl` can't decrypt SOPS secrets.** Don't try `kubectl apply -f grafana-tls-secret.yaml` directly. Only Flux + the SOPS controller can handle decryption. Keep your imperative kubectl habits away from encrypted manifests.
242+
243+
---
244+
245+
## Repo
246+
247+
All manifests for this setup live in [github.com/MrGuato/pi-cluster](https://github.com/MrGuato/pi-cluster).
248+
249+
*Built with ❤️ by Jonathan — If it is not in Git, it does not exist.*

0 commit comments

Comments
 (0)