Valkey statefulset needs to be restarted daily

[zoez7]

Expected Behavior

Actual Behavior

Seems like valkey is not reloading the tls-cert-file after process start up?

Steps to Reproduce the Problem

Specifications

• Version:
• Platform:

[npolshakova]

Ran into this as part of #146. If we're looking at fixing this holistically, maybe we can:

1. make the valkey pods announce stable DNS names (e.g., valkey-0.valkey-headless.ns.svc) instead of pod IPs, so nodes.conf doesn't go stale when pods are recreated.
2. maybe add some sort of automated cert reload when mounted certs rotate

[BenTheElder]

I also think if we stick with valkey we should reduce the cost to spin it up in kind. It's too "robust" in that sense. It's the slowest part of bringing up a test cluster.
Maybe drop the replicas and just keep shards. But there's an open topic about the API storage layer, so I'd focus on just a simple patch in the short term.