Fix traefik tls certs during refresh

[Thanhphan1147]

The release of traefik revision 281 introduces managing certificates in "APP" mode. However, due to an issue with the tls-certiicates library, the corresponding private_key did not had the correct scope, which introduce a risk of service disruption . This PR proposes a fix so that deployments that are in a "degraded" state can self-heal after running the sunbeam refresh command.

The PR adds a custom logic towards the end of the refresh command to find outstanding CSRs, check that:

1. The subject matches the certificates provided by the operator via sunbeam tls ca unit_certs
2. The public key of the certificate matches the outstanding CSR
3. The certificate has not expired

Then rerun provide-certifcate action on manual-tls-certificates using a certificate that matched from the previous step.

We considered a few options:

1. Make this into a dedicated command ( like sunbeam fix-certs for example )
2. Make the logic only trigger via a new flag added to the refresh command ( like sunbeam refresh --fix-certs )
3. Make the logic runs automatically on each refresh under specific conditions

This PR implements option 3 and option 1 to simplify user UX but also provide an isolated way to fix the certificates.

Assisted-By: copilot

Related-PR: https://github.com/canonical/traefik-k8s-operator/pull/729/changes
Related-bug: canonical/traefik-k8s-operator#709

QA steps

These steps need to be performed on a sunbeam deployment after enabling tls ca for the public endpoints

1. Save the current private key

TLS_KEY_SECRET_ID=$(juju secrets --format yaml | python3 -c "
import sys,yaml
s=yaml.safe_load(sys.stdin)
[print(k) for k,v in s.items() if v.get('label')=='tls-key']")

juju show-secret ${TLS_KEY_SECRET_ID} --reveal --format yaml \
    | python3 -c "
import sys,yaml,json
d=list(yaml.safe_load(sys.stdin).values())[0]
yaml.dump({'private-keys':d['content']['private-keys']},open('/tmp/orig.yaml','w'))"

2. Replace the current private key with a random one ( we'll throw it away later, it's just to simulate a replacement of the CSR )

juju update-secret ${TLS_KEY_SECRET_ID} --file /tmp/new_tls_key.yaml
jhack fire traefik-public/0 config-changed
sleep 15

3. Restore the original key

juju update-secret ${TLS_KEY_SECRET_ID} --file /tmp/orig.yaml
jhack fire traefik-public/0 config-changed
sleep 15

4. Re-provide certificate

openstack.sunbeam cluster refresh

or

openstack.sunbeam cluster refresh certificates

To only trigger the certificate refresh.

[hmlanigan]

Question: how do we test this charnge?

[Raven-182]

Thanks for the patch. Some comments:

• ce0361a and f8b7da7 seem like fixups to 268ebdd so should be squashed.

[Thanhphan1147]

@Raven-182 I squashed all commits into 1

[Thanhphan1147]

@hmlanigan Here are the steps to test the changes ( I tested it on a 1-node sunbeam deployment after enabling tls ca for the public endpoints). The idea is to simulate a replacement of CSR after a leader change for example.

1. Save the current private key

TLS_KEY_SECRET_ID=$(juju secrets --format yaml | python3 -c "
import sys,yaml
s=yaml.safe_load(sys.stdin)
[print(k) for k,v in s.items() if v.get('label')=='tls-key']")

juju show-secret ${TLS_KEY_SECRET_ID} --reveal --format yaml \
    | python3 -c "
import sys,yaml,json
d=list(yaml.safe_load(sys.stdin).values())[0]
yaml.dump({'private-keys':d['content']['private-keys']},open('/tmp/orig.yaml','w'))"

2. Replace the current private key with a random one ( we'll throw it away later, it's just to simulate a replacement of the CSR )

juju update-secret ${TLS_KEY_SECRET_ID} --file /tmp/new_tls_key.yaml
jhack fire traefik-public/0 config-changed
sleep 15

3. Restore the original key

juju update-secret ${TLS_KEY_SECRET_ID} --file /tmp/orig.yaml
jhack fire traefik-public/0 config-changed
sleep 15

4. Re-provide certificate

openstack.sunbeam cluster refresh

or

openstack.sunbeam cluster refresh certificates

To only trigger the certificate refresh.

Do you think I need to turn this into a doc or integration test?

[hmlanigan]

@Thanhphan1147, the QA steps are for someone to be able to verify the changes in the PR. The tests added should be appropriate for regression.