Skip to content

Commit 89a9ef9

Browse files
vakwetuclaude
authored and
Ade Lee
committed
[skmo] Add Skupper for cross-region RabbitMQ and Keystone internal routing
Add hook playbooks and configuration to establish Skupper virtual services for RabbitMQ and Keystone internal endpoints, enabling cross-region connectivity in the multi-namespace SKMO scenario. skupper-connector.yaml: query the RabbitMQ TLS secret from the correct CRD - rabbitmq.openstack.org/v1beta1 (RabbitMq) as used by the OpenStack infra-operator, not the community rabbitmq.com/v1beta1 (RabbitmqCluster). Add retries to wait for spec.tls.secretName to be populated before creating the Skupper Connector. skupper-keystone-connector.yaml: add retries to the KeystoneAPI CR lookup to wait for spec.tls.api.internal.secretName to be available, since that field is not populated until Keystone completes TLS setup. configure-leaf-keystone-internal.yaml: after patching the leaf OSCP to use the Skupper Keystone virtual service, also create a MetalLB LoadBalancer Service (keystone-regionone-lb) and a DNSData CR (keystone-skupper) so that EDPM compute nodes outside the OCP cluster can resolve and connect to the Keystone auth_url. The Skupper Listener creates a ClusterIP-only Service that is unreachable from EDPM nodes; the LoadBalancer Service obtains a MetalLB IP on the leaf internalapi network and the DNSData entry registers both the short (.svc) and fully-qualified (.svc.cluster.local) names in the dnsmasq instance serving those nodes. Signed-off-by: Ade Lee <alee@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Made-with: Cursor
1 parent 40e754a commit 89a9ef9

9 files changed

Lines changed: 791 additions & 34 deletions

‎hooks/playbooks/skmo/configure-leaf-keystone-internal.yaml‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
---
2+
# Patch the leaf/workload region OpenStackControlPlane to use the Skupper
3+
# Listener virtual endpoint for internal Keystone authentication traffic.
4+
#
5+
# The public endpoint override is left unchanged so that end-user traffic and
6+
# the Keystone service catalog continue to reference the central region's
7+
# external (public) URL. Only the *internal* override — used for all
8+
# service-to-service communication inside the workload namespace — is updated
9+
# to point at the Skupper Listener.
10+
#
11+
# After the OSCP is updated, this playbook also ensures that EDPM compute
12+
# nodes can resolve the Skupper Keystone virtual service name. The Skupper
13+
# Listener creates a ClusterIP-only Service (keystone-regionone) that is not
14+
# reachable from outside the OCP cluster. EDPM nodes use the dnsmasq
15+
# LoadBalancer Service (in the leaf namespace) as their DNS server and require
16+
# a resolvable hostname for the Keystone auth_url configured in nova.conf.
17+
# To bridge this gap, the playbook:
18+
# 1. Creates a dedicated MetalLB LoadBalancer Service that selects the
19+
# Skupper router pod and exposes port 5000 on the leaf internalapi network.
20+
# 2. Creates a DNSData CR so dnsmasq resolves both the short (.svc) and
21+
# fully-qualified (.svc.cluster.local) names to the LoadBalancer IP.
22+
#
23+
# This playbook is a no-op when cifmw_skupper_keystone_enabled is false. In
24+
# that case, skupper-keystone-connector.yaml (pre_stage hook) has already
25+
# written the public Keystone URL into skmo-values.yaml so the leaf OSCP is
26+
# created with the public endpoint — no Skupper tunnel is used and there is no
27+
# matching Connector for the Skupper Listener, so the OSCP must not be patched
28+
# to point at the Skupper virtual Service.
29+
#
30+
# Run skupper-keystone-connector.yaml and skupper-keystone-listener.yaml
31+
# before this playbook so that the Skupper virtual service is in place.
32+
#
33+
# Variables:
34+
# cifmw_skupper_keystone_enabled (default: true)
35+
# When false, all tasks in this playbook are skipped.
36+
# cifmw_skupper_leaf_namespace (default: openstack2)
37+
# cifmw_skupper_keystone_listener_host (default: keystone-regionone)
38+
# cifmw_skupper_keystone_port (default: 5000)
39+
# cifmw_skupper_keystone_metallb_pool (default: internalapi2)
40+
# MetalLB address-pool name for the leaf internalapi network. An IP is
41+
# auto-assigned from this pool; no static address is required.
42+
- name: Configure leaf region to use Skupper Keystone internal endpoint
43+
hosts: localhost
44+
gather_facts: false
45+
vars:
46+
cifmw_skupper_keystone_enabled: true
47+
cifmw_skupper_leaf_namespace: openstack2
48+
cifmw_skupper_keystone_listener_host: keystone-regionone
49+
cifmw_skupper_keystone_port: 5000
50+
cifmw_skupper_keystone_metallb_pool: internalapi2
51+
tasks:
52+
- name: Skip all tasks when Skupper Keystone routing is disabled
53+
ansible.builtin.meta: end_play
54+
when: not cifmw_skupper_keystone_enabled | bool
55+
56+
- name: Build the Skupper Keystone internal URL
57+
ansible.builtin.set_fact:
58+
_skupper_keystone_internal_url: >-
59+
https://{{ cifmw_skupper_keystone_listener_host }}.{{ cifmw_skupper_leaf_namespace }}.svc.cluster.local:{{ cifmw_skupper_keystone_port }}
60+
61+
- name: Patch leaf OSCP internal Keystone override to use Skupper endpoint
62+
# This switches the internal keystone endpoint URL from the central
63+
# region's public URL to the Skupper Listener virtual service. The
64+
# public endpoint override is not touched.
65+
kubernetes.core.k8s:
66+
state: patched
67+
api_version: core.openstack.org/v1beta1
68+
kind: OpenStackControlPlane
69+
name: controlplane
70+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
71+
definition:
72+
spec:
73+
keystone:
74+
template:
75+
override:
76+
service:
77+
internal:
78+
endpointURL: "{{ _skupper_keystone_internal_url }}"
79+
80+
- name: Wait for leaf OSCP to reconcile after Keystone endpoint change
81+
kubernetes.core.k8s_info:
82+
api_version: core.openstack.org/v1beta1
83+
kind: OpenStackControlPlane
84+
name: controlplane
85+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
86+
register: _leaf_oscp
87+
retries: 60
88+
delay: 30
89+
until:
90+
- _leaf_oscp.resources | length > 0
91+
- _leaf_oscp.resources[0].status is defined
92+
- _leaf_oscp.resources[0].status.conditions is defined
93+
- _leaf_oscp.resources[0].status.conditions |
94+
selectattr('type', 'equalto', 'Ready') |
95+
selectattr('status', 'equalto', 'True') | list | length > 0
96+
97+
- name: Create LoadBalancer service to expose Skupper Keystone for EDPM nodes
98+
# The Skupper Listener creates a ClusterIP-only Service that EDPM nodes
99+
# outside the OCP cluster cannot reach. This LoadBalancer Service selects
100+
# the same Skupper router pod and obtains a MetalLB IP on the leaf
101+
# internalapi network, making port 5000 reachable from EDPM compute nodes.
102+
kubernetes.core.k8s:
103+
state: present
104+
definition:
105+
apiVersion: v1
106+
kind: Service
107+
metadata:
108+
name: "{{ cifmw_skupper_keystone_listener_host }}-lb"
109+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
110+
annotations:
111+
metallb.universe.tf/address-pool: "{{ cifmw_skupper_keystone_metallb_pool }}"
112+
spec:
113+
type: LoadBalancer
114+
selector:
115+
application: skupper-router
116+
skupper.io/component: router
117+
ports:
118+
- name: keystone-internal
119+
port: "{{ cifmw_skupper_keystone_port | int }}"
120+
protocol: TCP
121+
targetPort: 1024
122+
123+
- name: Wait for MetalLB to assign an external IP to the keystone LoadBalancer
124+
kubernetes.core.k8s_info:
125+
api_version: v1
126+
kind: Service
127+
name: "{{ cifmw_skupper_keystone_listener_host }}-lb"
128+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
129+
register: _keystone_lb_svc
130+
retries: 12
131+
delay: 10
132+
until:
133+
- _keystone_lb_svc.resources | length > 0
134+
- _keystone_lb_svc.resources[0].status.loadBalancer.ingress is defined
135+
- _keystone_lb_svc.resources[0].status.loadBalancer.ingress | length > 0
136+
137+
- name: Set keystone LoadBalancer IP fact
138+
ansible.builtin.set_fact:
139+
_keystone_lb_ip: >-
140+
{{ _keystone_lb_svc.resources[0].status.loadBalancer.ingress[0].ip }}
141+
142+
- name: Create DNSData entry for Skupper Keystone endpoint
143+
# Adds both the short (.svc) and fully-qualified (.svc.cluster.local)
144+
# names to the dnsmasq instance serving EDPM nodes, so that nova-compute
145+
# auth_url lookups resolve to the LoadBalancer IP above.
146+
kubernetes.core.k8s:
147+
state: present
148+
definition:
149+
apiVersion: network.openstack.org/v1beta1
150+
kind: DNSData
151+
metadata:
152+
name: keystone-skupper
153+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
154+
spec:
155+
dnsDataLabelSelectorValue: dnsdata
156+
hosts:
157+
- hostnames:
158+
- "{{ cifmw_skupper_keystone_listener_host }}.{{ cifmw_skupper_leaf_namespace }}.svc"
159+
- "{{ cifmw_skupper_keystone_listener_host }}.{{ cifmw_skupper_leaf_namespace }}.svc.cluster.local"
160+
ip: "{{ _keystone_lb_ip }}"

‎hooks/playbooks/skmo/configure-leaf-listener.yaml‎

Lines changed: 41 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,64 @@
11
---
2-
- name: Patch leaf control plane with barbican-keystone-listener transport URL
2+
# Configure the leaf barbican-keystone-listener to use the Skupper
3+
# application network for cross-region RabbitMQ access.
4+
#
5+
# In the leaf region:
6+
# - Read the RabbitMQ credentials from the dedicated user credentials secret
7+
# created by the RabbitMQ operator when the TransportURL CR is reconciled.
8+
# - Patch barbicanKeystoneListener to connect to the central RabbitMQ via the
9+
# Skupper Listener endpoint using those credentials and its own pool_name.
10+
#
11+
# Variables:
12+
# cifmw_skupper_central_namespace (default: openstack)
13+
# cifmw_skupper_leaf_namespace (default: openstack2)
14+
# cifmw_skupper_listener_host (default: rabbitmq-regionone)
15+
# Must match the host set in skupper-listener.yaml.
16+
# cifmw_skupper_rabbitmq_port (default: 5671)
17+
# cifmw_skupper_transport_url_name (default: barbican-keystone-listener-regiontwo)
18+
# Name of the TransportURL CR created in prepare-leaf.yaml. The operator
19+
# creates a user credentials secret named:
20+
# rabbitmq-user-<name>-<username>-user
21+
# cifmw_skupper_transport_url_username (default: barbican-keystone-listener-regiontwo)
22+
# Must match the username field set on the TransportURL CR in prepare-leaf.yaml.
23+
- name: Configure barbican-keystone-listener to use Skupper for cross-region RabbitMQ
324
hosts: localhost
425
gather_facts: false
526
vars:
6-
central_namespace: openstack
7-
leaf_namespace: openstack2
8-
leaf_transport_url_name: rabbitmq-transport-url-barbican-keystone-listener-regiontwo
27+
cifmw_skupper_central_namespace: openstack
28+
cifmw_skupper_leaf_namespace: openstack2
29+
cifmw_skupper_listener_host: rabbitmq-regionone
30+
cifmw_skupper_rabbitmq_port: 5671
31+
cifmw_skupper_transport_url_name: barbican-keystone-listener-regiontwo
32+
cifmw_skupper_transport_url_username: barbican-keystone-listener-regiontwo
933
tasks:
10-
- name: Get transport URL secret from central namespace
34+
- name: Get RabbitMQ user credentials secret for leaf listener
35+
# The RabbitMQ operator creates a secret named
36+
# rabbitmq-user-<transport-url-name>-<username>-user that contains
37+
# the username and password fields for the dedicated RabbitMQ user.
1138
kubernetes.core.k8s_info:
1239
api_version: v1
1340
kind: Secret
14-
namespace: "{{ central_namespace }}"
15-
name: "{{ leaf_transport_url_name }}"
16-
register: _transport_secret
41+
namespace: "{{ cifmw_skupper_central_namespace }}"
42+
name: "rabbitmq-user-{{ cifmw_skupper_transport_url_name }}-{{ cifmw_skupper_transport_url_username }}-user"
43+
register: _rabbitmq_user_secret
1744

18-
- name: Patch OpenStackControlPlane in leaf region with notifications transport_url
45+
- name: Patch leaf barbicanKeystoneListener to use Skupper RabbitMQ endpoint
1946
vars:
20-
_transport_url: "{{ _transport_secret.resources[0].data['transport_url'] | b64decode }}"
47+
_username: "{{ _rabbitmq_user_secret.resources[0].data['username'] | b64decode }}"
48+
_password: "{{ _rabbitmq_user_secret.resources[0].data['password'] | b64decode }}"
2149
kubernetes.core.k8s:
2250
state: patched
2351
api_version: core.openstack.org/v1beta1
2452
kind: OpenStackControlPlane
2553
name: controlplane
26-
namespace: "{{ leaf_namespace }}"
54+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
2755
definition:
2856
spec:
2957
barbican:
3058
template:
3159
barbicanKeystoneListener:
3260
customServiceConfig: |
3361
[DEFAULT]
34-
transport_url = {{ _transport_url }}
62+
transport_url = rabbit://{{ _username }}:{{ _password }}@{{ cifmw_skupper_listener_host }}:{{ cifmw_skupper_rabbitmq_port }}/?ssl=1
3563
[keystone_notifications]
36-
pool_name = barbican-listener-regionTwo
64+
pool_name = barbican-listener-regiontwo

‎hooks/playbooks/skmo/prepare-leaf.yaml‎

Lines changed: 2 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@
1010
central_rootca_secret: rootca-public
1111
central_rootca_internal_secret: rootca-internal
1212
leaf_transport_url_name: barbican-keystone-listener-regiontwo
13+
leaf_transport_url_username: barbican-keystone-listener-regiontwo
1314
leaf_transport_url_name_secret: rabbitmq-transport-url-barbican-keystone-listener-regiontwo
14-
leaf_transport_url_secret_copy: barbican-keystone-listener-regiontwo-transport
1515
tasks:
1616
- name: Wait for central Keystone API to be ready
1717
kubernetes.core.k8s_info:
@@ -159,6 +159,7 @@
159159
namespace: "{{ central_namespace }}"
160160
spec:
161161
rabbitmqClusterName: rabbitmq
162+
username: "{{ leaf_transport_url_username }}"
162163

163164
- name: Wait for TransportURL to be ready
164165
kubernetes.core.k8s_info:
@@ -176,23 +177,3 @@
176177
- _transport_url_info.resources[0].status.conditions |
177178
selectattr('type', 'equalto', 'Ready') |
178179
selectattr('status', 'equalto', 'True') | list | length > 0
179-
180-
- name: Get transport URL secret from central namespace
181-
kubernetes.core.k8s_info:
182-
api_version: v1
183-
kind: Secret
184-
namespace: "{{ central_namespace }}"
185-
name: "{{ leaf_transport_url_name_secret }}"
186-
register: _transport_secret
187-
188-
- name: Copy transport URL secret to leaf namespace
189-
kubernetes.core.k8s:
190-
state: present
191-
definition:
192-
apiVersion: v1
193-
kind: Secret
194-
metadata:
195-
name: "{{ leaf_transport_url_secret_copy }}"
196-
namespace: "{{ leaf_namespace }}"
197-
type: "{{ _transport_secret.resources[0].type }}"
198-
data: "{{ _transport_secret.resources[0].data }}"

‎hooks/playbooks/skmo/skupper-connector-tasks.yaml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
# Shared task file: Create a Skupper Connector CR and wait for it to be
3+
# Configured. Include this from service-specific connector playbooks after
4+
# the TLS credentials have been discovered and stored in the variables below.
5+
#
6+
# Expected variables (set via include_tasks vars: block):
7+
# _cifmw_connector_name Skupper Connector CR name
8+
# _cifmw_connector_namespace Namespace for the Connector
9+
# _cifmw_connector_routing_key Skupper routing key (must match Listener)
10+
# _cifmw_connector_host Backend service hostname
11+
# _cifmw_connector_port Backend service port
12+
# _cifmw_connector_tls_credentials Name of the TLS Secret for the backend
13+
# _cifmw_connector_verify_hostname Whether Skupper verifies the backend cert hostname
14+
# _cifmw_connector_ignore_wait_errors Whether to ignore wait failures
15+
16+
- name: Create Skupper Connector
17+
kubernetes.core.k8s:
18+
state: present
19+
definition:
20+
apiVersion: skupper.io/v2alpha1
21+
kind: Connector
22+
metadata:
23+
name: "{{ _cifmw_connector_name }}"
24+
namespace: "{{ _cifmw_connector_namespace }}"
25+
spec:
26+
routingKey: "{{ _cifmw_connector_routing_key }}"
27+
host: "{{ _cifmw_connector_host }}"
28+
port: "{{ _cifmw_connector_port }}"
29+
type: tcp
30+
tlsCredentials: "{{ _cifmw_connector_tls_credentials }}"
31+
verifyHostname: "{{ _cifmw_connector_verify_hostname }}"
32+
33+
- name: Wait for Skupper Connector to be configured
34+
# A Connector shows "Ready" only after a matching Listener is deployed in
35+
# the remote namespace. Waiting for "Configured" is sufficient here.
36+
ignore_errors: "{{ _cifmw_connector_ignore_wait_errors | bool }}"
37+
kubernetes.core.k8s_info:
38+
api_version: skupper.io/v2alpha1
39+
kind: Connector
40+
name: "{{ _cifmw_connector_name }}"
41+
namespace: "{{ _cifmw_connector_namespace }}"
42+
register: _connector
43+
retries: 30
44+
delay: 10
45+
until:
46+
- _connector.resources | length > 0
47+
- _connector.resources[0].status is defined
48+
- _connector.resources[0].status.conditions is defined
49+
- _connector.resources[0].status.conditions |
50+
selectattr('type', 'equalto', 'Configured') |
51+
selectattr('status', 'equalto', 'True') | list | length > 0

‎hooks/playbooks/skmo/skupper-connector.yaml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
# Create a Skupper Connector in the central namespace that exposes the central
3+
# RabbitMQ service to workload regions over the Skupper application network.
4+
#
5+
# The TLS Secret name is auto-discovered from the RabbitmqCluster CR.
6+
# Connector creation and the wait for Configured status are handled by the
7+
# shared skupper-connector-tasks.yaml task file.
8+
#
9+
# Variables:
10+
# cifmw_skupper_central_namespace (default: openstack)
11+
# cifmw_skupper_routing_key (default: rabbit-keystone)
12+
# cifmw_skupper_rabbitmq_port (default: 5671)
13+
- name: Create Skupper Connector for central RabbitMQ
14+
hosts: localhost
15+
gather_facts: false
16+
vars:
17+
cifmw_skupper_central_namespace: openstack
18+
cifmw_skupper_routing_key: rabbit-keystone
19+
cifmw_skupper_rabbitmq_port: 5671
20+
tasks:
21+
- name: Get RabbitMQ TLS certificate secret name from RabbitMq CR
22+
# The OpenStack infra-operator uses rabbitmq.openstack.org/v1beta1 (RabbitMq),
23+
# not the community rabbitmq.com/v1beta1 (RabbitmqCluster).
24+
kubernetes.core.k8s_info:
25+
api_version: rabbitmq.openstack.org/v1beta1
26+
kind: RabbitMq
27+
name: rabbitmq
28+
namespace: "{{ cifmw_skupper_central_namespace }}"
29+
register: _rabbitmq_cluster
30+
retries: 30
31+
delay: 10
32+
until:
33+
- _rabbitmq_cluster.resources | length > 0
34+
- _rabbitmq_cluster.resources[0].spec.tls is defined
35+
- _rabbitmq_cluster.resources[0].spec.tls.secretName is defined
36+
- _rabbitmq_cluster.resources[0].spec.tls.secretName | length > 0
37+
38+
- name: Create Skupper Connector and wait for Configured
39+
ansible.builtin.include_tasks: skupper-connector-tasks.yaml
40+
vars:
41+
_cifmw_connector_name: rabbitmq-keystone
42+
_cifmw_connector_namespace: "{{ cifmw_skupper_central_namespace }}"
43+
_cifmw_connector_routing_key: "{{ cifmw_skupper_routing_key }}"
44+
_cifmw_connector_host: "rabbitmq.{{ cifmw_skupper_central_namespace }}.svc.cluster.local"
45+
_cifmw_connector_port: "{{ cifmw_skupper_rabbitmq_port }}"
46+
_cifmw_connector_tls_credentials: "{{ _rabbitmq_cluster.resources[0].spec.tls.secretName }}"
47+
_cifmw_connector_verify_hostname: true
48+
_cifmw_connector_ignore_wait_errors: false

0 commit comments

Comments
 (0)