Skip to content

Commit d3f6a8e

Browse files
vakwetuclaude
vakwetu
and
claude
committed
[skmo] Add Skupper for cross-region RabbitMQ and Keystone internal routing
Add hook playbooks and configuration to establish Skupper virtual services for RabbitMQ and Keystone internal endpoints, enabling cross-region connectivity in the multi-namespace SKMO scenario. skupper-connector.yaml: query the RabbitMQ TLS secret from the correct CRD - rabbitmq.openstack.org/v1beta1 (RabbitMq) as used by the OpenStack infra-operator, not the community rabbitmq.com/v1beta1 (RabbitmqCluster). Add retries to wait for spec.tls.secretName to be populated before creating the Skupper Connector. skupper-keystone-connector.yaml: add retries to the KeystoneAPI CR lookup to wait for spec.tls.api.internal.secretName to be available, since that field is not populated until Keystone completes TLS setup. configure-leaf-keystone-internal.yaml: after patching the leaf OSCP to use the Skupper Keystone virtual service, also create a MetalLB LoadBalancer Service (keystone-regionone-lb) and a DNSData CR (keystone-skupper) so that EDPM compute nodes outside the OCP cluster can resolve and connect to the Keystone auth_url. The Skupper Listener creates a ClusterIP-only Service that is unreachable from EDPM nodes; the LoadBalancer Service obtains a MetalLB IP on the leaf internalapi network and the DNSData entry registers both the short (.svc) and fully-qualified (.svc.cluster.local) names in the dnsmasq instance serving those nodes. Signed-off-by: Ade Lee <alee@redhat.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent d4ae12d commit d3f6a8e

9 files changed

Lines changed: 764 additions & 34 deletions

‎hooks/playbooks/skmo/configure-leaf-keystone-internal.yaml‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
---
2+
# Patch the leaf/workload region OpenStackControlPlane to use the Skupper
3+
# Listener virtual endpoint for internal Keystone authentication traffic.
4+
#
5+
# The public endpoint override is left unchanged so that end-user traffic and
6+
# the Keystone service catalog continue to reference the central region's
7+
# external (public) URL. Only the *internal* override — used for all
8+
# service-to-service communication inside the workload namespace — is updated
9+
# to point at the Skupper Listener.
10+
#
11+
# After the OSCP is updated, this playbook also ensures that EDPM compute
12+
# nodes can resolve the Skupper Keystone virtual service name. The Skupper
13+
# Listener creates a ClusterIP-only Service (keystone-regionone) that is not
14+
# reachable from outside the OCP cluster. EDPM nodes use the dnsmasq
15+
# LoadBalancer Service (in the leaf namespace) as their DNS server and require
16+
# a resolvable hostname for the Keystone auth_url configured in nova.conf.
17+
# To bridge this gap, the playbook:
18+
# 1. Creates a dedicated MetalLB LoadBalancer Service that selects the
19+
# Skupper router pod and exposes port 5000 on the leaf internalapi network.
20+
# 2. Creates a DNSData CR so dnsmasq resolves both the short (.svc) and
21+
# fully-qualified (.svc.cluster.local) names to the LoadBalancer IP.
22+
#
23+
# Run skupper-keystone-connector.yaml and skupper-keystone-listener.yaml
24+
# before this playbook so that the Skupper virtual service is in place.
25+
#
26+
# Variables:
27+
# cifmw_skupper_leaf_namespace (default: openstack2)
28+
# cifmw_skupper_keystone_listener_host (default: keystone-regionone)
29+
# cifmw_skupper_keystone_port (default: 5000)
30+
# cifmw_skupper_keystone_metallb_pool (default: internalapi2)
31+
# MetalLB address-pool name for the leaf internalapi network. An IP is
32+
# auto-assigned from this pool; no static address is required.
33+
- name: Configure leaf region to use Skupper Keystone internal endpoint
34+
hosts: localhost
35+
gather_facts: false
36+
vars:
37+
cifmw_skupper_leaf_namespace: openstack2
38+
cifmw_skupper_keystone_listener_host: keystone-regionone
39+
cifmw_skupper_keystone_port: 5000
40+
cifmw_skupper_keystone_metallb_pool: internalapi2
41+
tasks:
42+
- name: Build the Skupper Keystone internal URL
43+
ansible.builtin.set_fact:
44+
_skupper_keystone_internal_url: >-
45+
https://{{ cifmw_skupper_keystone_listener_host }}.{{ cifmw_skupper_leaf_namespace }}.svc.cluster.local:{{ cifmw_skupper_keystone_port }}
46+
47+
- name: Patch leaf OSCP internal Keystone override to use Skupper endpoint
48+
# This switches the internal keystone endpoint URL from the central
49+
# region's public URL to the Skupper Listener virtual service. The
50+
# public endpoint override is not touched.
51+
kubernetes.core.k8s:
52+
state: patched
53+
api_version: core.openstack.org/v1beta1
54+
kind: OpenStackControlPlane
55+
name: controlplane
56+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
57+
definition:
58+
spec:
59+
keystone:
60+
template:
61+
override:
62+
service:
63+
internal:
64+
endpointURL: "{{ _skupper_keystone_internal_url }}"
65+
66+
- name: Wait for leaf OSCP to reconcile after Keystone endpoint change
67+
kubernetes.core.k8s_info:
68+
api_version: core.openstack.org/v1beta1
69+
kind: OpenStackControlPlane
70+
name: controlplane
71+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
72+
register: _leaf_oscp
73+
retries: 60
74+
delay: 30
75+
until:
76+
- _leaf_oscp.resources | length > 0
77+
- _leaf_oscp.resources[0].status is defined
78+
- _leaf_oscp.resources[0].status.conditions is defined
79+
- _leaf_oscp.resources[0].status.conditions |
80+
selectattr('type', 'equalto', 'Ready') |
81+
selectattr('status', 'equalto', 'True') | list | length > 0
82+
83+
- name: Create LoadBalancer service to expose Skupper Keystone for EDPM nodes
84+
# The Skupper Listener creates a ClusterIP-only Service that EDPM nodes
85+
# outside the OCP cluster cannot reach. This LoadBalancer Service selects
86+
# the same Skupper router pod and obtains a MetalLB IP on the leaf
87+
# internalapi network, making port 5000 reachable from EDPM compute nodes.
88+
kubernetes.core.k8s:
89+
state: present
90+
definition:
91+
apiVersion: v1
92+
kind: Service
93+
metadata:
94+
name: "{{ cifmw_skupper_keystone_listener_host }}-lb"
95+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
96+
annotations:
97+
metallb.universe.tf/address-pool: "{{ cifmw_skupper_keystone_metallb_pool }}"
98+
spec:
99+
type: LoadBalancer
100+
selector:
101+
application: skupper-router
102+
skupper.io/component: router
103+
ports:
104+
- name: keystone-internal
105+
port: "{{ cifmw_skupper_keystone_port | int }}"
106+
protocol: TCP
107+
targetPort: 1024
108+
109+
- name: Wait for MetalLB to assign an external IP to the keystone LoadBalancer
110+
kubernetes.core.k8s_info:
111+
api_version: v1
112+
kind: Service
113+
name: "{{ cifmw_skupper_keystone_listener_host }}-lb"
114+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
115+
register: _keystone_lb_svc
116+
retries: 12
117+
delay: 10
118+
until:
119+
- _keystone_lb_svc.resources | length > 0
120+
- _keystone_lb_svc.resources[0].status.loadBalancer.ingress is defined
121+
- _keystone_lb_svc.resources[0].status.loadBalancer.ingress | length > 0
122+
123+
- name: Set keystone LoadBalancer IP fact
124+
ansible.builtin.set_fact:
125+
_keystone_lb_ip: >-
126+
{{ _keystone_lb_svc.resources[0].status.loadBalancer.ingress[0].ip }}
127+
128+
- name: Create DNSData entry for Skupper Keystone endpoint
129+
# Adds both the short (.svc) and fully-qualified (.svc.cluster.local)
130+
# names to the dnsmasq instance serving EDPM nodes, so that nova-compute
131+
# auth_url lookups resolve to the LoadBalancer IP above.
132+
kubernetes.core.k8s:
133+
state: present
134+
definition:
135+
apiVersion: network.openstack.org/v1beta1
136+
kind: DNSData
137+
metadata:
138+
name: keystone-skupper
139+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
140+
spec:
141+
dnsDataLabelSelectorValue: dnsdata
142+
hosts:
143+
- hostnames:
144+
- "{{ cifmw_skupper_keystone_listener_host }}.{{ cifmw_skupper_leaf_namespace }}.svc"
145+
- "{{ cifmw_skupper_keystone_listener_host }}.{{ cifmw_skupper_leaf_namespace }}.svc.cluster.local"
146+
ip: "{{ _keystone_lb_ip }}"

‎hooks/playbooks/skmo/configure-leaf-listener.yaml‎

Lines changed: 41 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,64 @@
11
---
2-
- name: Patch leaf control plane with barbican-keystone-listener transport URL
2+
# Configure the leaf barbican-keystone-listener to use the Skupper
3+
# application network for cross-region RabbitMQ access.
4+
#
5+
# In the leaf region:
6+
# - Read the RabbitMQ credentials from the dedicated user credentials secret
7+
# created by the RabbitMQ operator when the TransportURL CR is reconciled.
8+
# - Patch barbicanKeystoneListener to connect to the central RabbitMQ via the
9+
# Skupper Listener endpoint using those credentials and its own pool_name.
10+
#
11+
# Variables:
12+
# cifmw_skupper_central_namespace (default: openstack)
13+
# cifmw_skupper_leaf_namespace (default: openstack2)
14+
# cifmw_skupper_listener_host (default: rabbitmq-regionone)
15+
# Must match the host set in skupper-listener.yaml.
16+
# cifmw_skupper_rabbitmq_port (default: 5671)
17+
# cifmw_skupper_transport_url_name (default: barbican-keystone-listener-regiontwo)
18+
# Name of the TransportURL CR created in prepare-leaf.yaml. The operator
19+
# creates a user credentials secret named:
20+
# rabbitmq-user-<name>-<username>-user
21+
# cifmw_skupper_transport_url_username (default: barbican-keystone-listener-regiontwo)
22+
# Must match the username field set on the TransportURL CR in prepare-leaf.yaml.
23+
- name: Configure barbican-keystone-listener to use Skupper for cross-region RabbitMQ
324
hosts: localhost
425
gather_facts: false
526
vars:
6-
central_namespace: openstack
7-
leaf_namespace: openstack2
8-
leaf_transport_url_name: rabbitmq-transport-url-barbican-keystone-listener-regiontwo
27+
cifmw_skupper_central_namespace: openstack
28+
cifmw_skupper_leaf_namespace: openstack2
29+
cifmw_skupper_listener_host: rabbitmq-regionone
30+
cifmw_skupper_rabbitmq_port: 5671
31+
cifmw_skupper_transport_url_name: barbican-keystone-listener-regiontwo
32+
cifmw_skupper_transport_url_username: barbican-keystone-listener-regiontwo
933
tasks:
10-
- name: Get transport URL secret from central namespace
34+
- name: Get RabbitMQ user credentials secret for leaf listener
35+
# The RabbitMQ operator creates a secret named
36+
# rabbitmq-user-<transport-url-name>-<username>-user that contains
37+
# the username and password fields for the dedicated RabbitMQ user.
1138
kubernetes.core.k8s_info:
1239
api_version: v1
1340
kind: Secret
14-
namespace: "{{ central_namespace }}"
15-
name: "{{ leaf_transport_url_name }}"
16-
register: _transport_secret
41+
namespace: "{{ cifmw_skupper_central_namespace }}"
42+
name: "rabbitmq-user-{{ cifmw_skupper_transport_url_name }}-{{ cifmw_skupper_transport_url_username }}-user"
43+
register: _rabbitmq_user_secret
1744

18-
- name: Patch OpenStackControlPlane in leaf region with notifications transport_url
45+
- name: Patch leaf barbicanKeystoneListener to use Skupper RabbitMQ endpoint
1946
vars:
20-
_transport_url: "{{ _transport_secret.resources[0].data['transport_url'] | b64decode }}"
47+
_username: "{{ _rabbitmq_user_secret.resources[0].data['username'] | b64decode }}"
48+
_password: "{{ _rabbitmq_user_secret.resources[0].data['password'] | b64decode }}"
2149
kubernetes.core.k8s:
2250
state: patched
2351
api_version: core.openstack.org/v1beta1
2452
kind: OpenStackControlPlane
2553
name: controlplane
26-
namespace: "{{ leaf_namespace }}"
54+
namespace: "{{ cifmw_skupper_leaf_namespace }}"
2755
definition:
2856
spec:
2957
barbican:
3058
template:
3159
barbicanKeystoneListener:
3260
customServiceConfig: |
3361
[DEFAULT]
34-
transport_url = {{ _transport_url }}
62+
transport_url = rabbit://{{ _username }}:{{ _password }}@{{ cifmw_skupper_listener_host }}:{{ cifmw_skupper_rabbitmq_port }}/?ssl=1
3563
[keystone_notifications]
36-
pool_name = barbican-listener-regionTwo
64+
pool_name = barbican-listener-regiontwo

‎hooks/playbooks/skmo/prepare-leaf.yaml‎

Lines changed: 2 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@
1010
central_rootca_secret: rootca-public
1111
central_rootca_internal_secret: rootca-internal
1212
leaf_transport_url_name: barbican-keystone-listener-regiontwo
13+
leaf_transport_url_username: barbican-keystone-listener-regiontwo
1314
leaf_transport_url_name_secret: rabbitmq-transport-url-barbican-keystone-listener-regiontwo
14-
leaf_transport_url_secret_copy: barbican-keystone-listener-regiontwo-transport
1515
tasks:
1616
- name: Wait for central Keystone API to be ready
1717
kubernetes.core.k8s_info:
@@ -159,6 +159,7 @@
159159
namespace: "{{ central_namespace }}"
160160
spec:
161161
rabbitmqClusterName: rabbitmq
162+
username: "{{ leaf_transport_url_username }}"
162163

163164
- name: Wait for TransportURL to be ready
164165
kubernetes.core.k8s_info:
@@ -176,23 +177,3 @@
176177
- _transport_url_info.resources[0].status.conditions |
177178
selectattr('type', 'equalto', 'Ready') |
178179
selectattr('status', 'equalto', 'True') | list | length > 0
179-
180-
- name: Get transport URL secret from central namespace
181-
kubernetes.core.k8s_info:
182-
api_version: v1
183-
kind: Secret
184-
namespace: "{{ central_namespace }}"
185-
name: "{{ leaf_transport_url_name_secret }}"
186-
register: _transport_secret
187-
188-
- name: Copy transport URL secret to leaf namespace
189-
kubernetes.core.k8s:
190-
state: present
191-
definition:
192-
apiVersion: v1
193-
kind: Secret
194-
metadata:
195-
name: "{{ leaf_transport_url_secret_copy }}"
196-
namespace: "{{ leaf_namespace }}"
197-
type: "{{ _transport_secret.resources[0].type }}"
198-
data: "{{ _transport_secret.resources[0].data }}"

‎hooks/playbooks/skmo/skupper-connector-tasks.yaml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
# Shared task file: Create a Skupper Connector CR and wait for it to be
3+
# Configured. Include this from service-specific connector playbooks after
4+
# the TLS credentials have been discovered and stored in the variables below.
5+
#
6+
# Expected variables (set via include_tasks vars: block):
7+
# _cifmw_connector_name Skupper Connector CR name
8+
# _cifmw_connector_namespace Namespace for the Connector
9+
# _cifmw_connector_routing_key Skupper routing key (must match Listener)
10+
# _cifmw_connector_host Backend service hostname
11+
# _cifmw_connector_port Backend service port
12+
# _cifmw_connector_tls_credentials Name of the TLS Secret for the backend
13+
# _cifmw_connector_verify_hostname Whether Skupper verifies the backend cert hostname
14+
# _cifmw_connector_ignore_wait_errors Whether to ignore wait failures
15+
16+
- name: Create Skupper Connector
17+
kubernetes.core.k8s:
18+
state: present
19+
definition:
20+
apiVersion: skupper.io/v2alpha1
21+
kind: Connector
22+
metadata:
23+
name: "{{ _cifmw_connector_name }}"
24+
namespace: "{{ _cifmw_connector_namespace }}"
25+
spec:
26+
routingKey: "{{ _cifmw_connector_routing_key }}"
27+
host: "{{ _cifmw_connector_host }}"
28+
port: "{{ _cifmw_connector_port }}"
29+
type: tcp
30+
tlsCredentials: "{{ _cifmw_connector_tls_credentials }}"
31+
verifyHostname: "{{ _cifmw_connector_verify_hostname }}"
32+
33+
- name: Wait for Skupper Connector to be configured
34+
# A Connector shows "Ready" only after a matching Listener is deployed in
35+
# the remote namespace. Waiting for "Configured" is sufficient here.
36+
ignore_errors: "{{ _cifmw_connector_ignore_wait_errors | bool }}"
37+
kubernetes.core.k8s_info:
38+
api_version: skupper.io/v2alpha1
39+
kind: Connector
40+
name: "{{ _cifmw_connector_name }}"
41+
namespace: "{{ _cifmw_connector_namespace }}"
42+
register: _connector
43+
retries: 30
44+
delay: 10
45+
until:
46+
- _connector.resources | length > 0
47+
- _connector.resources[0].status is defined
48+
- _connector.resources[0].status.conditions is defined
49+
- _connector.resources[0].status.conditions |
50+
selectattr('type', 'equalto', 'Configured') |
51+
selectattr('status', 'equalto', 'True') | list | length > 0

‎hooks/playbooks/skmo/skupper-connector.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
# Create a Skupper Connector in the central namespace that exposes the central
3+
# RabbitMQ service to workload regions over the Skupper application network.
4+
#
5+
# The TLS Secret name is auto-discovered from the RabbitmqCluster CR.
6+
# Connector creation and the wait for Configured status are handled by the
7+
# shared skupper-connector-tasks.yaml task file.
8+
#
9+
# Variables:
10+
# cifmw_skupper_central_namespace (default: openstack)
11+
# cifmw_skupper_routing_key (default: rabbit-keystone)
12+
# cifmw_skupper_rabbitmq_port (default: 5671)
13+
- name: Create Skupper Connector for central RabbitMQ
14+
hosts: localhost
15+
gather_facts: false
16+
vars:
17+
cifmw_skupper_central_namespace: openstack
18+
cifmw_skupper_routing_key: rabbit-keystone
19+
cifmw_skupper_rabbitmq_port: 5671
20+
tasks:
21+
- name: Get RabbitMQ TLS certificate secret name from RabbitMq CR
22+
# The OpenStack infra-operator uses rabbitmq.openstack.org/v1beta1 (RabbitMq),
23+
# not the community rabbitmq.com/v1beta1 (RabbitmqCluster).
24+
kubernetes.core.k8s_info:
25+
api_version: rabbitmq.openstack.org/v1beta1
26+
kind: RabbitMq
27+
name: rabbitmq
28+
namespace: "{{ cifmw_skupper_central_namespace }}"
29+
register: _rabbitmq_cluster
30+
retries: 30
31+
delay: 10
32+
until:
33+
- _rabbitmq_cluster.resources | length > 0
34+
- _rabbitmq_cluster.resources[0].spec.tls is defined
35+
- _rabbitmq_cluster.resources[0].spec.tls.secretName is defined
36+
37+
- name: Create Skupper Connector and wait for Configured
38+
ansible.builtin.include_tasks: skupper-connector-tasks.yaml
39+
vars:
40+
_cifmw_connector_name: rabbitmq-keystone
41+
_cifmw_connector_namespace: "{{ cifmw_skupper_central_namespace }}"
42+
_cifmw_connector_routing_key: "{{ cifmw_skupper_routing_key }}"
43+
_cifmw_connector_host: "rabbitmq.{{ cifmw_skupper_central_namespace }}.svc.cluster.local"
44+
_cifmw_connector_port: "{{ cifmw_skupper_rabbitmq_port }}"
45+
_cifmw_connector_tls_credentials: "{{ _rabbitmq_cluster.resources[0].spec.tls.secretName }}"
46+
_cifmw_connector_verify_hostname: true
47+
_cifmw_connector_ignore_wait_errors: false

0 commit comments

Comments
 (0)