Systemd %S breaking changes in Rocky Linux 9.8

[DavidePrincipi]

Newer release of Systemd RPM in Rocky Linux and Alma Linux 9.8 is not backward-compatible in the way it expands %S.

Expansion of %S changed from %h/.config to %h/.local/state, breaking many Systemd units.

Refs https://git.rockylinux.org/staging/src-rhel/rpms/systemd/-/blob/c9s/SOURCES/1313-execute-add-support-for-XDG_STATE_HOME-for-placing-s.patch?ref_type=heads

Steps to reproduce

• Update base distro to release 9.8:

   dnf --enablerepo=baseos,appstream --disablerepo=ns-baseos,ns-appstream update -y
  

• Restart a rootless application from UI, or reboot the node

Expected behavior

The rootless application is working as usual.

Actual behavior

The agent.service and application container restart fail.

Jun 03 10:08:45 rocky-linux98 systemd[13089]: Starting Rootless module/traefik1 agent...
Jun 03 10:08:45 rocky-linux98 systemd[13108]: agent.service: Failed at step CHDIR spawning /usr/bin/chmod: No such file or directory
Jun 03 10:08:45 rocky-linux98 systemd[13109]: agent.service: Failed at step CHDIR spawning /usr/local/bin/agent: No such file or directory
Jun 03 10:08:45 rocky-linux98 systemd[13089]: agent.service: Main process exited, code=exited, status=200/CHDIR
Jun 03 10:08:45 rocky-linux98 systemd[13089]: agent.service: Failed with result 'exit-code'.

Components

• Systemd systemd-252-67.el9_8.2.rocky.0.1.x86_64

See also

• Bug thread https://community.nethserver.org/t/upgrade-almalinux-9-7-to-9-8-didnt-work/27720
• Forum fix release announcement https://community.nethserver.org/t/ns8-core-3-19-1-released-required-compatibility-update-for-rocky-linux-and-almalinux-9-8/27773

---

Thanks to CoBlau

[DavidePrincipi]

In Testing core 3.19.1-dev.1

Prepare a virtualized system with the stable version 3.19.0 for each OS

• rocky linux 9.7
• debian 12 [testing apps on Debian is just duplication as there are no changes to test]

Install and configure the following apps:

• OpenLDAP
• Samba DC
• Mail
• Webtop
• Mattermost
• Nextcloud
• Ejabberd
• NethVoice

Create a VM snapshot

Test case 1 - standard RL 9.8 upgrade path

Run the command to upgrade the system to rocky linux 9.8:

dnf --enablerepo=baseos,appstream --disablerepo=ns-baseos,ns-appstream update -y

Update core to testing version:

api-cli run update-core --data '{"core_url":"ghcr.io/nethserver/core:3.19.1-dev.1","nodes":[1]}'

• Make sure the applications are still running without errors.
• Reboot the system and check apps again.

Test case 2 - inverted RL 9.8 upgrade path

Update to core 3.19.1-dev.1:

api-cli run update-core --data '{"core_url":"ghcr.io/nethserver/core:3.19.1-dev.1","nodes":[1]}'

• Make sure the applications are still running without errors.

Upgrade base system to rocky linux 9.8.

dnf --enablerepo=baseos,appstream --disablerepo=ns-baseos,ns-appstream update -y

• Make sure the applications are still running without errors.

Test case 3 - Debian 12 update

• Update to core 3.19.1-dev.1
• Make sure the apps Traefik and Loki are running correctly
• Reboot the system
• Check the apps Traefik and Loki again

Test case 4 - Debian 13 update

[start from base debian 12 snapshot]

• Upgrade to debian 13 / reboot
• Update to core 3.19.1-dev.1
• Make sure the apps Traefik and Loki are running correctly
• Reboot the system
• Check the apps Traefik and Loki again

[DavidePrincipi]

• Test case verified: 1 & 2
• Ongoing 3 & 4

[DavidePrincipi]

VERIFIED

Test case Rocky Linux (1,2), and Debian (3,4) are verified.