systemd --user vs Slurm

2023-08-23 13:13:38 +02:00
parent c1f9f1d09a
commit 474523a360
1 changed files with 47 additions and 0 deletions
@@ -93,6 +93,53 @@ This needs to be fixed manually:
 - run the complaining command and approve (or use `--yes`)
 - run `puppet agent -t` to finalize the configuration

+### Puppet run fails to install KCM related service/timer on Slurm node
+
+The Puppet run fails with
+```
+Notice: /Stage[main]/Profile::Aaa/Systemd::Service[kcm-destroy]/Exec[start-global-user-service-kcm-destroy]/returns: Failed to connect to bus: Connection refused
+Error: '/usr/bin/systemctl --quiet start --global kcm-destroy.service' returned 1 instead of one of [0]
+Error: /Stage[main]/Profile::Aaa/Systemd::Service[kcm-destroy]/Exec[start-global-user-service-kcm-destroy]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/systemctl --quiet start --global kcm-destroy.service' returned 1 instead of one of [0] (corrective)
+Notice: /Stage[main]/Profile::Aaa/Profile::Custom_timer[kcm-cleanup]/Systemd::Timer[kcm-cleanup]/Exec[start-global-user-timer-kcm-cleanup]/returns: Failed to connect to bus: Connection refused
+Error: '/usr/bin/systemctl --quiet start --global kcm-cleanup.timer' returned 1 instead of one of [0]
+Error: /Stage[main]/Profile::Aaa/Profile::Custom_timer[kcm-cleanup]/Systemd::Timer[kcm-cleanup]/Exec[start-global-user-timer-kcm-cleanup]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/systemctl --quiet start --global kcm-cleanup.timer' returned 1 instead of one of [0] (corrective)
+```
+
+This is caused by the use of KCM as default Kerberos credential cache in RHEL8:
+
+- for RHEL8 it was recommended to use the KCM provided by sssd as Kerberos Credential Cache.
+- a major issue of this KCM is that it does not remove outdated caches
+- this leads to a Denial-of-Service situation when all 64 slots are filled, new logins start to fail after (this is persistent, reboot does not help).
+- we fix this issue by running regularly cleanup script in user context
+- this "user context" is handled by the `systemd --user` instance, which is started on the first login and keeps running until the last session ends.
+- that systemd user instance is started by `pam_systemd.so`
+- `pam_systemd.so` and `pam_slurm_adopt.so` conflict because both want to set up cgroups
+- because of this there is no `pam_systemd.so` configured on Slurm nodes thus there is no `systemd --user` instance
+
+I see two options to solve this issue:
+- do not use KCM
+- get somehow systemd user instance running
+
+
+#### do not use KCM
+Can be done in Hiera, to get back to RHEL7 behavior do
+
+    aaa::default_krb_cache: "KEYRING:persistent:%{literal('%')}{uid}"
+
+then there will be no KCM magic any more.
+We could also make this automatically happen in Puppet when Slurm is enabled.
+
+
+#### get somehow systemd user instance running
+`pam_systemd.so` does not want to take its hands off cgroups:
+https://github.com/systemd/systemd/issues/13535
+
+But there is documented how to get (part?) of the `pam_systemd.so` functionality running with Slurm:
+https://slurm.schedmd.com/pam_slurm_adopt.html#PAM_CONFIG
+(the Prolog, TaskProlog and Epilog part).
+I wonder if that also starts a `systemd --user` instance or not. Or if it is possible to somehow integrate the start of it therein.
+
+
 ### Workstation Installation Takes Long and Seams to Hang
 On the very first puppet run the command to install the GUI packages takes up to 10 minutes and it looks like it
 is hanging. Usually it is after the installation of `/etc/sssd/sssd.conf`. Just give it a bit time.