Controls-docs/rhel8/kerberos.md

Kerberos on RHEL 8

This document describes the state of Kerberos on RHEL 8. This includes the current open issues, a user guide and how we solved the KCM (Kerberos Cache Manager) issues.

Open Problems

• cleanup of caches, else we might end up in DoS situation. Best we do this systemd --unit managed.
• Kerberos with Firefox does not work yet.

User Guide

Manage Ticket for Admin User

If you need for administrative operations a TGT from your admin user (e.g. buchel_k-adm), then do

OLD_KRB5CCNAME=$KRB5CCNAME
export KRB5CCNAME=KCM:$(id -u):admin
kinit $(id -un)-adm

and after you are done do

kdestroy
export KRB5CCNAME=$OLD_KRB5CCNAME

to delete your administrative tickets and to get back to your normal credential cache.

List all Credential Caches

KRB5CCNAME=KCM: klist -l

lists all caches and

KRB5CCNAME=KCM: klist -A

also the tickets therein.

Kerberos Use and Test Cases

• ssh authentication (authentication method gssapi-with-mic)
• ssh TGT (ticket granting ticket) delegation (with GSSAPIDelegateCredentials yes)
• AFS authentication (aklog)
• AFS administrative operations where the user switches to a separate admin principal (e.g. buchel_k-adm)
• local desktop: get new TGT on login
• local desktop: TGT renewal after reauthentication on lock screen
• remote desktop with NoMachine NX: get new TGT on login
• remote desktop with NoMachine NX: TGT renewal after reconnection
• website authentication (SPNEGO with Firefox, Chrome)

KCM (Kerberos Cache Manager)

In RHEL 7 we are using the KEYRING (kernel keyring) cache, whereas for RHEL 8 there came early the wish to use KCM instead, which also is the new default.

The Kerberos documentation contains a reference for all available cache types.

The KCM cache is provided by a dedicated daemon, for RHEL8 this is sssd_kcm which has been programmed by Red Hat itself.

Advantages of KCM

The advantage of KCM is that the caches are permanent and survive daemon restarts and system reboots without the need to fiddle around with files and file permission. This simplifies daemon and container use cases. It also automatically renews tickets which is handy for every use case.

User Based vs Session Based

Intuitively I would expect that something delicate as authentication is managed per session (ssh, desktop, console login, ...).

Aparently with KCM this is not the case. It provides a default cache which is supposed to be the optimal for you and that can change any time.

Problems I see with this are

• user may change his principal, eg. for admin operations (kinit buchel_k-adm) which is then used by all sessions
• user may destroy the cache (it is good security practice to have a kdestroy in .bash_logout to ensure nobody on the machine can use your tokens after logging out)
• software may put tokens into the cache which suddenly are not there any more
• the magic/heuristic used to select might not work optimally for all use cases (as we see below sshd-kcm fails horribly...)

So if we have more than one session on a machine (e.g. people connecting via remote desktop and ssh at the same time), the cross-session side-effects can cause unexpected behaviour.

In contrast to this for AFS token renewal having access to new tokens is helpful, as this allows prolong the time a PAG (group of processes authenticated against AFS) is working as long as there is at least one valid ticket available. Or even to recover when a new ticket comes available again.

A way to get KCM of of the business of selecting the "optimal" cache is to select it yourself and provide the session/software one specific cache by setting the KRB5CCNAME environment variable accordingly (e.g. KCM:44951:66120). Note when set to KCM: it will use as default cache the one KCM believes should be the default cache. And that can change for whatever reason.

Problems of sssd_kcm

To check the Kerberos credential cache, you can use klist to look a the current default cache and klist -l to look at all available caches. Note that the first listed cache is the default cache. Of course that is only valid when there is no KRB5CCNAME environment variable set or it is KCM:.

No Cleanup of Expired Caches

The most obvious and well known problem of sshd-kcm is that it does not remove expired tokens and credential caches. I agree that it should not have an impact as this is mostly cosmetic. But that is only the case when everything can cope with that...

By default is is limited to 64 caches, but when that limit was hit, then it was not possible any more to authenticate on the lock screen:

Okt 05 14:57:11 lxdev01.psi.ch krb5_child[43689]: Internal credentials cache error

So this causes a denial of service problem, we need to deal with somehow, e.g. by regulary removing expired caches. And note that these caches are persistent and do not get removed on reboot.

Use of Expired Credential Caches

In below example you see that on the ssh login, I got a new default cache. But after a few minutes (there was a Desktop login from my side and maybe an automatic AFS token renewal in between), I get an expired cache as default cache.

$ ssh lxdev01.psi.ch
Last login: Tue Oct  4 09:50:33 2022
[buchel_k@lxdev01 ~]$ klist -l
Principal name                 Cache name
--------------                 ----------
buchel_k@D.PSI.CH              KCM:44951:42923
buchel_k@D.PSI.CH              KCM:44951:12312 (Expired)
buchel_k@D.PSI.CH              KCM:44951:42199 (Expired)
buchel_k@D.PSI.CH              KCM:44951:40168
buchel_k@D.PSI.CH              KCM:44951:8914 (Expired)
buchel_k@D.PSI.CH              KCM:44951:62275 (Expired)
buchel_k@D.PSI.CH              KCM:44951:27078 (Expired)
buchel_k@D.PSI.CH              KCM:44951:73924 (Expired)
buchel_k@D.PSI.CH              KCM:44951:72006
buchel_k@D.PSI.CH              KCM:44951:64449 (Expired)
buchel_k@D.PSI.CH              KCM:44951:60061 (Expired)
buchel_k@D.PSI.CH              KCM:44951:36925 (Expired)
buchel_k@D.PSI.CH              KCM:44951:48361 (Expired)
buchel_k@D.PSI.CH              KCM:44951:49651 (Expired)
buchel_k@D.PSI.CH              KCM:44951:76984 (Expired)
buchel_k@D.PSI.CH              KCM:44951:54227 (Expired)
buchel_k@D.PSI.CH              KCM:44951:85800 (Expired)
[buchel_k@lxdev01 ~]$ klist -l
Principal name                 Cache name
--------------                 ----------
buchel_k@D.PSI.CH              KCM:44951:12312 (Expired)
buchel_k@D.PSI.CH              KCM:44951:42199 (Expired)
buchel_k@D.PSI.CH              KCM:44951:40168
buchel_k@D.PSI.CH              KCM:44951:8914 (Expired)
buchel_k@D.PSI.CH              KCM:44951:62275 (Expired)
buchel_k@D.PSI.CH              KCM:44951:27078 (Expired)
buchel_k@D.PSI.CH              KCM:44951:73924 (Expired)
buchel_k@D.PSI.CH              KCM:44951:72006
buchel_k@D.PSI.CH              KCM:44951:64449 (Expired)
buchel_k@D.PSI.CH              KCM:44951:60061 (Expired)
buchel_k@D.PSI.CH              KCM:44951:36925 (Expired)
buchel_k@D.PSI.CH              KCM:44951:48361 (Expired)
buchel_k@D.PSI.CH              KCM:44951:42923
buchel_k@D.PSI.CH              KCM:44951:49651 (Expired)
buchel_k@D.PSI.CH              KCM:44951:76984 (Expired)
buchel_k@D.PSI.CH              KCM:44951:54227 (Expired)
buchel_k@D.PSI.CH              KCM:44951:85800 (Expired)
[buchel_k@lxdev01 ~]$

Note that the automatic AFS token renewal was created after we have experienced this issue.

Busy Loop of goa-daemon

If the GNOME Online Accounts encounters a number of Kerberos credential caches it goes into a busy loop and causes sssd-kcm to consume 100% of one core. Happily ignored bugs at Red Hat and Gnome.

Zombie Caches by NoMachine NX

On a machine with remote desktop access using NoMachine NX I have seen following cache list in the log:

# /usr/bin/klist -l
Principal name                 Cache name
--------------                 ----------
fische_r@D.PSI.CH              KCM:45334:73632 (Expired)
buchel_k@D.PSI.CH              KCM:45334:55706 (Expired)
fische_r@D.PSI.CH              KCM:45334:44226 (Expired)
fische_r@D.PSI.CH              KCM:45334:40904 (Expired)
fische_r@D.PSI.CH              KCM:45334:62275 (Expired)
fische_r@D.PSI.CH              KCM:45334:89020 (Expired)
buchel_k@D.PSI.CH              KCM:45334:25061 (Expired)
buchel_k@D.PSI.CH              KCM:45334:35168 (Expired)
fische_r@D.PSI.CH              KCM:45334:73845 (Expired)
fische_r@D.PSI.CH              KCM:45334:47508 (Expired)
fische_r@D.PSI.CH              KCM:45334:34317 (Expired)
fische_r@D.PSI.CH              KCM:45334:52058 (Expired)
fische_r@D.PSI.CH              KCM:45334:16150 (Expired)
fische_r@D.PSI.CH              KCM:45334:84445 (Expired)
fische_r@D.PSI.CH              KCM:45334:69076 (Expired)
buchel_k@D.PSI.CH              KCM:45334:87346 (Expired)
fische_r@D.PSI.CH              KCM:45334:57070 (Expired)

or on another machine in my personal list:

[buchel_k@pc14831 ~]$ klist -l
Principal name                 Cache name
--------------                 ----------
buchel_k@D.PSI.CH              KCM:44951:69748
buchel_k@D.PSI.CH              KCM:44951:18506 (Expired)
buchel_k@D.PSI.CH              KCM:44951:5113 (Expired)
buchel_k@D.PSI.CH              KCM:44951:52685 (Expired)
buchel_k@D.PSI.CH              KCM:44951:13951 (Expired)
PC14831$@D.PSI.CH              KCM:44951:43248 (Expired)
PC14831$@D.PSI.CH              KCM:44951:58459 (Expired)
buchel_k@D.PSI.CH              KCM:44951:14668 (Expired)
buchel_k@D.PSI.CH              KCM:44951:92516 (Expired)
[buchel_k@pc14831 ~]$ 

Both show principals which I am very sure that they have not been added manually by the user. And somewhere there is a security issue, either sssd-kcm or NoMachine NX.

In another experiment I logged into a machine with ssh and did kdestroy -A which should destroy all caches:

[buchel_k@mpc2959 ~]$ kdestroy -A
[buchel_k@mpc2959 ~]$ klist -l
Principal name Cache name
[buchel_k@mpc2959 ~]$

After I logged in via NoMachine NX I got a cache expired since more than two month:

[buchel_k@mpc2959 ~]$ klist -l
Principal name Cache name

buchel_k@D.PSI.CH KCM:44951:16795 (Expired)
buchel_k@D.PSI.CH KCM:44951:69306
[buchel_k@mpc2959 ~]$ klist
Ticket cache: KCM:44951:16795
Default principal: buchel_k@D.PSI.CH

Valid starting Expires Service principal
13.07.2022 11:35:51 13.07.2022 21:26:19 krbtgt/D.PSI.CH@D.PSI.CH
renew until 14.07.2022 11:26:19
[buchel_k@mpc2959 ~]$ date
Do Sep 22 08:37:41 CEST 2022
[buchel_k@mpc2959 ~]$ 

Note that a non-expired cache is available, but NoMachine NX explicitely sets KRB5CCNAME to a specific KCM cache. And it contains a ticket/cache which is supposed to the gone.

So there is a security bug in sssd-kcm: it does not fully destroy tickets when being told so. And there is another security issue in the NoMachine NX -> sssd-kcm interaction. I assume that it talks with the KCM as root and gets somehow (or has saved somewhere) old caches and moves them over into user context. But the cache may originally not have belonged to the user...

I have not found a lot concerning Kerberos on the NoMachine website.

Solution Attempts

Ideally we would get to a solution which can do the following:

• interactive user sessions are isolated do not interfer with each other
• AFS can get hold of new tickets and inject them into the PAGs as long as the user somehow regularly authenticates
• systemd --user which is residing outside of the interactive user sessions is happy as well
• goa-daemon sees only one cache
• expired caches get somehow cleaned up

Only One Cache

The sssd-kcm limits the number of caches by default to 64, but that can be changed to 1 with the max_uid_ccaches. So there would be only one cache, shared by all sessions, but at least the KCM cannot serve anything but the latest.

But some logins do not work any more when the maximum number of caches is hit as already documented above in the chapter "No Cleanup of Expired Caches".

renew-afstoken Script/Daemon

For AFS we (Achim and I) made the script renew-afstoken which is started as per PAG daemon by PAM upon login. Out of the available KCM caches it selects a suitable one to regulary get a new AFS token. This now works very robust and can also recover from expiration when a new ticket gets available.

Setup Shared or Isolated Caches with KRB5CCNAME in own PAM Module

The self-made PAM module pam_single_kcm_cache.so improves the situation by setting

• KRB5CCNAME=KCM:$UID:desktop to use a shared credential cache for desktop sessions and systemd --user
• KRB5CCNAME=KCM:$UID:$RANDOM_LETTERS for text sessions to provide session isolation

and providing a working TGT in these caches.

I identified so far two cases of the program flow in PAM to manage:

• TGT delegation as done by sshd with authentication method gssapi-with-mic, where a new cache is created by sshd and then filled with the delegated ticket
• TGT creation as done by sss.so upon password authentication, where a new TGT is created an placed into the KCM managed default cache.

Now there is no simple and bullet proof selection of where the TGT ends up in KCM. The KCM designated default cache might it be or not. To work around this, the module iterates through all credential caches provided by the KCM copies a TGT which is younger than 10 s and has a principal fitting the username.

Note that the reason for systemd --user to use the same credential cache as the desktop sessions is that at least Gnome uses it to start the user programs like Evolution or Firefox.

The code you find in Gitlab where there is currently an open merge request for the initial commit. I plan to make that public on Github.

Red Hat Ticket

I have an ticket open with Red Hat on this case. On the first part I concentrated on the missing session isolation, but it showed that this is the supposed behaviour of a KCM setup.

One Problem is that our machines have some non-standard software which is not covered by the support. It is YFS for AFS and NoMachine NX.

Then it is not that easy to reproduce as the problem is best seen in a long running and used system. To create such a test system with several users and many expired sessions means quite some effort.

I posted a few strange looking klist outputs and asked for explanation, but that seamed not yet to have reached someone with intimidate sssd-kcm knowledge.

I hope that sharing this document with them will help.

Other Options

• another selfmade daemon to monitor/clean up sssd-kcm

Fill in your ideas.

PS

There is an advanage in the broken sssd-kcm default cache selection: it forces us to make our stuff robust against KCM glitches, which might also occur with a better manager, just way less often and then it would be more harder to explain and to track down.