5.7 KiB
SSH Gateways
The purpose of the ssh gateways is to give access to protected networks and resources (for a finite period of time).
Users are only supposed to use ssh to connect and on the gateways. They are also supposed to only use the ssh command to further connect to other machines. It is not intended that users keep state on the gateways (e.g. screen/tmux sessions)
Depending on the gateway the user authenticates via password or password/MFA combination.
All ssh gateways are located in an isolated network 129.129.197.x. Communication to and from the ssh gateways always go through the PSI firewall and needs to be explicitly enabled.
Following communication is currently possible:
The access to a gateway is always controlled via an AD group. The name of the AD group always follows the same pattern: unx-gw<gateway-name>_. gateway-name is always the part of ssh gateways name before the -gw (example: sls-gw.psi.ch > unx-gw_sls).
(temporary solution) Depending on the gateway the members of the group are either managed in DUO by the beamline scientist or via https://git.psi.ch/controls_it/unix_group_management.
However, the general baseline is that always the responsible of the protected network must approve that a user is added to the group.
(temporary solution) The effective update of the AD groups is currently done on gfa-admin.psi.ch via some webhooks / timers.
- /etc/systemd/system/update_ad_gw_groups.service
- /etc/systemd/system/ldaputils_webhook.service
The administration and management of the gateways is done via hiera: https://git.psi.ch/linux-infra/hiera/data-lx (all the machines are in the sshgw group)
Troubleshooting
Checklist
- Is gateway up and running
- Is user part of the AD group giving access to the gateway (ideally check on the gateway itself)
getent group unx-gw_<gateway name>- In case the user is not part of the group, the user needs to contact the respective responsible (i.e. beamline scientist in case of a beamline) to add him to the group.
General
Howto identify and kill high load sessions on the ssh gateway, useful commands for usage diagnostic:
top or htop wil list heavy CPU consumers (see manpages for details)
w will list all user connections (see manpage for details)
w <username> will list connections for a specific user
Show heavy CPU consumers
[ ~]$ top -b -d 5 | head -n 20
top - 11:47:44 up 67 days, 6:09, 51 users, load average: 9.63, 10.87, 10.50
Tasks: 406 total, 10 running, 396 sleeping, 0 stopped, 0 zombie
%Cpu(s): 74.3 us, 20.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si, 0.0 st
KiB Mem : 8008520 total, 5377448 free, 908264 used, 1722808 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 6806948 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9674 schnorr+ 20 0 197068 6032 4196 S 10.3 0.1 225:29.90 ssh
10012 arrell_c 20 0 202292 11976 1264 R 10.3 0.1 1257:25 sshd
10043 arrell_c 20 0 202124 11052 4196 R 10.3 0.1 1350:13 ssh
27819 xie_x 20 0 205632 10324 1228 R 10.3 0.1 3462:20 sshd
9629 schnorr+ 20 0 192728 3700 1220 S 6.9 0.0 211:37.81 sshd
10160 beard_c 20 0 201304 5908 1228 S 6.9 0.1 501:14.56 sshd
10193 beard_c 20 0 199268 8140 4192 R 6.9 0.1 535:54.23 ssh
17510 loehl_f1 20 0 198616 3028 1252 R 6.9 0.0 15:08.35 sshd
18082 lombosi+ 20 0 204092 15148 1252 S 6.9 0.2 3:42.87 sshd
18786 vallott+ 20 0 196448 5332 4188 S 6.9 0.1 1:36.56 ssh
19719 demirb_u 20 0 199692 4404 1228 S 6.9 0.1 71:23.15 sshd
23834 menzel_r 20 0 199096 3612 1204 R 6.9 0.0 156:01.83 sshd
23872 menzel_r 20 0 198564 7540 4192 R 6.9 0.1 167:02.21 ssh
Show all connections from a specific user:
[ ~]$ w divall_e
11:10:40 up 67 days, 5:32, 51 users, load average: 10.26, 7.79, 7.64
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
divall_e pts/0 pc11006.psi.ch 12Sep23 20days 12:03 0.36s sshd: divall_e [priv]
divall_e pts/10 satese-cons-06.p 11Sep23 17days 0.02s 0.02s -bash
divall_e pts/15 satese-cons-06.p Thu17 28:40 2:24m 2:24m ssh -XY sf-lca
divall_e pts/68 pc11006.psi.ch 03Oct23 4days 4:19m 4:19m ssh -XY sf-lc7a
divall_e pts/79 satesf-cons-07.p 26Sep23 13days 3:54m 3:54m ssh -CXY sf-lc7a
divall_e pts/85 pc11006.psi.ch 21Sep23 1:58m 11:01m 11:01m ssh -CXY sf-lc7a
Show listing of last logged in users:
[ ~]$ last
brands pts/28 macstudvonhelge. Tue Oct 10 11:25 - 11:28 (00:03)
brands pts/42 macstudvonhelge. Tue Oct 10 11:17 - 11:17 (00:00)
brands pts/28 macstudvonhelge. Tue Oct 10 11:16 - 11:17 (00:00)
follath_ pts/41 pc9681.psi.ch Tue Oct 10 11:08 still logged in
kapeller pts/40 mela.psi.ch Tue Oct 10 10:58 still logged in
vallotto pts/39 nx-node-2.psi.ch Tue Oct 10 10:46 still logged in
[…]
List all outbound connections for a specific user:
[ ~]$ pgrep -au divall_e | grep -w ssh
8101 ssh -CXY sf-lc7a
9101 ssh -XY sf-lca
14058 ssh -CXY sf-lc7a
26888 ssh -CXY sf-lc7a
32317 ssh -XY sf-lc7a
List all inbound connections for a specific user:
[ ~]$ pgrep -au divall_e | grep -w sshd
9066 sshd: divall_e@pts/15
14018 sshd: divall_e@pts/85
26857 sshd: divall_e@pts/79
30364 sshd: divall_e@pts/0
32177 sshd: divall_e@pts/10
32286 sshd: divall_e@pts/68
Terminate Sessions
The following command will terminate all sessions from user ‘dival_e’:
[ ~]$ sudo pkill -u brands
The following command will terminate a specific sessions:
[ ~]$ sudo kill 30364