7.4 KiB
SSH Gateways
The purpose of the ssh gateways is to give access to protected networks and resources (for a finite period of time). The gateway always gives access to the networks first name is indicating, i.e. sf-gw is giving access to all sf networks, sls-gw is giving access to sls networks.
Users are only supposed to use ssh to connect to the gateways as well as them to further connect to other machines. Never the less, for ease of use, there are some protocols/ports that can directly be accessed from the ssh gateway. These ports include: 5900 VNC, 3389 RDP, 4000 NX, ICMP/PING. Therefore direct portforwarding on those ports will work.
ssh -L 3389:machine-you-want-to-connect:3389 protected-network-gw
It is not intended that users keep state on the gateways (e.g. screen/tmux sessions)
Depending on the gateway the user authenticates via password or password/MFA combination.
All ssh gateways are located in an isolated network 129.129.197.x. Communication to and from the ssh gateways always go through the PSI firewall and needs to be explicitly enabled.
Following communication is currently possible:
The access to a gateway is always controlled via an AD group. The name of the AD group always follows the same pattern: unx-gw<gateway-name>_. gateway-name is always the part of ssh gateways name before the -gw (example: sls-gw.psi.ch > unx-gw_sls).
(temporary solution) Depending on the gateway the members of the group are either managed in DUO by the beamline scientist or via https://git.psi.ch/controls_it/unix_group_management.
However, the general baseline is that always the responsible of the protected network must approve that a user is added to the group.
(temporary solution) The effective update of the AD groups is currently done on gfa-admin.psi.ch via some webhooks / timers.
- /etc/systemd/system/update_ad_gw_groups.service
- /etc/systemd/system/ldaputils_webhook.service
The administration and management of the gateways is done via hiera: https://git.psi.ch/linux-infra/hiera/data-lx (all the machines are in the sshgw group)
Gateway List
The list of supported gateways can be found here: https://git.psi.ch/linux-infra/ansible/playbooks/lx_ansible/-/blob/main/inventory.yaml#L3
Group Membership / Access Groups
The memberships and the approver of the different gateway access groups (naming pattern: unx-gw_XX) can be found on this Service Now page:
Once you have the list, click on a group to see the details for this group:
To see/check the member of this group scroll down and select the Group Members tab:
Grant User Access to Gateway
To grant a user access to a gateway use the same Workflow as described in the SSH Gateway - User Guide
Remove / Revoke User Access
Remove a user from a group, please open a normal Incident in service now. (Need to be improved!)
Troubleshooting
Checklist
-
Is gateway up and running?
-
Is user part of the AD group giving access to the gateway (ideally check on the gateway itself)
getent group unx-gw_<gateway name>or
id whaeveruser_l | sed 's/,/\n/g' | grep unx-gw_ 35526(unx-gw_twlha) 35514(unx-gw_hipa) 35524(unx-gw_sls) 35525(unx-gw_sf) -bash-4.2$- In case the user is not part of the group, the user needs to contact the respective responsible (i.e. beamline scientist in case of a beamline) to add him to the group. The management of the group membership is currently done in DUO.
General
Howto identify and kill high load sessions on the ssh gateway, useful commands for usage diagnostic:
top or htop wil list heavy CPU consumers (see manpages for details)
w will list all user connections (see manpage for details)
w <username> will list connections for a specific user
Show heavy CPU consumers
[ ~]$ top -b -d 5 | head -n 20
top - 11:47:44 up 67 days, 6:09, 51 users, load average: 9.63, 10.87, 10.50
Tasks: 406 total, 10 running, 396 sleeping, 0 stopped, 0 zombie
%Cpu(s): 74.3 us, 20.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si, 0.0 st
KiB Mem : 8008520 total, 5377448 free, 908264 used, 1722808 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 6806948 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9674 xxxxxx+ 20 0 197068 6032 4196 S 10.3 0.1 225:29.90 ssh
10012 xxxxxx 20 0 202292 11976 1264 R 10.3 0.1 1257:25 sshd
10043 xxxxxx 20 0 202124 11052 4196 R 10.3 0.1 1350:13 ssh
27819 xxxxxx 20 0 205632 10324 1228 R 10.3 0.1 3462:20 sshd
9629 xxxxxx 20 0 192728 3700 1220 S 6.9 0.0 211:37.81 sshd
10160 xxxxxx 20 0 201304 5908 1228 S 6.9 0.1 501:14.56 sshd
10193 xxxxxx 20 0 199268 8140 4192 R 6.9 0.1 535:54.23 ssh
17510 xxxxxx 20 0 198616 3028 1252 R 6.9 0.0 15:08.35 sshd
18082 xxxxxx 20 0 204092 15148 1252 S 6.9 0.2 3:42.87 sshd
18786 xxxxxx 20 0 196448 5332 4188 S 6.9 0.1 1:36.56 ssh
19719 xxxxxx 20 0 199692 4404 1228 S 6.9 0.1 71:23.15 sshd
23834 xxxxxx 20 0 199096 3612 1204 R 6.9 0.0 156:01.83 sshd
23872 xxxxxx 20 0 198564 7540 4192 R 6.9 0.1 167:02.21 ssh
Show all connections from a specific user:
[ ~]$ w wally_e
11:10:40 up 67 days, 5:32, 51 users, load average: 10.26, 7.79, 7.64
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
wally_e pts/0 pc11006.psi.ch 12Sep23 20days 12:03 0.36s sshd: wally_e [priv]
wally_e pts/10 satese-cons-06.p 11Sep23 17days 0.02s 0.02s -bash
wally_e pts/15 satese-cons-06.p Thu17 28:40 2:24m 2:24m ssh -XY sf-lca
wally_e pts/68 pc11006.psi.ch 03Oct23 4days 4:19m 4:19m ssh -XY sf-lc7a
wally_e pts/79 satesf-cons-07.p 26Sep23 13days 3:54m 3:54m ssh -CXY sf-lc7a
wally_e pts/85 pc11006.psi.ch 21Sep23 1:58m 11:01m 11:01m ssh -CXY sf-lc7a
Show listing of last logged in users:
[ ~]$ last
bob_b pts/28 macstudvonhelge. Tue Oct 10 11:25 - 11:28 (00:03)
bob_b pts/42 macstudvonhelge. Tue Oct 10 11:17 - 11:17 (00:00)
bob_b pts/28 macstudvonhelge. Tue Oct 10 11:16 - 11:17 (00:00)
builder_b pts/41 pc9681.psi.ch Tue Oct 10 11:08 still logged in
[…]
List all outbound connections for a specific user:
[ ~]$ pgrep -au wally_e | grep -w ssh
8101 ssh -CXY sf-lc7a
9101 ssh -XY sf-lca
14058 ssh -CXY sf-lc7a
26888 ssh -CXY sf-lc7a
32317 ssh -XY sf-lc7a
List all inbound connections for a specific user:
[ ~]$ pgrep -au wally_e | grep -w sshd
9066 sshd: wally_e@pts/15
14018 sshd: wally_e@pts/85
26857 sshd: wally_e@pts/79
30364 sshd: wally_e@pts/0
32177 sshd: wally_e@pts/10
32286 sshd: wally_e@pts/68
Terminate Sessions
The following command will terminate all sessions from user ‘wally_e’:
[ ~]$ sudo pkill -u wally_e
The following command will terminate a specific sessions:
[ ~]$ sudo kill 30364