# SSH Gateways The purpose of the ssh gateways is to give access to protected networks and resources (for a finite period of time). The gateway always gives access to the networks first name is indicating, i.e. sf-gw is giving access to all sf networks, sls-gw is giving access to sls networks. Users are only supposed to use ssh to connect to the gateways as well as them to further connect to other machines. Never the less, for ease of use, there are some protocols/ports that can directly be accessed from the ssh gateway. These ports include: 5900 VNC, 3389 RDP, 4000 NX, ICMP/PING. Therefore direct portforwarding on those ports will work. ``` ssh -L 3389:machine-you-want-to-connect:3389 protected-network-gw ``` It is not intended that users keep state on the gateways (e.g. screen/tmux sessions) Depending on the gateway the user authenticates via password or password/MFA combination. All ssh gateways are located in an isolated network __129.129.197.x__. Communication to and from the ssh gateways always go through the PSI firewall and needs to be explicitly enabled. Following communication is currently possible: ![](_static/ssh_gateways_overview.drawio.svg) The access to a gateway is always controlled via an AD group. The name of the AD group always follows the same pattern: __unx-gw_<gateway-name>__. __gateway-name__ is always the part of ssh gateways name before the __-gw__ (example: sls-gw.psi.ch > unx-gw_sls). (temporary solution) Depending on the gateway the members of the group are either managed in [DUO](https://duo.psi.ch) by the beamline scientist or via https://git.psi.ch/controls_it/unix_group_management. However, the general baseline is that always the responsible of the protected network must approve that a user is added to the group. (temporary solution) The effective update of the AD groups is currently done on gfa-admin.psi.ch via some webhooks / timers. - /etc/systemd/system/update_ad_gw_groups.service - /etc/systemd/system/ldaputils_webhook.service The administration and management of the gateways is done via hiera: https://git.psi.ch/linux-infra/hiera/data-lx (all the machines are in the sshgw group) ## Gateway List The list of supported gateways can be found here: https://git.psi.ch/linux-infra/ansible/playbooks/lx_ansible/-/blob/main/inventory.yaml#L3 ## Group Membership / Access Groups The memberships and the approver of the different gateway access groups (naming pattern: unx-gw_XX) can be found on this Service Now page: https://psi.service-now.com/now/nav/ui/classic/params/target/sys_user_group_list.do%3Fsysparm_query%3DnameSTARTSWITHunx-gw_%255Eactive%253Dtrue%26sysparm_first_row%3D1%26sysparm_view%3Dlinux_groups%26sysparm_choice_query_raw%3D%26sysparm_list_header_search%3Dtrue Once you have the list, click on a group to see the details for this group: ![](_static/ssh_gateways_list-01.png) To see/check the member of this group scroll down and select the __Group Members__ tab: ![](_static/ssh_gateways_list-02.png) ### Grant User Access to Gateway To grant a user access to a gateway use the same Workflow as described in the [SSH Gateway - User Guide](https://linux.psi.ch/services-user-guide/ssh_gateways.html) ### Remove / Revoke User Access Remove a user from a group, please open a normal Incident in service now. (Need to be improved!) ## Troubleshooting ### Checklist - Is gateway up and running? - Is user part of the AD group giving access to the gateway (ideally check on the gateway itself) ``` getent group unx-gw_ ``` or ``` id whaeveruser_l | sed 's/,/\n/g' | grep unx-gw_ 35526(unx-gw_twlha) 35514(unx-gw_hipa) 35524(unx-gw_sls) 35525(unx-gw_sf) -bash-4.2$ ``` - In case the user is not part of the group, the user needs to contact the respective responsible (i.e. beamline scientist in case of a beamline) to add him to the group. The management of the group membership is currently done in DUO. ### General Howto identify and kill high load sessions on the ssh gateway, useful commands for usage diagnostic: `top` or `htop` wil list heavy CPU consumers (see manpages for details) `w` will list all user connections (see manpage for details) `w ` will list connections for a specific user ### Show heavy CPU consumers ```bash [ ~]$ top -b -d 5 | head -n 20 top - 11:47:44 up 67 days, 6:09, 51 users, load average: 9.63, 10.87, 10.50 Tasks: 406 total, 10 running, 396 sleeping, 0 stopped, 0 zombie %Cpu(s): 74.3 us, 20.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si, 0.0 st KiB Mem : 8008520 total, 5377448 free, 908264 used, 1722808 buff/cache KiB Swap: 0 total, 0 free, 0 used. 6806948 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9674 xxxxxx+ 20 0 197068 6032 4196 S 10.3 0.1 225:29.90 ssh 10012 xxxxxx 20 0 202292 11976 1264 R 10.3 0.1 1257:25 sshd 10043 xxxxxx 20 0 202124 11052 4196 R 10.3 0.1 1350:13 ssh 27819 xxxxxx 20 0 205632 10324 1228 R 10.3 0.1 3462:20 sshd 9629 xxxxxx 20 0 192728 3700 1220 S 6.9 0.0 211:37.81 sshd 10160 xxxxxx 20 0 201304 5908 1228 S 6.9 0.1 501:14.56 sshd 10193 xxxxxx 20 0 199268 8140 4192 R 6.9 0.1 535:54.23 ssh 17510 xxxxxx 20 0 198616 3028 1252 R 6.9 0.0 15:08.35 sshd 18082 xxxxxx 20 0 204092 15148 1252 S 6.9 0.2 3:42.87 sshd 18786 xxxxxx 20 0 196448 5332 4188 S 6.9 0.1 1:36.56 ssh 19719 xxxxxx 20 0 199692 4404 1228 S 6.9 0.1 71:23.15 sshd 23834 xxxxxx 20 0 199096 3612 1204 R 6.9 0.0 156:01.83 sshd 23872 xxxxxx 20 0 198564 7540 4192 R 6.9 0.1 167:02.21 ssh ``` ### Show all connections from a specific user: ```bash [ ~]$ w wally_e 11:10:40 up 67 days, 5:32, 51 users, load average: 10.26, 7.79, 7.64 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT wally_e pts/0 pc11006.psi.ch 12Sep23 20days 12:03 0.36s sshd: wally_e [priv] wally_e pts/10 satese-cons-06.p 11Sep23 17days 0.02s 0.02s -bash wally_e pts/15 satese-cons-06.p Thu17 28:40 2:24m 2:24m ssh -XY sf-lca wally_e pts/68 pc11006.psi.ch 03Oct23 4days 4:19m 4:19m ssh -XY sf-lc7a wally_e pts/79 satesf-cons-07.p 26Sep23 13days 3:54m 3:54m ssh -CXY sf-lc7a wally_e pts/85 pc11006.psi.ch 21Sep23 1:58m 11:01m 11:01m ssh -CXY sf-lc7a ``` ### Show listing of last logged in users: ```bash [ ~]$ last bob_b pts/28 macstudvonhelge. Tue Oct 10 11:25 - 11:28 (00:03) bob_b pts/42 macstudvonhelge. Tue Oct 10 11:17 - 11:17 (00:00) bob_b pts/28 macstudvonhelge. Tue Oct 10 11:16 - 11:17 (00:00) builder_b pts/41 pc9681.psi.ch Tue Oct 10 11:08 still logged in […] ``` ### List all outbound connections for a specific user: ```bash [ ~]$ pgrep -au wally_e | grep -w ssh 8101 ssh -CXY sf-lc7a 9101 ssh -XY sf-lca 14058 ssh -CXY sf-lc7a 26888 ssh -CXY sf-lc7a 32317 ssh -XY sf-lc7a ``` ### List all inbound connections for a specific user: ```bash [ ~]$ pgrep -au wally_e | grep -w sshd 9066 sshd: wally_e@pts/15 14018 sshd: wally_e@pts/85 26857 sshd: wally_e@pts/79 30364 sshd: wally_e@pts/0 32177 sshd: wally_e@pts/10 32286 sshd: wally_e@pts/68 ``` Terminate Sessions ### The following command will terminate all sessions from user ‘wally_e’: ```bash [ ~]$ sudo pkill -u wally_e ``` ### The following command will terminate a specific sessions: ```bash [ ~]$ sudo kill 30364 ```