179 lines
7.4 KiB
Markdown
179 lines
7.4 KiB
Markdown
# SSH Gateways
|
||
|
||
The purpose of the ssh gateways is to give access to protected networks and resources (for a finite period of time). The gateway always gives access to the networks first name is indicating, i.e. sf-gw is giving access to all sf networks, sls-gw is giving access to sls networks.
|
||
|
||
Users are only supposed to use ssh to connect to the gateways as well as them to further connect to other machines. Never the less, for ease of use, there are some protocols/ports that can directly be accessed from the ssh gateway. These ports include: 5900 VNC, 3389 RDP, 4000 NX, ICMP/PING.
|
||
Therefore direct portforwarding on those ports will work.
|
||
|
||
```
|
||
ssh -L 3389:machine-you-want-to-connect:3389 protected-network-gw
|
||
```
|
||
|
||
It is not intended that users keep state on the gateways (e.g. screen/tmux sessions)
|
||
|
||
Depending on the gateway the user authenticates via password or password/MFA combination.
|
||
|
||
All ssh gateways are located in an isolated network __129.129.197.x__. Communication to and from the ssh gateways always go through the PSI firewall and needs to be explicitly enabled.
|
||
|
||
Following communication is currently possible:
|
||
|
||
|
||
|
||
The access to a gateway is always controlled via an AD group. The name of the AD group always follows the same pattern: __unx-gw_<gateway-name>__. __gateway-name__ is always the part of ssh gateways name before the __-gw__ (example: sls-gw.psi.ch > unx-gw_sls).
|
||
|
||
|
||
(temporary solution) Depending on the gateway the members of the group are either managed in [DUO](https://duo.psi.ch) by the beamline scientist or via https://git.psi.ch/controls_it/unix_group_management.
|
||
|
||
However, the general baseline is that always the responsible of the protected network must approve that a user is added to the group.
|
||
|
||
(temporary solution) The effective update of the AD groups is currently done on gfa-admin.psi.ch via some webhooks / timers.
|
||
- /etc/systemd/system/update_ad_gw_groups.service
|
||
- /etc/systemd/system/ldaputils_webhook.service
|
||
|
||
|
||
The administration and management of the gateways is done via hiera:
|
||
https://git.psi.ch/linux-infra/hiera/data-lx (all the machines are in the sshgw group)
|
||
|
||
## Gateway List
|
||
The list of supported gateways can be found here:
|
||
https://git.psi.ch/linux-infra/ansible/playbooks/lx_ansible/-/blob/main/inventory.yaml#L3
|
||
|
||
|
||
## Group Membership / Access Groups
|
||
|
||
The memberships and the approver of the different gateway access groups (naming pattern: unx-gw_XX) can be found on this Service Now page:
|
||
|
||
https://psi.service-now.com/now/nav/ui/classic/params/target/sys_user_group_list.do%3Fsysparm_query%3DnameSTARTSWITHunx-gw_%255Eactive%253Dtrue%26sysparm_first_row%3D1%26sysparm_view%3Dlinux_groups%26sysparm_choice_query_raw%3D%26sysparm_list_header_search%3Dtrue
|
||
|
||
Once you have the list, click on a group to see the details for this group:
|
||
|
||
|
||
To see/check the member of this group scroll down and select the __Group Members__ tab:
|
||
|
||
|
||
|
||
|
||
|
||
### Grant User Access to Gateway
|
||
To grant a user access to a gateway use the same Workflow as described in the [SSH Gateway - User Guide](https://linux.psi.ch/services-user-guide/ssh_gateways.html)
|
||
|
||
### Remove / Revoke User Access
|
||
Remove a user from a group, please open a normal Incident in service now. (Need to be improved!)
|
||
|
||
|
||
## Troubleshooting
|
||
|
||
### Checklist
|
||
|
||
- Is gateway up and running?
|
||
- Is user part of the AD group giving access to the gateway (ideally check on the gateway itself)
|
||
```
|
||
getent group unx-gw_<gateway name>
|
||
```
|
||
or
|
||
|
||
```
|
||
id whaeveruser_l | sed 's/,/\n/g' | grep unx-gw_
|
||
35526(unx-gw_twlha)
|
||
35514(unx-gw_hipa)
|
||
35524(unx-gw_sls)
|
||
35525(unx-gw_sf)
|
||
-bash-4.2$
|
||
```
|
||
|
||
- In case the user is not part of the group, the user needs to contact the respective responsible (i.e. beamline scientist in case of a beamline) to add him to the group. The management of the group membership is currently done in DUO.
|
||
|
||
### General
|
||
Howto identify and kill high load sessions on the ssh gateway, useful commands for usage diagnostic:
|
||
|
||
`top` or `htop` wil list heavy CPU consumers (see manpages for details)
|
||
`w` will list all user connections (see manpage for details)
|
||
`w <username>` will list connections for a specific user
|
||
|
||
|
||
### Show heavy CPU consumers
|
||
```bash
|
||
[ ~]$ top -b -d 5 | head -n 20
|
||
top - 11:47:44 up 67 days, 6:09, 51 users, load average: 9.63, 10.87, 10.50
|
||
Tasks: 406 total, 10 running, 396 sleeping, 0 stopped, 0 zombie
|
||
%Cpu(s): 74.3 us, 20.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si, 0.0 st
|
||
KiB Mem : 8008520 total, 5377448 free, 908264 used, 1722808 buff/cache
|
||
KiB Swap: 0 total, 0 free, 0 used. 6806948 avail Mem
|
||
|
||
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
|
||
9674 xxxxxx+ 20 0 197068 6032 4196 S 10.3 0.1 225:29.90 ssh
|
||
10012 xxxxxx 20 0 202292 11976 1264 R 10.3 0.1 1257:25 sshd
|
||
10043 xxxxxx 20 0 202124 11052 4196 R 10.3 0.1 1350:13 ssh
|
||
27819 xxxxxx 20 0 205632 10324 1228 R 10.3 0.1 3462:20 sshd
|
||
9629 xxxxxx 20 0 192728 3700 1220 S 6.9 0.0 211:37.81 sshd
|
||
10160 xxxxxx 20 0 201304 5908 1228 S 6.9 0.1 501:14.56 sshd
|
||
10193 xxxxxx 20 0 199268 8140 4192 R 6.9 0.1 535:54.23 ssh
|
||
17510 xxxxxx 20 0 198616 3028 1252 R 6.9 0.0 15:08.35 sshd
|
||
18082 xxxxxx 20 0 204092 15148 1252 S 6.9 0.2 3:42.87 sshd
|
||
18786 xxxxxx 20 0 196448 5332 4188 S 6.9 0.1 1:36.56 ssh
|
||
19719 xxxxxx 20 0 199692 4404 1228 S 6.9 0.1 71:23.15 sshd
|
||
23834 xxxxxx 20 0 199096 3612 1204 R 6.9 0.0 156:01.83 sshd
|
||
23872 xxxxxx 20 0 198564 7540 4192 R 6.9 0.1 167:02.21 ssh
|
||
```
|
||
|
||
|
||
### Show all connections from a specific user:
|
||
```bash
|
||
[ ~]$ w wally_e
|
||
11:10:40 up 67 days, 5:32, 51 users, load average: 10.26, 7.79, 7.64
|
||
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
|
||
wally_e pts/0 pc11006.psi.ch 12Sep23 20days 12:03 0.36s sshd: wally_e [priv]
|
||
wally_e pts/10 satese-cons-06.p 11Sep23 17days 0.02s 0.02s -bash
|
||
wally_e pts/15 satese-cons-06.p Thu17 28:40 2:24m 2:24m ssh -XY sf-lca
|
||
wally_e pts/68 pc11006.psi.ch 03Oct23 4days 4:19m 4:19m ssh -XY sf-lc7a
|
||
wally_e pts/79 satesf-cons-07.p 26Sep23 13days 3:54m 3:54m ssh -CXY sf-lc7a
|
||
wally_e pts/85 pc11006.psi.ch 21Sep23 1:58m 11:01m 11:01m ssh -CXY sf-lc7a
|
||
```
|
||
|
||
### Show listing of last logged in users:
|
||
```bash
|
||
[ ~]$ last
|
||
bob_b pts/28 macstudvonhelge. Tue Oct 10 11:25 - 11:28 (00:03)
|
||
bob_b pts/42 macstudvonhelge. Tue Oct 10 11:17 - 11:17 (00:00)
|
||
bob_b pts/28 macstudvonhelge. Tue Oct 10 11:16 - 11:17 (00:00)
|
||
builder_b pts/41 pc9681.psi.ch Tue Oct 10 11:08 still logged in
|
||
[…]
|
||
```
|
||
|
||
### List all outbound connections for a specific user:
|
||
```bash
|
||
[ ~]$ pgrep -au wally_e | grep -w ssh
|
||
8101 ssh -CXY sf-lc7a
|
||
9101 ssh -XY sf-lca
|
||
14058 ssh -CXY sf-lc7a
|
||
26888 ssh -CXY sf-lc7a
|
||
32317 ssh -XY sf-lc7a
|
||
```
|
||
|
||
### List all inbound connections for a specific user:
|
||
```bash
|
||
[ ~]$ pgrep -au wally_e | grep -w sshd
|
||
9066 sshd: wally_e@pts/15
|
||
14018 sshd: wally_e@pts/85
|
||
26857 sshd: wally_e@pts/79
|
||
30364 sshd: wally_e@pts/0
|
||
32177 sshd: wally_e@pts/10
|
||
32286 sshd: wally_e@pts/68
|
||
```
|
||
|
||
|
||
Terminate Sessions
|
||
|
||
### The following command will terminate all sessions from user ‘wally_e’:
|
||
|
||
```bash
|
||
[ ~]$ sudo pkill -u wally_e
|
||
```
|
||
|
||
### The following command will terminate a specific sessions:
|
||
```bash
|
||
[ ~]$ sudo kill 30364
|
||
```
|
||
|
||
|