Added interactive-jobs.md and linux/macos/windows client recipes

_data/sidebars/merlin6_sidebar.yml

@@ -23,6 +23,12 @@ entries:
       url: /merlin6/request-project.html
     - title: Accessing Interactive Nodes
       url: /merlin6/interactive.html
+    - title: Accessing from a Linux client
+      url: /merlin6/connect-from-linux.html
+    - title: Accessing from a Windows client
+      url: /merlin6/connect-from-windows.html
+    - title: Accessing from a MacOS client
+      url: /merlin6/connect-from-macos.html
     - title: Merlin6 Storage
       url: /merlin6/storage.html
     - title: Accessing Slurm Cluster
@@ -31,7 +37,7 @@ entries:
       url: /merlin6/transfer-data.html
     - title: NoMachine
       url: /merlin6/nomachine.html
-  - title: Merlin6 Slurm
+  - title: Job Submission
     folderitems:
     - title: Using PModules
       url: /merlin6/using-modules.html
@@ -39,6 +45,8 @@ entries:
       url: /merlin6/slurm-basics.html
     - title: Running Jobs
       url: /merlin6/running-jobs.html
+    - title: Interactive Jobs
+      url: /merlin6/interactive-jobs.html
     - title: Slurm Examples
       url: /merlin6/slurm-examples.html
     - title: Slurm Configuration

css/customstyles.css

@@ -2,6 +2,38 @@ body {
     font-size:15px;
 }
 
+/* HPCE Custom CSS styles - BEGIN */
+pre.terminal {
+  background-color: black;
+  color: white;
+  font-size: 75%;
+  font-family: Consolas,Monaco,Lucida Console,Liberation Mono,DejaVu Sans Mono,Bitstream Vera Sans Mono,Courier New, monospace;
+  width: 100%;
+  display: inline-block;
+}
+details {
+  border: 1px solid #347DBE;
+  border-radius: 4px;
+  padding: .5em .5em 0;
+  color: #347DBE
+}
+
+summary {
+  font-weight: bold;
+  margin: -.5em -.5em 0;
+  padding: .5em;
+}
+
+details[open] {
+  padding: .5em;
+}
+
+details[open] summary {
+  border-bottom: 1px solid #347DBE;
+  margin-bottom: .5em;
+}
+/* HPCE Custom CSS styles - END */
+
 .bs-callout {
     padding: 20px;
     margin: 20px 0;

pages/merlin6/02 accessing-merlin6/accessing-interactive-nodes.md

@@ -21,9 +21,6 @@ The Merlin6 login nodes are the following:
 | merlin-l-001.psi.ch     | yes | yes       | 2 x 22 | 2        | Intel Xeon Gold 6152  | 384GB  | 1.8TB NVMe | ``/scratch``        |
 | merlin-l-002.psi.ch     | yes | yes       | 2 x 22 | 2        | Intel Xeon Gold 6142  | 384GB  | 1.8TB NVMe | ``/scratch``        |
 | merlin-l-01.psi.ch      | yes | -         | 2 x 16 | 2        | Intel Xeon E5-2697Av4 | 512GB  | 100GB SAS  | ``/scratch``        |
-| ~~merlin-l-02.psi.ch~~  | -   | -         | 2 x 16 | 2        | Intel Xeon E5-2697Av4 | 512GB  | 100GB SAS  | ``/scratch``        |
-
-* **merlin-l-02.psi.ch** has an severe incident and is not available to users: HP is trying to locate and fix the problem.
 
 ---
 
@@ -31,22 +28,29 @@ The Merlin6 login nodes are the following:
 
 ### SSH Access
 
-For interactive command shell access, use an SSH client. We recommend to activate SSH's X11 forwarding to allow you to use graphical applications (e.g. a text editor. For more performant graphical access, refer to the sections below).
+For interactive command shell access, use an SSH client. We recommend to activate SSH's X11 forwarding to allow you to use graphical 
+applications (e.g. a text editor, but for more performant graphical access, refer to the sections below). X applications are supported 
+in the login nodes and X11 forwarding can be used for those users who have properly configured X11 support in their desktops, however:
 
-E.g. for Linux:
-
-```bash
-ssh -XY $username@merlin-l-01.psi.ch
-```
-
-X applications are supported in the login nodes and X11 forwarding can be used for those users who have properly configured X11 support in their desktops:
 * Merlin6 administrators **do not offer support** for user desktop configuration (Windows, MacOS, Linux).
    * Hence, Merlin6 administrators **do not offer official support** for X11 client setup.
-   * However, a generic guide for X11 client setup (Windows, Linux and MacOS) will be provided.
+   * Nevertheless, a generic guide for X11 client setup (*Linux*, *Windows* and *MacOS*) is provided below.
 * PSI desktop configuration issues must be addressed through  **[PSI Service Now](https://psi.service-now.com/psisp)** as an *Incident Request*.
    * Ticket will be redirected to the corresponding Desktop support group (Windows, Linux).
 
-### More efficient graphical access using a **NoMachine** client
+#### Accessing from a Linux client
+
+Refer to [{Accessing Merlin -> Accessing from Linux Clients}](/merlin6/connect-from-linux.html) for **Linux** SSH client and X11 configuration.
+
+#### Accessing from a Windows client
+
+Refer to [{Accessing Merlin -> Accessing from Windows Clients}](/merlin6/connect-from-windows.html) for **Windows** SSH client and X11 configuration.
+
+#### Accessing from a MacOS client
+
+Refer to [{Accessing Merlin -> Accessing from MacOS Clients}](/merlin6/connect-from-macos.html) for **MacOS** SSH client and X11 configuration.
+
+### Graphical access using **NoMachine** client
 
 X applications are supported in the login nodes and can run efficiently through a **NoMachine** client. This is the officially supported way to run more demanding X applications on Merlin6. The client software can be downloaded from [the Nomachine Website](https://www.nomachine.com/product&p=NoMachine%20Enterprise%20Client).
 

pages/merlin6/02 accessing-merlin6/connect-from-linux.md

@@ -0,0 +1,51 @@
+---
+title: Connecting from a Linux Client
+
+#tags:
+keywords: Linux, connecting, client, configuration, SSH, X11
+last_updated: 23 Oct 2019
+summary: "This document describes a recommended setup for a Linux client."
+sidebar: merlin6_sidebar
+permalink: /merlin6/connect-from-linux.html
+---
+
+## SSH without X11 Forwarding
+
+This is the standard method. Official X11 support is provided through [NoMachine](/merlin6/nomachine.html).
+For normal SSH sessions, use your SSH client as follows:
+
+```bash
+ssh $username@merlin-l-01.psi.ch
+ssh $username@merlin-l-001.psi.ch
+ssh $username@merlin-l-002.psi.ch
+```
+
+## SSH with X11 Forwarding
+
+Official X11 Forwarding support is through NoMachine. Please follow the document
+[Job Submission -> Interactive Jobs](/merlin6/interactive-jobs.html#Requirements) and
+[Accessing Merlin -> NoMachine](/merlin6/nomachine.html) for more details. However,
+we provide a small recipe for enabling X11 Forwarding in Linux.
+
+* For enabling client X11 forwarding, add the following to the start of ``~/.ssh/config``
+to implicitly add ``-Y`` to all ssh connections:
+
+   ```bash
+   ForwardAgent yes
+   ForwardX11Trusted yes
+   ```
+
+* Alternatively, you can add the option ``-Y`` to the ``ssh`` command. In example:
+
+   ```bash
+   ssh -Y $username@merlin-l-01.psi.ch
+   ssh -Y $username@merlin-l-001.psi.ch
+   ssh -Y $username@merlin-l-002.psi.ch
+   ```
+
+* For testing that X11 forwarding works, just run ``xclock``. A X11 based clock should
+popup in your client session:
+
+   ```bash
+   xclock
+   ```

pages/merlin6/02 accessing-merlin6/connect-from-macos.md

@@ -0,0 +1,51 @@
+---
+title: Connecting from a MacOS Client
+
+#tags:
+keywords: MacOS, connecting, client, configuration, SSH, X11
+last_updated: 23 Oct 2019
+summary: "This document describes a recommended setup for a MacOS client."
+sidebar: merlin6_sidebar
+permalink: /merlin6/connect-from-macos.html
+---
+
+## SSH without X11 Forwarding
+
+This is the standard method. Official X11 support is provided through [NoMachine](/merlin6/nomachine.html).
+For normal SSH sessions, use your SSH client as follows:
+
+```bash
+ssh $username@merlin-l-01.psi.ch
+ssh $username@merlin-l-001.psi.ch
+ssh $username@merlin-l-002.psi.ch
+```
+
+## SSH with X11 Forwarding
+
+Official X11 Forwarding support is through NoMachine. Please follow the document
+[Job Submission -> Interactive Jobs](/merlin6/interactive-jobs.html#Requirements) and
+[Accessing Merlin -> NoMachine](/merlin6/nomachine.html) for more details. However,
+we provide a small recipe for enabling X11 Forwarding in MacOS.
+
+* For enabling client X11 forwarding, add the following to the start of ``~/.ssh/config``
+to implicitly add ``-Y`` to all ssh connections:
+   
+   ```bash
+   ForwardAgent yes
+   ForwardX11Trusted yes
+   ```
+
+* Alternatively, you can add the option ``-Y`` to the ``ssh`` command. In example:
+
+   ```bash
+   ssh -Y $username@merlin-l-01.psi.ch
+   ssh -Y $username@merlin-l-001.psi.ch
+   ssh -Y $username@merlin-l-002.psi.ch
+   ```
+
+* For testing that X11 forwarding works, just run ``xclock``. A X11 based clock should
+popup in your client session.
+
+   ```bash
+   xclock
+   ```

pages/merlin6/02 accessing-merlin6/connect-from-windows.md

@@ -0,0 +1,49 @@
+---
+title: Connecting from a Windows Client
+
+#tags:
+keywords: Windows, connecting, client, configuration, SSH, X11
+last_updated: 23 Oct 2019
+summary: "This document describes a recommended setup for a Windows client."
+sidebar: merlin6_sidebar
+permalink: /merlin6/connect-from-windows.html
+---
+
+## SSH with PuTTY without X11 Forwarding
+
+PuTTY is one of the most common tools for SSH.
+
+Check, if the following software packages are installed on the Windows workstation by 
+inspecting the Start menu (hint: Use the search to save time):
+* PuTTY (should already be installed)
+* *[Optional]* Xming (needed for [SSH with X11 Forwarding](/merlin6/connect-from-windows.html#ssh-with-x11-forwarding))
+
+If they are missing, you can install them using the Software Kiosk icon on the Desktop.
+
+1. Start PuTTY
+
+2. *[Optional]* Enable ``xterm`` to have similar mouse behavour as in Linux:
+
+   ![Enable 'xterm']({{ "/images/PuTTY/Putty_Mouse_XTerm.png" }})
+
+3. Create session to a Merlin login node and *Open*:
+
+   ![Create Merlin Session]({{ "/images/PuTTY/Putty_Session.png" }})
+
+
+## SSH with PuTTY with X11 Forwarding
+
+Official X11 Forwarding support is through NoMachine. Please follow the document
+[Job Submission -> Interactive Jobs](/merlin6/interactive-jobs.html#Requirements) and
+[Accessing Merlin -> NoMachine](/merlin6/nomachine.html) for more details. However,
+we provide a small recipe for enabling X11 Forwarding in Windows.
+
+Check, if the **Xming** is installed on the Windows workstation by inspecting the 
+Start menu (hint: Use the search to save time). If missing, you can install it by
+using the Software Kiosk icon (should be on the Desktop). 
+
+1. Ensrue that a X server (**Xming**) is running. Otherwise, start it.
+
+2. Enable X11 Forwarding in your SSH client. In example, for Putty:
+
+   ![Enable X11 Forwarding in Putty]({{ "/images/PuTTY/Putty_X11_Forwarding.png" }})

pages/merlin6/03 Job Submission/interactive-jobs.md

@@ -0,0 +1,199 @@
+---
+title: Interactive Jobs
+#tags:
+keywords: interactive, X11, X, srun
+last_updated: 22 October 2019
+summary: "This document describes how to run interactive jobs as well as X based software."
+sidebar: merlin6_sidebar
+permalink: /merlin6/interactive-jobs.html
+---
+
+## Running interactive jobs
+
+There are two different ways for running interactive jobs in Slurm. This is possible by using
+the ``srun`` or the ``salloc`` commands.
+
+### srun
+
+Is run is used to run parallel jobs in the batch system. It can be used within a batch script
+(which can be run with ``sbatch``), or within a job allocation (which can be run with ``salloc``).
+Also, it can be used as a direct command (in example, from the login nodes).
+
+When used inside a batch script or during a job allocation, ``srun`` is constricted to the 
+amount of resources allocated by the ``sbatch``/``salloc`` commands. In ``sbatch``, usually 
+these resources are defined inside the batch script with the format ``#SBATCH <option>=<value>``. 
+In other words, if you define in your batch script or allocation 88 tasks (and 1 thread / core) 
+and 2 nodes, ``srun`` is constricted to these amount of resources (you can use less, but never 
+exceed those limits).
+
+When used from the login node, usually is used to run a specific command or software in an 
+interactive way. ``srun`` is a blocking process (it will block bash prompt until the ``srun`` 
+command finishes, unless you run it in background with ``&``). This can be very useful to run 
+interactive software which pops up a Window and then submits jobs or run sub-tasks in the 
+background (in example, **Relion**, **cisTEM**, etc.)
+
+Refer to ``man srun`` for exploring all possible options for that command.
+
+<details>
+<summary>[Show 'srun' example]: Running 'hostname' command on 3 nodes, using 2 cores (1 task/core) per node</summary>
+<pre class="terminal code highlight js-syntax-highlight plaintext" lang="plaintext" markdown="false">
+(base) [caubet_m@merlin-l-001 ~]$ srun --clusters=merlin6 --ntasks=6 --ntasks-per-node=2 --nodes=3 hostname
+srun: job 135088230 queued and waiting for resources
+srun: job 135088230 has been allocated resources
+merlin-c-102.psi.ch
+merlin-c-102.psi.ch
+merlin-c-101.psi.ch
+merlin-c-101.psi.ch
+merlin-c-103.psi.ch
+merlin-c-103.psi.ch
+</pre>
+</details>
+
+### salloc
+
+``salloc`` is used to obtain a Slurm job allocation (a set of nodes). Once job is allocated,
+users are able to execute interactive command(s). Once finished (``exit`` or ``Ctrl+D``), 
+the allocation is released. 
+
+Please, not that ``salloc`` is by default a blocking process, and once the resources get 
+allocated user gets the bash prompt back. Once prompt is back, a new bash shell is created, 
+and user can run any ``srun`` commands, which will run tasks on the allocated resources. Once
+finished, exiting the bash shell will release the allocation.
+
+<details>
+<summary>[Show 'salloc' example]: Allocating 2 cores (1 task/core) in 2 nodes (1 core/node)</summary>
+<pre class="terminal code highlight js-syntax-highlight plaintext" lang="plaintext" markdown="false">
+(base) [caubet_m@merlin-l-001 ~]$ salloc --clusters=merlin6 --ntasks=2 --nodes=2
+salloc: Pending job allocation 135087844
+salloc: job 135087844 queued and waiting for resources
+salloc: job 135087844 has been allocated resources
+salloc: Granted job allocation 135087844
+
+(base) [caubet_m@merlin-l-001 ~]$ srun hostname
+merlin-c-120.psi.ch
+merlin-c-119.psi.ch
+
+(base) [caubet_m@merlin-c-119 ~]$ exit
+logout
+Connection to merlin-c-119 closed.
+
+(base) [caubet_m@merlin-l-001 ~]$ exit
+exit
+salloc: Relinquishing job allocation 135087844
+</pre>
+</details>
+
+## Running interactive jobs with X11 support
+
+### Requirements
+
+#### Graphical access
+
+[NoMachine](/merlin6/nomachine.html) is the official supported service for graphical 
+access in the Merlin cluster. This service is running on the login nodes. Check the 
+document [Accessing Merlin -> NoMachine](/merlin6/nomachine.html) for details about 
+how to connect to the **NoMachine** service in the Merlin cluster.
+
+For other non officially supported graphical access (X11 forwarding):
+
+* For Linux clients, please follow [{Accessing Merlin -> Accessing from Linux Clients}](/merlin6/connect-from-linux.html)
+* For Windows clients, please follow [{Accessing Merlin -> Accessing from Windows Clients}](/merlin6/connect-from-windows.html)
+* For MacOS clients, please follow [{Accessing Merlin -> Accessing from MacOS Clients}](/merlin6/connect-from-macos.html)
+
+#### Enable SSH Keys authentication
+
+For running ``srun`` with **X11** support (``srun --x11``) , you need to setup RSA keys properly. 
+
+1. Generate the RSA keys as follows:
+
+   ```bash
+   ssh-keygen -t rsa
+   ```
+
+   You will be requested for an *optional* passphrase. Entering it, provides more security (if somebody steals your private key he will 
+   need to know the passphrase, however every time you use RSA keys you will need to type it). Whether to set a passphrase or not is up
+   to the users.
+
+2. Add the public key to the ``~/.ssh/authorized_keys`` file 
+
+   ```bash
+   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
+   ```
+
+3. Ensure that ``~/.ssh/authorized_keys`` has proper permissions:
+
+   ```bash
+   chmod 600 ~/.ssh/authorized_keys
+   ```
+
+<details>
+<summary>[Show 'ssh-keygen' example]: Generate RSA keys with default key filenames</summary>
+<pre class="terminal code highlight js-syntax-highlight plaintext" lang="plaintext" markdown="false">
+(base) [caubet_m@merlin-l-001 .ssh]$ ssh-keygen -t rsa
+Generating public/private rsa key pair.
+Enter file in which to save the key (/psi/home/caubet_m/.ssh/id_rsa): 
+Enter passphrase (empty for no passphrase): 
+Enter same passphrase again: 
+Your identification has been saved in /psi/home/caubet_m/.ssh/id_rsa.
+Your public key has been saved in /psi/home/caubet_m/.ssh/id_rsa.pub.
+The key fingerprint is:
+SHA256:AMvGhBWxXs1MXHvwTpvXCOpjUZgy30E+5V38bcj4k2I caubet_m@merlin-l-001.psi.ch
+The key's randomart image is:
++---[RSA 2048]----+
+|   o*o ...o . ...|
+|  .+ + =.  O o .o|
+|    * o * + Xo..+|
+|   o . . + B.*oo+|
+|    .   S + =.oo.|
+|         . .E.+  |
+|          +. . . |
+|         . .     |
+|                 |
++----[SHA256]-----+
+
+(base) [caubet_m@merlin-l-001 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
+(base) [caubet_m@merlin-l-001 .ssh]$ chmod 600 ~/.ssh/authorized_keys
+</pre>
+</details>
+
+### 'srun' with x11 support
+
+Once RSA keys are setup, you can run any windows based application. For that, you need to 
+add the option ``--x11`` to the ``srun`` command. In example:
+
+```bash
+srun --x11 xclock
+```
+
+will popup a X11 based clock.
+
+In the same manner, you can create a bash shell with x11 support. For doing that, you need
+to add the option ``--pty`` to the ``srun --x11`` command. Once resource is allocated, from
+there you can interactively run X11 and non-X11 based commands.
+
+```bash
+srun --x11 --pty bash
+```
+
+<details>
+<summary>[Show 'srun' with X11 support examples]</summary>
+<pre class="terminal code highlight js-syntax-highlight plaintext" lang="plaintext" markdown="false">
+(base) [caubet_m@merlin-l-001 ~]$ srun --x11 xclock
+srun: job 135095591 queued and waiting for resources
+srun: job 135095591 has been allocated resources
+
+(base) [caubet_m@merlin-l-001 ~]$ 
+
+(base) [caubet_m@merlin-l-001 ~]$ srun --x11 --pty bash
+srun: job 135095592 queued and waiting for resources
+srun: job 135095592 has been allocated resources
+
+(base) [caubet_m@merlin-c-205 ~]$ xclock
+
+(base) [caubet_m@merlin-c-205 ~]$ echo "This was an example"
+This was an example
+
+(base) [caubet_m@merlin-c-205 ~]$ exit
+exit
+</pre>
+</details>