HPCE/gitea-pages
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Activity

Update Mrlin6 documentation with latest changes, added new pages

Browse Source
This commit is contained in:
caubet_m
2019-06-13 08:41:29 +02:00
parent caa63db616
commit caa558a090
9 changed files with 942 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

300
pages/merlin6-user-guide/accessing-merlin6.md Normal file
View File

@ -0,0 +1,300 @@
---
layout: default
title: Accessing Merlin6
parent: Merlin6 User Guide
nav_order: 4
---
# Accessing Merlin6
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
---
## Requesting Merlin6 Access
PSI users with their Linux account belonging to the **svc-cluster_merlin6** group are allowed to use Merlin6.
Registration for **Merlin6** access *must be done* through **[PSI Service Now](https://psi.service-now.com/psisp)**:
* Please open a ticket as *Incident Request*, with subject: ``[Merlin6] Access Request for user '<username>'``
* Text content example:
> Dear HelpDesk,
>
> my name is [Name] [Surname] with PSI username [username] and I would like to request access to the Merlin6 cluster.
>
> Please add me to the following Unix groups:
> * 'svc-cluster_merlin6'
>
> Thanks a lot,
> <Name> <Username>
### Requesting extra Unix groups
* Some users may require to be added to some extra specific Unix groups.
* In example, some BIO groups may belong to a specific BIO group.
* Extra groups can be added in the *Incident Request* described in [Requesting Merlin6 Access](#Requesting-Merlin6-Access).
* Alternatively, this step can be done later in the future on a different **[PSI Service Now](https://psi.service-now.com/psisp)** ticket.
---
## Requesting Merlin5 Access
Merlin5 computing nodes will be available for some time as a **best efford** service. If you need to use the old resources, users should belong to the **svc-cluster_merlin5** Unix Group.
Registration for **Merlin5** access *must be done* through **[PSI Service Now](https://psi.service-now.com/psisp)**
* Please open a ticket as *Incident Request*, with subject: ``[Merlin5] Access Request for user '<username>'``
* Text content example:
> Dear HelpDesk,
>
> my name is [Name] [Surname] with PSI username [username] and I would like to request access to the old Merlin5 cluster.
>
> Please add me to the following Unix groups:
> * 'svc-cluster_merlin5'
>
> Thanks a lot,
> <Name> <Username>
* As described before, you can add in the *Incident Request* which extra Unix groups should be added to your user.
---
## Accessing Login/Interactive Nodes
The Merlin6 login nodes are the following:
* <tt><b>merlin-l-01.psi.ch</b></tt> - SSH Access
* *SSH* Access
* **Hardware description:** 32 cores (2 x 16 core Intel Xeon E5-2697A v4), 512GB RAM, 100GB ``/scratch`` on SAS disk.
* <tt><b>merlin-l-02.psi.ch</b></tt>
* *SSH* & **NoMachine** Access
* **Hardware description:** 32 cores (2 x 16 core Intel Xeon E5-2697A v4), 512GB RAM, 100GB ``/scratch`` on SAS disk.
<!--* <tt><b>merlin-l-001.psi.ch</b></tt> - SSH Access -->
<!--* <tt><b>merlin-l-002.psi.ch</b></tt> - SSH Access -->
Login nodes are the official service for accessing the Merlin6 cluster. From there users can submit jobs to the Slurm batch system as well as visualize or compile their software.
### SSH Access
For interactive command access, Use SSH for accessing the login nodes. In example:
```bash
ssh -XY [username]@merlin-l-01.psi.ch
```
X applications are supported in the login nodes and can be opened through SSH for those users who have properly configured X11 in their desktops. SSH options ``-XY`` are mandatory for that.
* Merlin6 administrators **do not offer support** with configuration for users Desktops (Windows, MAC and Linux). Hence, Merlin6 administrators **do not offer support** for X11 client setup.
* Any desktop configuration issues must be addressed through **[PSI Service Now](https://psi.service-now.com/psisp)** as an *Incident Request*.
* Ticket will be redirected to the corresponding Desktop support group.
### NoMachine Access
X applications are supported in the login nodes and can run through NoMachine. This service is officially supported in the Merlin6 cluster and is the official X service.
* NoMachine client installation/configuration support has to be done through **[PSI Service Now](https://psi.service-now.com/psisp)** as an *Incident Request*.
* Ticket will be redirected to the corresponding support group.
---
## Accessing Merlin6 data
### Merlin6 directory structure
Merlin6 contain the following directories available for users:
* ``/psi/home/<username>``: private user **home** directory
* ``/data/user/<username>``: private user **home** directory
* ``/data/project/general/<projectname>``: Shared **Project** directory
* ``/scratch``: Local *scratch* disk.
* ``/shared-scratch``: Shared *scratch* disk.
#### User home directory
The user home directory can be found on login nodes and computing nodes under the ``/psi/home/<username>`` dirctory. This is the default directory users will land when login in to any Merlin6 machine.
Home directories are part of the PSI NFS Central Home storage provided by AIT, however quota administration for the Merlin6 cluster is delegated to Merlin6 administrators.
Home directory policies:
* **Per User-based Quota policy**:
* **Soft**: 10GB
* **Hard**: 11GB
* Quota can only be increased when strictly justified.
* Check home quota with the command: ``quota -s``
* **Backup policy**:
* **Daily snapshots for 1 week**: users can recover up to 1 week of their lost data.
* **Snaphot location**: ``/psi/home/.snapshop/<username>``
* **Restrictions**
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the home directories for IO intensive tasks (in example, IO intensive data access during job runtime).
* Use ``/scratch``, ``/shared-scratch``, ``/data/user`` or ``/data/project`` for this purpose.
#### User data directory
The user data directory can be found on login nodes and computing nodes under the ``/data/user/<username>`` dirctory.
This storage is intended for fast IO access and keeping large amount of private data.
User data directories are part of the Merlin6 storage cluster and technology is based on GPFS.
User data directory policies:
* **Per User-based Quota policy** (as known as GPFS USR Block Limits):
* **Soft**: 1TB
* **Hard**: 1.074TB
* Block quota limits can not be increased. For extra space, project must exist / be created (``/data/project/<projectname>``)
* Check data quota with the command: ``mmlsquota -u <username> --block-size auto merlin-user``
* **Per User-based Number of Files Quota policy** (as known as GPFS USR File Limits):
* **Soft**: 1,048,576
* **Hard**: 1,126,400
* File quota can be increased. For extra files, contact Merlin6 administrators.
* Check data quota with the command: ``mmlsquota -u <username> --block-size auto merlin-user``
* **Backup policy**:
* No backups: users are responsible for managing the backups of their data directories.
* **Restrictions**
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the data directories as ``scratch`` area during a job runtime.
* Use ``/scratch``, ``/shared-scratch`` for this purpose.
* For temporary interactive user data (from a login node), is allowed to use it as ``scratch``-like area (in example, compiling, deploying tars, etc.)
#### Project data directory
The project data directory can be found on login nodes and computing nodes under the ``/data/user/<username>`` dirctory.
This storage is intended for fast IO access and keeping large amount of private data, but also for sharing data amogst
different users sharing a project. Creating a project is the way in where users can expand his storage space and will
optimize the usage of the storage (by avoiding for instance, duplicated data for different users).
Is **highly** recommended the use of a project when multiple persons are involved in the same project managing similar/common data.
Quotas are defined in a *group* and *fileset* basis: Unix Group name must exist for a specific project or must be created for any new project.
Contact the Merlin6 administrators for more information about that.
Project data directories are part of the Merlin6 storage cluster and technology is based on GPFS.
Project data directory policies:
* **Per Group and Fileset-based Quota policy** (as known as GPFS GRP Block Limits):
* **Soft**: 1TB
* **Hard**: 1.074TB
* Block quota limits can be increased on demand when strictly justified and it can be defined when a project is created. For extra space, contact Merlin6 administrators.
* Check data quota with the command: ``mmlsquota -g <groupname> --block-size auto merlin-proj``
* **Per Group and Fileset-based Number of Files Quota policy** (as known as GPFS GRP File Limits):
* **Soft**: 1,048,576
* **Hard**: 1,126,400
* File quota can be increased on demand, and it can be defined when a new project is created. For extra files, contact Merlin6 administrators.
* Check data quota with the command: ``mmlsquota -g <groupname> --block-size auto merlin-proj``
* **Backup policy**:
* No backups: users are responsible for managing the backups of their data directories.
* **Restrictions**
* Read **[Important: Code of Conduct](## Important: Code of Conduct)** for more information about Merlin6 policies.
* Is **forbidden** to use the data directories as ``scratch`` area during a job runtime.
* Use ``/scratch``, ``/shared-scratch`` for this purpose.
* For temporary interactive user data (from a login node), is allowed to use it as ``scratch``-like area (in example, compiling, deploying tars, etc.)
#### Scratch directories
There are two different types of scratch disk: **local** (``/scratch``) and **shared** (``/shared-scratch``). Specific details of each type is described below.
Usually **shared** scratch will be used for those jobs which need to access to a common shared space storing temporary files, while **local** scratch should be used by those jobs which
need to store temporary files not used by jobs running on other computing nodes.
**local** scratch in computing nodes provide a huge number of IOPS thanks to the NVMe technology, while **shared** scratch, despite is also very fast, is an external storage with more latency.
By default, *always* use **local** first and only use **shared** if you specific use case needs a shared scratch area.
##### Local Scratch
Local scratch is used for creating temporary files/directories needed by running jobs. Temporary files *must be deleted at the end of the job by the user*. Remaining files will be
deleted by the system if detected. Use always **local** by default and only use *shared* when **local** does not fit to you.
Local scratch is available on login and computing nodes, and its size depends on the host:
* **Merlin6 login nodes:**
* **Old login nodes:** a */scratch* partition of ~100GB is available on each login node ``merlin-l-01`` and ``merlin-l-02``. ``/scratch`` is mounted on *SAS* disks
* **New login nodes:** a */scratch* partition of ~1.6TB is available on each login node ``merlin-l-001`` and ``merlin-l-002``. ``/scratch`` is mounted on extremely fast *NVMe Flash* disks.
* **Merlin5 computing nodes**: a */scratch* partition of ~50GB is available on each computing node. ``/scratch`` is mounted on *SAS* disks for Merlin5 nodes.
* **Merlin6 computing nodes**: a */scratch* partition of ~1.2TB is available on each computing node. ``/scratch`` is mounted on extremly fast *NVMe Flash* disks for Merlin6 nodes.
#### Shared Scratch
Shared scratch (``/shared-scratch``) is used for creating temporary files/directories needed by running jobs. Temporary files *must be deleted at the end of the job by the user*.
Remaining files will be deleted by the system if detected. Only use **shared** scratch when you need creating **shared** temporary files, otherwise use **local** by default.
``/shared-scratch`` is only available in the computing nodes, and its current size is 50TB. ``/shared-scratch`` is an independent GPFS filesystem in the new Merlin6 GPFS storagle cluster,
and it can be increased if necessary in the future.
---
## Using the Slurm batch system
Clusters at PSI use the [Slurm Workload Manager](http://slurm.schedmd.com/) as the batch system technology for managing and scheduling jobs.
Historically, *Merlin4* and *Merlin5* also used Slurm. In the same way, **Merlin6** has been also configured with this batch system.
Slurm has been installed in a **multi-clustered** configuration, allowing to integrate multiple clusters in the same batch system.
For understanding the Slurm configuration setup in the cluster, sometimes may be useful to check the following files:
* ``/etc/slurm/slurm.conf`` - can be found in the login nodes and computing nodes.
* ``/etc/slurm/cgroup.conf`` - can be found in the computing nodes, is also propagated to login nodes for user read access.
* ``/etc/slurm/gres.conf`` - can be found in the GPU nodes, is also propgated to login nodes and computing nodes for user read access.
The previous configuration files which can be found in the login nodes, correspond exclusively to the **merlin6** cluster configuration files.
Configuration files for the old **merlin5** cluster must be checked directly on any of the **merlin5** computing nodes: these are not propagated
to the **merlin6** login nodes.
### About Merlin5 & Merlin6
The new Slurm cluster is called **merlin6**. However, the old Slurm *merlin* cluster will be kept for some time, and it has been renamed to **merlin5**.
It will allow to keep running jobs in the old computing nodes until users have fully migrated their codes to the new cluster.
From July 2019, **merlin6** becomes the **default cluster** and any job submitted to Slurm will be submitted to that cluster. Users can keep submitting to
the old *merlin5* computing nodes by using the option ``--cluster=merlin5``.
In this documentation is only explained the usage of the **merlin6** Slurm cluster.
### Using Slurm 'merlin6' cluster
Basic usage for the **merlin6** cluster will be detailed here. For advanced usage, please use the following document [LINK TO SLURM ADVANCED CONFIG]()
#### Merlin6 Node definition
The following table show default and maximum resources that can be used per node:
| Nodes | Def.#CPUs | Max.#CPUs | Def.Mem/CPU | Max.Mem/CPU | Max.Mem/Node | Max.Swap | Def.#GPUs | Max.#GPUs |
|:---------------------------------- | ---------:| ---------:| -----------:| -----------:| ------------:| --------:| --------- | --------- |
| merlin-c-[001-022,101-122,201-222] | 1 core | 44 cores | 8000 | 352000 | 352000 | 10000 | N/A | N/A |
| merlin-g-[001] | 1 core | 8 cores | 8000 | 102498 | 102498 | 10000 | 1 | 2 |
| merlin-g-[002-009] | 1 core | 10 cores | 8000 | 102498 | 102498 | 10000 | 1 | 4 |
If nothing is specified, by default each core will use up to 8GB of memory. More memory per core can be specified with the ``--mem=<memory>`` option,
and maximum memory allowed is ``Max.Mem/Node``.
In *Merlin6*, memory is considered a Consumable Resource, as well as the CPU.
#### Merlin6 Slurm partitions
Partition can be specified when submitting a job with the ``--partition=<partitionname>`` option.
The following *partitions* (also known as *queues*) are configured in Slurm:
| Partition | Default Partition | Default Time | Max Time | Max Nodes | Priority |
|:----------- | ----------------- | ------------ | -------- | --------- | -------- |
| **general** | true | 1 day | 1 week | 50 | low |
| **daily** | false | 1 day | 1 day | 60 | medium |
| **hourly** | false | 1 hour | 1 hour | unlimited | highest |
General is the *default*, so when nothing is specified job will be by default assigned to that partition. General can not have more than 50 nodes
running jobs. For **daily** this limitation is extended to 60 nodes while for **hourly** there are no limits. Shorter jobs have more priority than
longer jobs, hence in general terms would be scheduled before (however, other factors such like user fair share value can affect to this decision).
#### Merlin6 User limits
By default, users can not use more than 528 cores at the same time (Max CPU per user). This limit applies for the **general** and **daily** partitions. For the **hourly** partition, there is no restriction.
These limits are softed for the **daily** partition during non working hours and during the weekend as follows:
| Partition | Mon-Fri 08h-18h | Sun-Thu 18h-0h | From Fri 18h to Sun 8h | From Sun 8h to Mon 18h |
|:----------- | --------------- | -------------- | ----------------------- | ---------------------- |
| **general** | 528 | 528 | 528 | 528 |
| **daily** | 528 | 792 | Unlimited | 792 |
| **hourly** | Unlimited | Unlimited | Unlimited | Unlimited |

35
pages/merlin6-user-guide/code-of-codunct.md Normal file
View File

@ -0,0 +1,35 @@
---
layout: default
title: Code Of Conduct
parent: Merlin6 User Guide
nav_order: 2
---
# Code Of Conduct
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
---
The basic principle is courtesy and consideration for other users.
* Merlin6 is a shared resource, not your laptop, therefore you are kindly requested to behave in a way you would be happy to see other users behaving towards you.
* Basic shell programming skills in Linux/UNIX environment is a must-have requirement for HPC users; a proficiency in shell programming would be greatly beneficial.
* The login nodes are for development and quick testing:
* Is **strictly forbidden to run production jobs** on the login nodes.
* Is **forbidden to run long processes** occupying big part of the resources.
* *Any miss-behaving running processes according to these rules will be killed.*
* All production jobs should be submitted using the batch system.
* Make sure that no broken or run-away processes are left when your job is done. Keep the process space clean on all nodes.
* During the runtime of a job, is mandatory to make use of the ``scratch`` and ``/shared-scratch`` partitions for temporary data. This also applies to temporary data generated on login nodes:
* Is **forbidden** to use the ``/data/user``, ``/data/project`` or ``/psi/home/`` for that purpose.
* Always remove files you do not need any more (e.g. core dumps, temporary files) as early as possible. Keep the disk space clean on all nodes.
* Read description in **[Merlin6 directory structure](### Merlin6 directory structure)** for correct usage of each partition type.
The system administrator has the right to block the access to Merlin6 for an account violating the Code of Conduct, in which case the issue will be escalated to the user's supervisor.
The system administrator has the right to delete files in the *scratch* directories exceeding the above rules.

109
pages/merlin6-user-guide/hardware-and-software-description.md Normal file
View File

@ -0,0 +1,109 @@
---
layout: default
title: Hardware And Software Description
parent: Merlin6 User Guide
nav_order: 3
---
# Hardware And Software Description
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
---
## Computing Nodes
The new Merlin6 cluster contains an homogeneous solution based on *three* HP Apollo k6000 systems. Each HP Apollo k6000 chassis contains 22 HP XL320k Gen10 blades. However,
each chassis can contain up to 24 blades, so is possible to upgradew with up to 2 nodes per chassis.
Each HP XL320k Gen 10 blade can contain up to two processors of the latest Intel® Xeon® Scalable Processor family. The hardware and software configuration is the following:
* 3 x HP Apollo k6000 chassis systems, each one:
* 22 x [HP Apollo XL230K Gen10](https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=a00016634enw), each one:
* 2 x *22 core* [Intel® Xeon® Gold 6152 Scalable Processor](https://ark.intel.com/products/120491/Intel-Xeon-Gold-6152-Processor-30-25M-Cache-2-10-GHz-) (2.10-3.70GHz).
* 12 x 32 GB (384 GB in total) of DDR4 memory clocked 2666 MHz.
* Dual Port !InfiniBand !ConnectX-5 EDR-100Gbps (low latency network); one active port per chassis.
* 1 x 1.6TB NVMe SSD Disk
* ~300GB reserved for the O.S.
* ~1.2TB reserved for local fast scratch ``/scratch``.
* Software:
* RedHat Enterprise Linux 7.6
* [Slurm](https://slurm.schedmd.com/) v18.08
* [GPFS](https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/ibmspectrumscale502_welcome.html) v5.0.2
* 1 x [HPE Apollo InfiniBand EDR 36-port Unmanaged Switch](https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=a00016643enw)
* 24 internal EDR-100Gbps ports (1 port per blade for internal low latency connectivity)
* 12 external EDR-100Gbps ports (for external for internal low latency connectivity)
---
## Login Nodes
### merlin-l-0[1,2]
Two login nodes are inherit from the previous Merlin5 cluster: ``merlin-l-01.psi.ch``, ``merlin-l-02.psi.ch``. The hardware and software configuration is the following:
* 2 x HP DL380 Gen9, each one:
* 2 x *16 core* [Intel® Xeon® Processor E5-2697AV4 Family](https://ark.intel.com/products/91768/Intel-Xeon-Processor-E5-2697A-v4-40M-Cache-2-60-GHz-) (2.60-3.60GHz)
* ``merlin-l-01.psi.ch`` hyper-threading disabled
* ``merlin-l-02.psi.ch`` hyper-threading enabled
* 16 x 32 GB (512 GB in total) of DDR4 memory clocked 2400 MHz.
* Dual Port Infiniband !ConnectIB FDR-56Gbps (low latency network).
* Software:
* RedHat Enterprise Linux 7.6
* [Slurm](https://slurm.schedmd.com/) v18.08
* [GPFS](https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/ibmspectrumscale502_welcome.html) v5.0.2
### merlin-l-00[1,2]
Two new login nodes are available in the new cluster: ``merlin-l-001.psi.ch``, ``merlin-l-002.psi.ch``. The hardware and software configuration is the following:
* 2 x HP DL380 Gen10, each one:
* 2 x *22 core* [Intel® Xeon® Gold 6152 Scalable Processor](https://ark.intel.com/products/120491/Intel-Xeon-Gold-6152-Processor-30-25M-Cache-2-10-GHz-) (2.10-3.70GHz).
* Hyper-threading disabled.
* 24 x 16GB (384 GB in total) of DDR4 memory clocked 2666 MHz.
* Dual Port Infiniband !ConnectX-5 EDR-100Gbps (low latency network).
* Software:
* [NoMachine Terminal Server](https://www.nomachine.com/)
* Currently only on: ``merlin-l-001.psi.ch``.
* RedHat Enterprise Linux 7.6
* [Slurm](https://slurm.schedmd.com/) v18.08
* [GPFS](https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.2/ibmspectrumscale502_welcome.html) v5.0.2
---
## Storage
The storage node is based on the [Lenovo Distributed Storage Solution for IBM Spectrum Scale](https://lenovopress.com/lp0626-lenovo-distributed-storage-solution-for-ibm-spectrum-scale-x3650-m5).
The solution is equipped with 334 x 10TB disks providing a useable capacity of 2.316 PiB (2.608PB). THe overall solution can provide a maximum read performance of 20GB/s.
* 1 x Lenovo DSS G240, composed by:
* 2 x ThinkSystem SR650, each one:
* 2 x Dual Port Infiniband ConnectX-5 EDR-100Gbps (low latency network).
* 2 x Dual Port Infiniband ConnectX-4 EDR-100Gbps (low latency network).
* 1 x ThinkSystem RAID 930-8i 2GB Flash PCIe 12Gb Adapter
* 1 x ThinkSystem SR630
* 1 x Dual Port Infiniband ConnectX-5 EDR-100Gbps (low latency network).
* 1 x Dual Port Infiniband ConnectX-4 EDR-100Gbps (low latency network).
* 4 x Lenovo Storage D3284 High Density Expansion Enclosure, each one:
* Holds 84 x 3.5" hot-swap drive bays in two drawers. Each drawer has three rows of drives, and each row has 14 drives.
* Each drive bay will contain a 10TB Helium 7.2K NL-SAS HDD.
* 2 x Mellanox SB7800 InfiniBand 1U Switch for High Availability and fast access to the storage with very low latency. Each one:
* 36 EDR-100Gbps ports
---
## Network
Merlin6 cluster connectivity is based on the [Infiniband](https://en.wikipedia.org/wiki/InfiniBand) technology. This allows fast access with very low latencies to the data as well as running
extremely efficient MPI-based jobs:
* Connectivity amongst different computing nodes on different chassis ensures up to 1200Gbps of aggregated bandwidth.
* Inter connectivity (communication amongst computing nodes in the same chassis) ensures up to 2400Gbps of aggregated bandwidth.
* Communication to the storage ensures up to 800Gbps of aggregated bandwidth.
Merlin6 cluster currently contains 5 Infiniband Managed switches and 3 Infiniband Unmanaged switches (one per HP Apollo chassis):
* 1 * MSX6710 (FDR) for connecting old GPU nodes, old login nodes and MeG cluster to the Merlin6 cluster (and storage). No High Availability mode possible.
* 2 * MSB7800 (EDR) for connecting Login Nodes, Storage and other nodes in High Availability mode.
* 3 * HP EDR Unmanaged switches, each one embedded to each HP Apollo k6000 chassis solution.
* 2 * MSB7700 (EDR) are the top switches, interconnecting the Apollo unmanaged switches and the managed switches (MSX6710, MSB7800).

92
pages/merlin6-user-guide/known-problems-and-troubleshooting.md Normal file
View File

@ -0,0 +1,92 @@
---
layout: default
title: Known Problems and Troubleshooting
parent: Merlin6 User Guide
nav_order: 7
---
# Known Problems and Troubleshooting
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
---
## Known Problems
### Paraview, ANSYS and OpenGL
Try to use X11(mesa) driver for Paraview and ANSYS instead of OpenGL:
```bash
# ANSYS
module load ANSYS
fluent -driver x11
# ParaView
module load paraview
paraview --mesa
```
###+ Illegal instructions
It may happened that your code, compiled on one machine will not be executed on another throwing exception like "(Illegal instruction)".
Check (with "hostname" command) on which of the node you are and compare it with the names from first item. We observe few applications
that can't be run on merlin-c-01..16 because of this problem (notice that these machines are more then 5 years old). Hint: you may
choose the particular flavour of the machines for your slurm job, check the "--cores-per-node" option for sbatch:
```bash
sbatch --cores-per-socket=8 Script.sh # will filter the selection of the machine and exclude the oldest one, merlin-c-01..16
```
## Troubleshooting
### Before asking for help
Please, if you have problems running jobs and you want to report something or just ask for help,
please gather and attach in advance the following information:
* Unix username and session (``who am i`` command output)
* Environment settings (``env`` command output)
* Slurm batch script location (path to script and input/output files)
* Slurm job_id (``id`` is returned on ``sbatch``/``salloc`` command, but also can be taken from ``squeue`` commmand)
### Troubleshooting SSH
Use the ssh command with the "-vvv" option and copy and paste (no screenshot please)
the output to your request in Service-Now. Example
```bash
ssh -Y -vvv bond_j@merlin-l-01
```
### Troubleshooting SLURM
If one copies Slurm commands or batch scripts from another cluster,
they may need some changes (often minor) to run successfully on Merlin5.
Examine carefully the error message, especially concerning the options
used in the slurm commands.
Try to submit jobs using the examples given in the section "Using Batch System to Submit Jobs to Merlin5".
If you can run successfully an example for a type of job (!OpenMP, MPI) similar to your one,
try to edit the example to run your application.
If the problem remains, then, in your request in Service-Now, describe the problem in details that
are needed to reproduce it. Include the output of the following commands:
```bash
date
hostname
pwd
module list
# All slurm commands used with the corresponding output
```
Do not delete any output and error files generated by Slurm.
Make a copy of the failed job script if you like to edit it meanwhile.

13
pages/merlin6-user-guide/merlin6-slurm/merlin6-slurm.md Normal file
View File

@ -0,0 +1,13 @@
---
layout: default
title: Merlin6 Slurm
parent: Merlin6 User Guide
nav_order: 5
has_children: true
permalink: /docs/merlin6-user-guide/merlin6-slurm/merlin6-slurm.html
---
# Merlin6 Slurm
Wellcome to the PSI Merlin6 Slurm Cluster.

152
pages/merlin6-user-guide/merlin6-slurm/slurm-basic-commands.md Normal file
View File

@ -0,0 +1,152 @@
---
layout: default
title: Slurm Basic Commands
parent: Merlin6 Slurm
grand_parent: Merlin6 User Guide
nav_order: 1
---
# Slurm Basic Commands
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
---
## Basic commands
Useful commands for the slurm:
```bash
sinfo # to see the name of nodes, their occupancy, name of slurm partitions, limits (try out with "-l" option)
squeue # to see the currently running/waiting jobs in slurm (additional "-l" option may also be useful)
sbatch Script.sh # to submit a script (example below) to the slurm.
srun command # to submit a command to Slurm. Same options as in 'sbatch' can be used.
salloc # to allocate computing nodes. Useful for running interactive jobs (ANSYS, Python Notebooks, etc.).
scancel job_id # to cancel slurm job, job id is the numeric id, seen by the squeue
```
Other advanced basic commands:
```bash
sinfo -N -l # list nodes, state, resources (number of CPUs, memory per node, etc.), and other information
sshare -a # to list shares of associations to a cluster
sprio -l # to view the factors that comprise a job's scheduling priority (add -u <username> for filtering user)
```
---
## Basic slurm example
You can copy-paste the following example in a file file called ``mySlurm.batch``.
Some basic parameters are explained in the example.
Please notice that ``#`` is an enabled option while ``##`` is a commented out option (no effect).
```bash
#!/bin/sh
#SBATCH --partition=daily # name of slurm partition to submit. Can be 'general' (default if not specified), 'daily', 'hourly'.
#SBATCH --job-name="mySlurmTest" # name of the job. Useful when submitting different types of jobs for filtering (i.e. 'squeue' command)
#SBATCH --time=0-12:00:00 # time limit. Here is shortened to 12 hours (default and max for 'daily' is 1 day).
#SBATCH --exclude=merlin-c-001 # exclude which nodes you don't want to submit
#SBATCH --nodes=10 # number of nodes you want to allocate the job
#SBATCH --ntasks=440 # number of tasks to run
##SBATCH --exclusive # enable if you need exclusive usage of a node. If this option is not specified, nodes are shared by default.
##SBATCH --ntasks-per-node=32 # number of tasks per node. Each Apollo node has 44 cores, using less with exclusive mode may help to turbo boost CPU frequency. If this option is enabled, setup --ntasks and --nodes according to it.
module load gcc/8.3.0 openmpi/3.1.3
echo "Example no-MPI:" ; hostname # will print one hostname per node
echo "Example MPI:" ; mpirun hostname # will print one hostname per ntask
```
### Submitting a job
Submit job to slurm and check it's status:
```bash
sbatch mySlurm.batch # submit this job to slurm
squeue # check its status
```
---
## Advanced slurm test script
Copy-paste the following example in a file called myAdvancedTest.batch):
```bash
#!/bin/bash
#SBATCH --partition=merlin # name of slurm partition to submit
#SBATCH --time=2:00:00 # limit the execution of this job to 2 hours, see sinfo for the max. allowance
#SBATCH --nodes=2 # number of nodes
#SBATCH --ntasks=24 # number of tasks
module load gcc/8.3.0 openmpi/3.1.3
module list
echo "Example no-MPI:" ; hostname # will print one hostname per node
echo "Example MPI:" ; mpirun hostname # will print one hostname per ntask
```
In the above example are specified the options ``--nodes=2`` and ``--ntasks=24``. This means that up 2 nodes are requested,
and is expected to run 24 tasks. Hence, 24 cores are needed for running that job. Slurm will try to allocate a maximum of 2 nodes,
both together having at least 24 cores. Since our nodes have 44 cores / each, if nodes are empty (no other users
have running jobs there), job will land on a single node (it has enough cores to run 24 tasks).
If we want to ensure that job is using at least two different nodes (i.e. for boosting CPU frequency, or because the job requires
more memory per core) you should specify other options.
A good example is ``--ntasks-per-node=12``. This will equally distribute 12 tasks on 2 nodes.
```bash
#SBATCH --ntasks-per-node=12
```
A different example could be by specifying how much memory per core is needed. For instance ``--mem-per-cpu=32000`` will reserve
~32000MB per core. Since we have a maximum of 352000MB per node, Slurm will be only able to allocate 11 cores (32000MB x 11cores = 352000MB).
It means that 3 nodes will be needed (can not run 12 tasks per node, only 11), or we need to decrease ``--mem-per-cpu`` to a lower value which
can allow the use of at least 12 cores per node (i.e. ``28000``)
```bash
#SBATCH --mem-per-cpu=28000
```
Finally, in order to ensure exclusivity of the node, an option *--exclusive* can be used (see below). This will ensure that
the requested nodes are exclusive for the job (no other users jobs will interact with this node, and only completely
free nodes will be allocated).
```bash
#SBATCH --exclusive
```
This can be combined with the previous examples.
More advanced configurations can be defined and can be combined with the previous examples. More information about advanced
options can be found in the following link: https://slurm.schedmd.com/sbatch.html (or run 'man sbatch').
If you have questions about how to properly execute your jobs, please contact us through merlin-admins@lists.psi.ch. Do not run
advanced configurations unless your are sure of what you are doing.
---
## Environment Modules
On top of the operating system stack we provide different software using the PSI developed
pmodule system. Useful commands:
```bash
module avail # to see the list of available software provided via pmodules
module load gnuplot/5.2.0 # to load specific version of gnuplot package
module search hdf # try it out to see which version of hdf5 package is provided and with which dependencies
module load gcc/6.2.0 openmpi/1.10.2 hdf5/1.8.17 # load the specific version of hdf5, compiled with specific version of gcc and openmpi
module use unstable # to get access to the packages which are not considered to be fully stable by module provider (may be very fresh version, or not yet tested by community)
module list # to see which software is loaded in your environment
```
### Requests for New Software
If you miss some package/version, contact us

90
pages/merlin6-user-guide/merlin6-slurm/slurm-configuration.md Normal file
View File

@ -0,0 +1,90 @@
---
layout: default
title: Slurm Configuration
parent: Merlin6 Slurm
grand_parent: Merlin6 User Guide
nav_order: 2
---
# Slurm Configuration
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
---
## Using the Slurm batch system
Clusters at PSI use the [Slurm Workload Manager](http://slurm.schedmd.com/) as the batch system technology for managing and scheduling jobs.
Historically, *Merlin4* and *Merlin5* also used Slurm. In the same way, **Merlin6** has been also configured with this batch system.
Slurm has been installed in a **multi-clustered** configuration, allowing to integrate multiple clusters in the same batch system.
For understanding the Slurm configuration setup in the cluster, sometimes may be useful to check the following files:
* ``/etc/slurm/slurm.conf`` - can be found in the login nodes and computing nodes.
* ``/etc/slurm/cgroup.conf`` - can be found in the computing nodes, is also propagated to login nodes for user read access.
* ``/etc/slurm/gres.conf`` - can be found in the GPU nodes, is also propgated to login nodes and computing nodes for user read access.
The previous configuration files which can be found in the login nodes, correspond exclusively to the **merlin6** cluster configuration files.
Configuration files for the old **merlin5** cluster must be checked directly on any of the **merlin5** computing nodes: these are not propagated
to the **merlin6** login nodes.
### About Merlin5 & Merlin6
The new Slurm cluster is called **merlin6**. However, the old Slurm *merlin* cluster will be kept for some time, and it has been renamed to **merlin5**.
It will allow to keep running jobs in the old computing nodes until users have fully migrated their codes to the new cluster.
From July 2019, **merlin6** becomes the **default cluster** and any job submitted to Slurm will be submitted to that cluster. Users can keep submitting to
the old *merlin5* computing nodes by using the option ``--cluster=merlin5``.
In this documentation is only explained the usage of the **merlin6** Slurm cluster.
### Using Slurm 'merlin6' cluster
Basic usage for the **merlin6** cluster will be detailed here. For advanced usage, please use the following document [LINK TO SLURM ADVANCED CONFIG]()
#### Merlin6 Node definition
The following table show default and maximum resources that can be used per node:
| Nodes | Def.#CPUs | Max.#CPUs | Def.Mem/CPU | Max.Mem/CPU | Max.Mem/Node | Max.Swap | Def.#GPUs | Max.#GPUs |
|:---------------------------------- | ---------:| ---------:| -----------:| -----------:| ------------:| --------:| --------- | --------- |
| merlin-c-[001-022,101-122,201-222] | 1 core | 44 cores | 8000 | 352000 | 352000 | 10000 | N/A | N/A |
| merlin-g-[001] | 1 core | 8 cores | 8000 | 102498 | 102498 | 10000 | 1 | 2 |
| merlin-g-[002-009] | 1 core | 10 cores | 8000 | 102498 | 102498 | 10000 | 1 | 4 |
If nothing is specified, by default each core will use up to 8GB of memory. More memory per core can be specified with the ``--mem=<memory>`` option,
and maximum memory allowed is ``Max.Mem/Node``.
In *Merlin6*, memory is considered a Consumable Resource, as well as the CPU.
#### Merlin6 Slurm partitions
Partition can be specified when submitting a job with the ``--partition=<partitionname>`` option.
The following *partitions* (also known as *queues*) are configured in Slurm:
| Partition | Default Partition | Default Time | Max Time | Max Nodes | Priority |
|:----------- | ----------------- | ------------ | -------- | --------- | -------- |
| **general** | true | 1 day | 1 week | 50 | low |
| **daily** | false | 1 day | 1 day | 60 | medium |
| **hourly** | false | 1 hour | 1 hour | unlimited | highest |
General is the *default*, so when nothing is specified job will be by default assigned to that partition. General can not have more than 50 nodes
running jobs. For **daily** this limitation is extended to 60 nodes while for **hourly** there are no limits. Shorter jobs have more priority than
longer jobs, hence in general terms would be scheduled before (however, other factors such like user fair share value can affect to this decision).
#### Merlin6 User limits
By default, users can not use more than 528 cores at the same time (Max CPU per user). This limit applies for the **general** and **daily** partitions. For the **hourly** partition, there is no restriction.
These limits are softed for the **daily** partition during non working hours and during the weekend as follows:
| Partition | Mon-Fri 08h-18h | Sun-Thu 18h-0h | From Fri 18h to Sun 8h | From Sun 8h to Mon 18h |
|:----------- | --------------- | -------------- | ----------------------- | ---------------------- |
| **general** | 528 | 528 | 528 | 528 |
| **daily** | 528 | 792 | Unlimited | 792 |
| **hourly** | Unlimited | Unlimited | Unlimited | Unlimited |

151
pages/merlin6-user-guide/migration-from-merlin5.md Normal file
View File

@ -0,0 +1,151 @@
---
layout: default
title: Migration From Merlin5
parent: Merlin6 User Guide
nav_order: 7
---
# Migration From Merlin5
{: .no_toc }
## Table of contents
{: .no_toc .text-delta }
1. TOC
{:toc}
---
## Merlin5 vs Merlin6
### Directories
| Cluster | Home Directory | User Home Directory | Group Home Directory |
| ------- |:-------------------- |:-------------------- |:---------------------------------------- |
| merlin5 | /gpfs/home/$username | /gpfs/data/$username | /gpfs/group/$laboratory |
| merlin6 | /psi/home/$username | /data/user/$username | /data/project/[general|bio]/$projectname |
### USR/GRP quota limits in Merlin6
| Directory | Quota_Type [Soft:Hard] (Block) | Quota_Type [Soft:Hard] (Files) | Quota Change Policy: Block | Quota Change Policy: Files |
| ---------------------------------- | ------------------------------ | ------------------------------ |:--------------------------------------------- |:--------------------------------------------- |
| /psi/home/$username | USR [10GB:11GB] | *Undef* | Up to x2 when strictly justified. | N/A |
| /data/user/$username | USR [1TB:1.074TB] | USR [1M:1.1M] | Inmutable. Need a project. | Changeable when justified. |
| /data/project/bio/$projectname | GRP [1TB:1.074TB] | GRP [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. |
| /data/project/general/$projectname | GRP [1TB:1.074TB] | GRP [1M:1.1M] | Changeable according to project requirements. | Changeable according to project requirements. |
where:
* **Block** is capacity size in GB and TB
* **Files** is number of files + directories in Millions (M)
* User data directorry ``/data/user`` has a strict user block quota limit policy. If more disk space is required, 'project' must be created.
### Project directory
#### Why 'project' would be needed?
In Merlin5 the concept *project* did not exist. A similar concept (*group*) was existing and was mostly focused for BIO experiments.
Quite often different users are working in *a similar* / *the same* project. Data was shared in different ways,
such like by allowing other users to access private data, or by having duplicates on each user directory needing access to that data.
This makes the storage usage unefficient and insecure.
Also, there is another problem related to that: when a user leaves, we have plenty of data which needs to be kept and nobody becomes
responsible for that. In addition, after several months user is unregistered from PSI and we end up with orphaned data which needs to
be kept, but we sometimes loose track of the user.
With that, we want to restrict the usage of individual data and bet for project (shared) data. There will be one main responsible for
this project, but if for some reason this person leaves, responsible can be somebody else (successor if exists, supervisor, or in the
worst case, the admin).
#### Requesting a *project*
For requesting a *project* users must provide:
* Define a *'project'* directory name. This must be unique.
* Have an existing *project* **Unix Group**.
* This can be requested through [PSI Service Now](https://psi.service-now.com/psisp)
* Unix group must start with *``unx-``*
* This Unix Group will be the default group for the *'project'*
* Define a project main responsible and supervisor
* Define and justify quota requirements:
* By default GRP quota will be: Block Quota GRP [1TB:1.074TB] and File Qota GRP [1M:1.1M]
* Individual USR quotas can be requested (by default are not set).
---
## Migration Schedule
### Phase 1 [June]: Pre-migration
* Users keep working on Merlin5
* Merlin5 production directories: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users may raise any problems (quota limits, unaccessible files, etc.) to merlin-admins@lists.psi.ch
* Users can start migrating data (see [Migration steps](# Migration steps))
* Users should copy their data from Merlin5 /gpfs/data to Merlin6 /data/user
* Users should copy their home from Merlin5 /gpfs/home to Merlin6 /psi/home
* Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins.
### Phase 2 [July-October]: Migration to Merlin6
* Merlin6 becomes official cluster, and directories are switched to the new structure:
* Merlin6 production directories: ``'/psi/home/'``, ``'/data/user'``, ``'/data/project'``
* Merlin5 directories available in RO: ``'/gpfs/home/'``, ``'/gpfs/data'``, ``'/gpfs/group'``
* Users can keep migrating their data (see [Migration steps](# Migration steps))
* ALL data must be migrated
* Job submissions by default to Merlin6. Submission to Merlin5 computing nodes possible.
* Users should inform when migration is done, and which directories were migrated. Deletion for such directories can be requested by admins.
### Phase 3 [November]: Merlin5 Decomission
* Old Merlin5 storage unmounted.
* Migrated directories reported by users will be deleted.
* Remaining Merlin5 data will be archived.
* Merlin5 Slurm cluster removed from production.
---
## Migration steps
### Cleanup / Archive files
* Users must cleanup and/or archive files, according to quota limits in the storage.
* If extra space is needed, *'project'* would be needed.
* If extra files are needed, you can request for an increasement of the quota/
#### File list
### Step 1: Migrating
First migration:
```bash
rsync -avAHXS <source_merlin5> <destination_merlin6>
rsync -avAHXS /gpfs/data/$username/* /data/user/$username
```
This can take several hours or days:
* You can try to parallelize multiple rsync commands in sub-directories for increasing transfer rate.
* Please do not parallelize many concurrent directories. Let's say, don't add more than 10 together.
* We may have other users doing the same and it could cause storage / UI performance problems in the Merlin5 cluster.
### Step 2: Mirroring
Once first migration is done, a second ``rsync`` should be ran. This is done with ``--delete``. With this option ``rsync`` will
behave in a way where it will delete from the destination all files that were removed in the source, but also will propagate
new files from the source to the destination.
```bash
rsync -avAHXS --delete <source_merlin5> <destination_merlin6>
rsync -avAHXS --delete /gpfs/data/$username/* /data/user/$username
```
### Step 3: Removing / Archiving old data
#### Removing migrated data
Once you ensure that everything is migrated to the new storage, data is ready to be deleted from the old storage.
Users must report when migration is finished and report which directories are affected and ready to be removed.
Merlin administrators will remove the directories, always asking for a last confirmation.
#### Archiving data
Once all migrated data has been removed from the old storage, missing data will be archived.

BIN
pages/merlin6-user-guide/source/images/merlinschema3.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 167 KiB