85 lines
5.4 KiB
Markdown
85 lines
5.4 KiB
Markdown
---
|
|
title: Slurm Configuration
|
|
#tags:
|
|
#keywords:
|
|
last_updated: 18 June 2019
|
|
#summary: ""
|
|
sidebar: merlin6_sidebar
|
|
permalink: /merlin6/slurm-configuration.html
|
|
---
|
|
|
|
## Using the Slurm batch system
|
|
|
|
Clusters at PSI use the [Slurm Workload Manager](http://slurm.schedmd.com/) as the batch system technology for managing and scheduling jobs.
|
|
Historically, *Merlin4* and *Merlin5* also used Slurm. In the same way, **Merlin6** has been also configured with this batch system.
|
|
|
|
Slurm has been installed in a **multi-clustered** configuration, allowing to integrate multiple clusters in the same batch system.
|
|
|
|
For understanding the Slurm configuration setup in the cluster, sometimes may be useful to check the following files:
|
|
|
|
* ``/etc/slurm/slurm.conf`` - can be found in the login nodes and computing nodes.
|
|
* ``/etc/slurm/cgroup.conf`` - can be found in the computing nodes, is also propagated to login nodes for user read access.
|
|
* ``/etc/slurm/gres.conf`` - can be found in the GPU nodes, is also propgated to login nodes and computing nodes for user read access.
|
|
|
|
The previous configuration files which can be found in the login nodes, correspond exclusively to the **merlin6** cluster configuration files.
|
|
Configuration files for the old **merlin5** cluster must be checked directly on any of the **merlin5** computing nodes: these are not propagated
|
|
to the **merlin6** login nodes.
|
|
|
|
### About Merlin5 & Merlin6
|
|
|
|
The new Slurm cluster is called **merlin6**. However, the old Slurm *merlin* cluster will be kept for some time, and it has been renamed to **merlin5**.
|
|
It will allow to keep running jobs in the old computing nodes until users have fully migrated their codes to the new cluster.
|
|
|
|
From July 2019, **merlin6** becomes the **default cluster** and any job submitted to Slurm will be submitted to that cluster. Users can keep submitting to
|
|
the old *merlin5* computing nodes by using the option ``--cluster=merlin5``.
|
|
|
|
In this documentation is only explained the usage of the **merlin6** Slurm cluster.
|
|
|
|
### Using Slurm 'merlin6' cluster
|
|
|
|
Basic usage for the **merlin6** cluster will be detailed here. For advanced usage, please use the following document [LINK TO SLURM ADVANCED CONFIG]()
|
|
|
|
#### Merlin6 Node definition
|
|
|
|
The following table show default and maximum resources that can be used per node:
|
|
|
|
| Nodes | Def.#CPUs | Max.#CPUs | #Threads | Def.Mem/CPU | Max.Mem/CPU | Max.Mem/Node | Max.Swap | Def.#GPUs | Max.#GPUs |
|
|
|:-------------------| ---------:| ---------:| -------- | -----------:| -----------:| ------------:| --------:| --------- | --------- |
|
|
| merlin-c-[001-022] | 1 core | 44 cores | 2 | 4000 | 352000 | 352000 | 10000 | N/A | N/A |
|
|
| merlin-c-[101-122] | 1 core | 44 cores | 2 | 4000 | 352000 | 352000 | 10000 | N/A | N/A |
|
|
| merlin-c-[201-222] | 1 core | 44 cores | 2 | 4000 | 352000 | 352000 | 10000 | N/A | N/A |
|
|
| merlin-g-[001] | 1 core | 8 cores | 2 | 4000 | 102498 | 102498 | 10000 | 1 | 2 |
|
|
| merlin-g-[002-009] | 1 core | 10 cores | 2 | 4000 | 102498 | 102498 | 10000 | 1 | 4 |
|
|
|
|
If nothing is specified, by default each core will use up to 8GB of memory. More memory per core can be specified with the ``--mem=<memory>`` option,
|
|
and maximum memory allowed is ``Max.Mem/Node``.
|
|
|
|
In *Merlin6*, memory is considered a Consumable Resource, as well as the CPU.
|
|
|
|
#### Merlin6 Slurm partitions
|
|
|
|
Partition can be specified when submitting a job with the ``--partition=<partitionname>`` option.
|
|
The following *partitions* (also known as *queues*) are configured in Slurm:
|
|
|
|
| Partition | Default Partition | Default Time | Max Time | Max Nodes | Priority |
|
|
|:----------- | ----------------- | ------------ | -------- | --------- | -------- |
|
|
| **general** | true | 1 day | 1 week | 50 | low |
|
|
| **daily** | false | 1 day | 1 day | 60 | medium |
|
|
| **hourly** | false | 1 hour | 1 hour | unlimited | highest |
|
|
|
|
General is the *default*, so when nothing is specified job will be by default assigned to that partition. General can not have more than 50 nodes
|
|
running jobs. For **daily** this limitation is extended to 60 nodes while for **hourly** there are no limits. Shorter jobs have more priority than
|
|
longer jobs, hence in general terms would be scheduled before (however, other factors such like user fair share value can affect to this decision).
|
|
|
|
#### Merlin6 User limits
|
|
|
|
By default, users can not use more than 704 cores at the same time (Max CPU per user). This is equivalent to 8 exclusive nodes.
|
|
This limit applies to the **general** and **daily** partitions. For the **hourly** partition, there is no restriction and user limits are removed.
|
|
Limits are softed for the **daily** partition during non working hours and during the weekend as follows:
|
|
|
|
| Partition | Mon-Fri 08h-18h | Sun-Thu 18h-0h | From Fri 18h to Sun 8h | From Sun 8h to Mon 18h |
|
|
|:----------- | ---------------- | -------------- | ----------------------- | ---------------------- |
|
|
| **general** | 704 (user limit) | 704 | 704 | 704 |
|
|
| **daily** | 704 (user limit) | 1408 | Unlimited | 1408 |
|
|
| **hourly** | Unlimited | Unlimited | Unlimited | Unlimited |
|