---
title: Slurm Configuration
#tags:
keywords: configuration, partitions, node definition
last_updated: 23 January 2020
summary: "This document describes a summary of the Merlin6 configuration."
sidebar: merlin6_sidebar
permalink: /merlin6/slurm-configuration.html
---

## About Merlin5 & Merlin6

The new Slurm cluster is called **merlin6**. However, the old Slurm *merlin* cluster will be kept for some time, and it has been renamed to **merlin5**.
It will allow to keep running jobs in the old computing nodes until users have fully migrated their codes to the new cluster.

From July 2019, **merlin6** becomes the **default cluster** and any job submitted to Slurm will be submitted to that cluster. Users can keep submitting to
the old *merlin5* computing nodes by using the option ``--cluster=merlin5``.

In this documentation is only explained the usage of the **merlin6** Slurm cluster.

## Using Slurm 'merlin6' cluster

Basic usage for the **merlin6** cluster will be detailed here. For advanced usage, please use the following document [LINK TO SLURM ADVANCED CONFIG]()

### Merlin6 Node definition

The following table show default and maximum resources that can be used per node:

| Nodes              | Def.#CPUs | Max.#CPUs | #Threads | Def.Mem/CPU | Max.Mem/CPU | Max.Mem/Node | Max.Swap | Def.#GPUs | Max.#GPUs |
|:-------------------| ---------:| ---------:| -------- | -----------:| -----------:| ------------:| --------:| --------- | --------- |
| merlin-c-[001-024] | 1 core    | 44 cores  | 2        | 4000        | 352000      | 352000       | 10000    | N/A       | N/A       |
| merlin-c-[101-124] | 1 core    | 44 cores  | 2        | 4000        | 352000      | 352000       | 10000    | N/A       | N/A       |
| merlin-c-[201-224] | 1 core    | 44 cores  | 2        | 4000        | 352000      | 352000       | 10000    | N/A       | N/A       |
| merlin-g-[001]     | 1 core    | 8 cores   | 1        | 4000        | 102400      | 102400       | 10000    | 1         | 2         |
| merlin-g-[002-009] | 1 core    | 20 cores  | 1        | 4000        | 102400      | 102400       | 10000    | 1         | 4         |

If nothing is specified, by default each core will use up to 8GB of memory. More memory per core can be specified with the ``--mem=<memory>`` option,
and maximum memory allowed is ``Max.Mem/Node``.

In *Merlin6*, memory is considered a Consumable Resource, as well as the CPU.

### Merlin6 Slurm partitions

Partition can be specified when submitting a job with the ``--partition=<partitionname>`` option.
The following *partitions* (also known as *queues*) are configured in Slurm:

| Partition   | Default Partition | Default Time | Max Time | Max Nodes | Priority |
|:----------- | ----------------- | ------------ | -------- | --------- | -------- |
| **general** | true              | 1 day        | 1 week   | 50        | low      |
| **daily**   | false             | 1 day        | 1 day    | 60        | medium   |
| **hourly**  | false             | 1 hour       | 1 hour   | unlimited | highest  |

General is the *default*, so when nothing is specified job will be by default assigned to that partition. General can not have more than 50 nodes
running jobs. For **daily** this limitation is extended to 60 nodes while for **hourly** there are no limits. Shorter jobs have more priority than
longer jobs, hence in general terms would be scheduled before (however, other factors such like user fair share value can affect to this decision).

### Merlin6 User limits

By default, users can not use more than 704 cores at the same time (Max CPU per user). This is equivalent to 8 exclusive nodes.
This limit applies to the **general** and **daily** partitions. For the **hourly** partition, there is no restriction and user limits are removed. 
Limits are softed for the **daily** partition during non working hours and during the weekend as follows:

| Partition   | Mon-Fri 08h-18h  | Sun-Thu 18h-0h | From Fri 18h to Sun 8h  | From Sun 8h to Mon 18h |
|:----------- | ---------------- | -------------- | ----------------------- | ---------------------- |
| **general** | 704 (user limit) | 704            | 704                     | 704                    |
| **daily**   | 704 (user limit) | 1408           | Unlimited               | 1408                   |
| **hourly**  | Unlimited        | Unlimited      | Unlimited               | Unlimited              |

## Understanding the Slurm configuration (for advanced users)

Clusters at PSI use the [Slurm Workload Manager](http://slurm.schedmd.com/) as the batch system technology for managing and scheduling jobs.
Historically, *Merlin4* and *Merlin5* also used Slurm. In the same way, **Merlin6** has been also configured with this batch system.

Slurm has been installed in a **multi-clustered** configuration, allowing to integrate multiple clusters in the same batch system.

For understanding the Slurm configuration setup in the cluster, sometimes may be useful to check the following files:

* ``/etc/slurm/slurm.conf`` - can be found in the login nodes and computing nodes.
* ``/etc/slurm/gres.conf`` - can be found in the GPU nodes, is also propgated to login nodes and computing nodes for user read access.
* ``/etc/slurm/cgroup.conf`` - can be found in the computing nodes, is also propagated to login nodes for user read access.

The previous configuration files which can be found in the login nodes, correspond exclusively to the **merlin6** cluster configuration files.
Configuration files for the old **merlin5** cluster must be checked directly on any of the **merlin5** computing nodes: these are not propagated
to the **merlin6** login nodes.
