add merlin6 quick start

This commit is contained in:
2025-11-27 10:14:06 +01:00
parent 83406ceb7d
commit d9a005f44a
9 changed files with 116 additions and 133 deletions

View File

@@ -1,52 +0,0 @@
---
title: Code Of Conduct
#tags:
keywords: code of conduct, rules, principle, policy, policies, administrator, backup
last_updated: 07 September 2022
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/code-of-conduct.html
---
## The Basic principle
The basic principle is courtesy and consideration for other users.
* Merlin6 is a system shared by many users, therefore you are kindly requested to apply common courtesy in using its resources. Please follow our guidelines which aim at providing and maintaining an efficient compute environment for all our users.
* Basic shell programming skills are an essential requirement in a Linux/UNIX HPC cluster environment; a proficiency in shell programming is greatly beneficial.
## Interactive nodes
* The interactive nodes (also known as login nodes) are for development and quick testing:
* It is **strictly forbidden to run production jobs** on the login nodes. All production jobs must be submitted to the batch system.
* It is **forbidden to run long processes** occupying big parts of a login node's resources.
* According to the previous rules, **misbehaving running processes will have to be killed.**
in order to keep the system responsive for other users.
## Batch system
* Make sure that no broken or run-away processes are left when your job is done. Keep the process space clean on all nodes.
* During the runtime of a job, it is mandatory to use the ``/scratch`` and ``/shared-scratch`` partitions for temporary data:
* It is **forbidden** to use the ``/data/user``, ``/data/project`` or ``/psi/home/`` for that purpose.
* Always remove files you do not need any more (e.g. core dumps, temporary files) as early as possible. Keep the disk space clean on all nodes.
* Prefer ``/scratch`` over ``/shared-scratch`` and use the latter only when you require the temporary files to be visible from multiple nodes.
* Read the description in **[Merlin6 directory structure](/merlin6/storage.html#merlin6-directories)** for learning about the correct usage of each partition type.
## User and project data
* ***Users are responsible for backing up their own data***. Is recommended to backup the data on third party independent systems (i.e. LTS, Archive, AFS, SwitchDrive, Windows Shares, etc.).
* **`/psi/home`**, as this contains a small amount of data, is the only directory where we can provide daily snapshots for one week. This can be found in the following directory **`/psi/home/.snapshot/`**
* ***When a user leaves PSI, she or her supervisor/team are responsible to backup and move the data out from the cluster***: every few months, the storage space will be recycled for those old users who do not have an existing and valid PSI account.
{{site.data.alerts.warning}}When a user leaves PSI and his account has been removed, her storage space in Merlin may be recycled.
Hence, <b>when a user leaves PSI</b>, she, her supervisor or team <b>must ensure that the data is backed up to an external storage</b>
{{site.data.alerts.end}}
## System Administrator Rights
* The system administrator has the right to temporarily block the access to Merlin6 for an account violating the Code of Conduct in order to maintain the efficiency and stability of the system.
* Repetitive violations by the same user will be escalated to the user's supervisor.
* The system administrator has the right to delete files in the **scratch** directories
* after a job, if the job failed to clean up its files.
* during the job in order to prevent a job from destabilizing a node or multiple nodes.
* The system administrator has the right to kill any misbehaving running processes.

5
docs/merlin6/index.md Normal file
View File

@@ -0,0 +1,5 @@
# Merlin6 HPC Cluster
!!! danger "Decommissioned"
This cluster is no longer generally available.

View File

@@ -1,24 +1,16 @@
---
title: Accessing Interactive Nodes
#tags:
keywords: How to, HowTo, access, accessing, nomachine, ssh
last_updated: 07 September 2022
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/interactive.html
---
# Accessing Interactive Nodes
## SSH Access
For interactive command shell access, use an SSH client. We recommend to activate SSH's X11 forwarding to allow you to use graphical
applications (e.g. a text editor, but for more performant graphical access, refer to the sections below). X applications are supported
For interactive command shell access, use an SSH client. We recommend to activate SSH's X11 forwarding to allow you to use graphical
applications (e.g. a text editor, but for more performant graphical access, refer to the sections below). X applications are supported
in the login nodes and X11 forwarding can be used for those users who have properly configured X11 support in their desktops, however:
* Merlin6 administrators **do not offer support** for user desktop configuration (Windows, MacOS, Linux).
* Hence, Merlin6 administrators **do not offer official support** for X11 client setup.
* Nevertheless, a generic guide for X11 client setup (*Linux*, *Windows* and *MacOS*) is provided below.
* Hence, Merlin6 administrators **do not offer official support** for X11 client setup.
* Nevertheless, a generic guide for X11 client setup (*Linux*, *Windows* and *MacOS*) is provided below.
* PSI desktop configuration issues must be addressed through **[PSI Service Now](https://psi.service-now.com/psisp)** as an *Incident Request*.
* Ticket will be redirected to the corresponding Desktop support group (Windows, Linux).
* Ticket will be redirected to the corresponding Desktop support group (Windows, Linux).
### Accessing from a Linux client
@@ -34,7 +26,8 @@ Refer to [{How To Use Merlin -> Accessing from MacOS Clients}](/merlin6/connect-
## NoMachine Remote Desktop Access
X applications are supported in the login nodes and can run efficiently through a **NoMachine** client. This is the officially supported way to run more demanding X applications on Merlin6.
X applications are supported in the login nodes and can run efficiently through a **NoMachine** client. This is the officially supported way to run more demanding X applications on Merlin6.
* For PSI Windows workstations, this can be installed from the Software Kiosk as 'NX Client'. If you have difficulties installing, please request support through **[PSI Service Now](https://psi.service-now.com/psisp)** as an *Incident Request*.
* For other workstations The client software can be downloaded from the [Nomachine Website](https://www.nomachine.com/product&p=NoMachine%20Enterprise%20Client).

View File

@@ -1,12 +1,4 @@
---
title: Accessing Slurm Cluster
#tags:
keywords: slurm, batch system, merlin5, merlin6, gmerlin6, cpu, gpu
last_updated: 07 September 2022
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/slurm-access.html
---
# Accessing Slurm Cluster
## The Merlin Slurm clusters
@@ -23,14 +15,17 @@ Any job submission must be performed from a **Merlin login node**. Please refer
for further information about how to access the cluster.
In addition, any job *must be submitted from a high performance storage area visible by the login nodes and by the computing nodes*. For this, the possible storage areas are the following:
* `/data/user`
* `/data/project`
* `/shared-scratch`
Please, avoid using `/psi/home` directories for submitting jobs.
### Merlin6 CPU cluster access
The **Merlin6 CPU cluster** (**`merlin6`**) is the default cluster configured in the login nodes. Any job submission will use by default this cluster, unless
The **Merlin6 CPU cluster** (**`merlin6`**) is the default cluster configured
in the login nodes. Any job submission will use by default this cluster, unless
the option `--cluster` is specified with another of the existing clusters.
For further information about how to use this cluster, please visit: [**Merlin6 CPU Slurm Cluster documentation**](/merlin6/slurm-configuration.html).
@@ -43,11 +38,12 @@ For further information about how to use this cluster, please visit: [**Merlin6
### Merlin5 CPU cluster access
The **Merlin5 CPU cluster** (**`merlin5`**) is visible from the login nodes. However, to submit jobs
The **Merlin5 CPU cluster** (**`merlin5`**) is visible from the login nodes. However, to submit jobs
to this cluster, one needs to specify the option `--cluster=merlin5` when submitting a job or allocation.
Using this cluster is in general not recommended, however this is still available for old users needing
extra computational resources or longer jobs. Have in mind that this cluster is only supported in a
**best effort basis**, and it contains very old hardware and configurations.
Using this cluster is in general not recommended, however this is still
available for old users needing extra computational resources or longer jobs.
Have in mind that this cluster is only supported in a **best effort basis**,
and it contains very old hardware and configurations.
For further information about how to use this cluster, please visit the [**Merlin5 CPU Slurm Cluster documentation**](/gmerlin6/slurm-configuration.html).

View File

@@ -0,0 +1,48 @@
# Code Of Conduct
## The Basic principle
The basic principle is courtesy and consideration for other users.
* Merlin6 is a system shared by many users, therefore you are kindly requested to apply common courtesy in using its resources. Please follow our guidelines which aim at providing and maintaining an efficient compute environment for all our users.
* Basic shell programming skills are an essential requirement in a Linux/UNIX HPC cluster environment; a proficiency in shell programming is greatly beneficial.
## Interactive nodes
* The interactive nodes (also known as login nodes) are for development and quick testing:
* It is **strictly forbidden to run production jobs** on the login nodes. All production jobs must
be submitted to the batch system.
* It is **forbidden to run long processes** occupying big parts of a login node's resources.
* According to the previous rules, **misbehaving running processes will have to be killed.**
in order to keep the system responsive for other users.
## Batch system
* Make sure that no broken or run-away processes are left when your job is done. Keep the process space clean on all nodes.
* During the runtime of a job, it is mandatory to use the ``/scratch`` and ``/shared-scratch`` partitions for temporary data:
* It is **forbidden** to use the ``/data/user``, ``/data/project`` or ``/psi/home/`` for that purpose.
* Always remove files you do not need any more (e.g. core dumps, temporary files) as early as possible. Keep the disk space clean on all nodes.
* Prefer ``/scratch`` over ``/shared-scratch`` and use the latter only when you require the temporary files to be visible from multiple nodes.
* Read the description in **[Merlin6 directory structure](/merlin6/storage.html#merlin6-directories)** for learning about the correct usage of each partition type.
## User and project data
* ***Users are responsible for backing up their own data***. Is recommended to backup the data on third party independent systems (i.e. LTS, Archive, AFS, SwitchDrive, Windows Shares, etc.).
* **`/psi/home`**, as this contains a small amount of data, is the only directory where we can provide daily snapshots for one week. This can be found in the following directory **`/psi/home/.snapshot/`**
* ***When a user leaves PSI, she or her supervisor/team are responsible to backup and move the data out from the cluster***: every few months, the storage space will be recycled for those old users who do not have an existing and valid PSI account.
!!! warning
When a user leaves PSI and his account has been removed, her storage space in Merlin may be recycled.
Hence, **when a user leaves PSI**, she, her supervisor or team **must ensure that the data is backed up to an external storage**
## System Administrator Rights
* The system administrator has the right to temporarily block the access to
Merlin6 for an account violating the Code of Conduct in order to maintain the
efficiency and stability of the system.
* Repetitive violations by the same user will be escalated to the user's supervisor.
* The system administrator has the right to delete files in the **scratch** directories
* after a job, if the job failed to clean up its files.
* during the job in order to prevent a job from destabilizing a node or multiple nodes.
* The system administrator has the right to kill any misbehaving running processes.

View File

@@ -1,33 +1,23 @@
---
title: Introduction
#tags:
keywords: introduction, home, welcome, architecture, design
last_updated: 07 September 2022
#summary: "Merlin 6 cluster overview"
sidebar: merlin6_sidebar
permalink: /merlin6/introduction.html
redirect_from:
- /merlin6
- /merlin6/index.html
---
# Introduction
## The Merlin local HPC cluster
Historically, the local HPC clusters at PSI were named **Merlin**. Over the years,
multiple generations of Merlin have been deployed.
At present, the **Merlin local HPC cluster** contains _two_ generations of it:
At present, the **Merlin local HPC cluster** contains _two_ generations of it:
* the old **Merlin5** cluster (`merlin5` Slurm cluster), and
* the newest generation **Merlin6**, which is divided in two Slurm clusters:
* `merlin6` as the Slurm CPU cluster
* `gmerlin6` as the Slurm GPU cluster.
* `merlin6` as the Slurm CPU cluster
* `gmerlin6` as the Slurm GPU cluster.
Access to the different Slurm clusters is possible from the [**Merlin login nodes**](/merlin6/interactive.html),
which can be accessed through the [SSH protocol](/merlin6/interactive.html#ssh-access) or the [NoMachine (NX) service](/merlin6/nomachine.html).
The following image shows the Slurm architecture design for the Merlin5 & Merlin6 (CPU & GPU) clusters:
![Merlin6 Slurm Architecture Design]({{ "/images/merlin-slurm-architecture.png" }})
![Merlin6 Slurm Architecture Design](/images/merlin-slurm-architecture.png)
### Merlin6
@@ -40,25 +30,27 @@ more compute nodes and cluster storage without significant increase of
the costs of the manpower and the operations.
Merlin6 contains all the main services needed for running cluster, including
**login nodes**, **storage**, **computing nodes** and other *subservices*,
connected to the central PSI IT infrastructure.
**login nodes**, **storage**, **computing nodes** and other _subservices_,
connected to the central PSI IT infrastructure.
#### CPU and GPU Slurm clusters
The Merlin6 **computing nodes** are mostly based on **CPU** resources. However,
it also contains a small amount of **GPU**-based resources, which are mostly used
by the BIO Division and by Deep Leaning project.
by the BIO Division and by Deep Leaning project.
These computational resources are split into **two** different **[Slurm](https://slurm.schedmd.com/overview.html)** clusters:
* The Merlin6 CPU nodes are in a dedicated **[Slurm](https://slurm.schedmd.com/overview.html)** cluster called [**`merlin6`**](/merlin6/slurm-configuration.html).
* This is the **default Slurm cluster** configured in the login nodes: any job submitted without the option `--cluster` will be submited to this cluster.
* This is the **default Slurm cluster** configured in the login nodes: any job submitted without the option `--cluster` will be submited to this cluster.
* The Merlin6 GPU resources are in a dedicated **[Slurm](https://slurm.schedmd.com/overview.html)** cluster called [**`gmerlin6`**](/gmerlin6/slurm-configuration.html).
* Users submitting to the **`gmerlin6`** GPU cluster need to specify the option ``--cluster=gmerlin6``.
* Users submitting to the **`gmerlin6`** GPU cluster need to specify the option ``--cluster=gmerlin6``.
### Merlin5
The old Slurm **CPU** *merlin* cluster is still active and is maintained in a best effort basis.
The old Slurm **CPU** _Merlin_ cluster is still active and is maintained in a best effort basis.
**Merlin5** only contains **computing nodes** resources in a dedicated **[Slurm](https://slurm.schedmd.com/overview.html)** cluster.
* The Merlin5 CPU cluster is called [**merlin5**](/merlin5/slurm-configuration.html).

View File

@@ -1,12 +1,4 @@
---
title: Requesting Merlin Accounts
#tags:
keywords: registration, register, account, merlin5, merlin6, snow, service now
last_updated: 07 September 2022
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/request-account.html
---
# Requesting Merlin Accounts
## Requesting Access to Merlin6
@@ -14,9 +6,10 @@ Access to Merlin6 is regulated by a PSI user's account being a member of the **`
Requesting **Merlin6** access *has to be done* with the corresponding **[Request Linux Group Membership](https://psi.service-now.com/psisp?id=psi_new_sc_cat_item&sys_id=84f2c0c81b04f110679febd9bb4bcbb1)** form, available in the [PSI Service Now Service Catalog](https://psi.service-now.com/psisp).
![Example: Requesting access to Merlin6]({{ "/images/Access/01-request-merlin6-membership.png" }})
![Example: Requesting access to Merlin6](/images/Access/01-request-merlin6-membership.png)
Mandatory customizable fields are the following:
* **`Order Access for user`**, which defaults to the logged in user. However, requesting access for another user it's also possible.
* **`Request membership for group`**, for Merlin6 the **`svc-cluster_merlin6`** must be selected.
* **`Justification`**, please add here a short justification why access to Merlin6 is necessary.
@@ -29,9 +22,10 @@ Access to Merlin5 is regulated by a PSI user's account being a member of the **`
Requesting **Merlin5** access *has to be done* with the corresponding **[Request Linux Group Membership](https://psi.service-now.com/psisp?id=psi_new_sc_cat_item&sys_id=84f2c0c81b04f110679febd9bb4bcbb1)** form, available in the [PSI Service Now Service Catalog](https://psi.service-now.com/psisp).
![Example: Requesting access to Merlin5]({{ "/images/Access/01-request-merlin5-membership.png" }})
![Example: Requesting access to Merlin5](/images/Access/01-request-merlin5-membership.png)
Mandatory customizable fields are the following:
* **`Order Access for user`**, which defaults to the logged in user. However, requesting access for another user it's also possible.
* **`Request membership for group`**, for Merlin5 the **`svc-cluster_merlin5`** must be selected.
* **`Justification`**, please add here a short justification why access to Merlin5 is necessary.
@@ -41,6 +35,7 @@ Once submitted, the Merlin responsible will approve the request as soon as possi
## Further documentation
Further information it's also available in the Linux Central Documentation:
* [Unix Group / Group Management for users](https://linux.psi.ch/documentation/services/user-guide/unix_groups.html)
* [Unix Group / Group Management for group managers](https://linux.psi.ch/documentation/services/admin-guide/unix_groups.html)

View File

@@ -1,12 +1,4 @@
---
title: Requesting a Merlin Project
#tags:
keywords: merlin project, project, snow, service now
last_updated: 07 September 2022
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/request-project.html
---
# Requesting a Merlin Project
A project owns its own storage area in Merlin, which can be accessed by other group members.
@@ -21,32 +13,35 @@ This document explains how to request new Unix group, to request membership for
## About Unix groups
Before requesting a Merlin project, it is important to have a Unix group that can be used to grant access to it to different members
Before requesting a Merlin project, it is important to have a Unix group that can be used to grant access to it to different members
of the project.
Unix groups in the PSI Active Directory (which is the PSI central database containing user and group information, and more) are defined by the `unx-` prefix, followed by a name.
In general, PSI employees working on Linux systems (including HPC clusters, like Merlin) can request for a non-existing Unix group, and can become responsible for managing it.
In general, PSI employees working on Linux systems (including HPC clusters, like Merlin) can request for a non-existing Unix group, and can become responsible for managing it.
In addition, a list of administrators can be set. The administrators, together with the group manager, can approve or deny membership requests. Further information about this topic
is covered in the [Linux Documentation - Services Admin Guides: Unix Groups / Group Management](https://linux.psi.ch/documentation/services/admin-guide/unix_groups.html), managed by the Central Linux Team.
To gran access to specific Merlin project directories, some users may require to be added to some specific **Unix groups**:
* Each Merlin project (i.e. `/data/project/{bio|general}/$projectname`) or experiment (i.e. `/data/experiment/$experimentname`) directory has access restricted by ownership and group membership (with a very few exceptions allowing public access).
* Users requiring access to a specific restricted project or experiment directory have to request membership for the corresponding Unix group owning the directory.
* Each Merlin project (i.e. `/data/project/{bio|general}/$projectname`) or experiment (i.e. `/data/experiment/$experimentname`) directory has access restricted by ownership and group membership (with a very few exceptions allowing public access).
* Users requiring access to a specific restricted project or experiment directory have to request membership for the corresponding Unix group owning the directory.
### Requesting a new Unix group
**If you need a new Unix group** to be created, you need to first get this group through a separate
**If you need a new Unix group** to be created, you need to first get this group through a separate
**[PSI Service Now ticket](https://psi.service-now.com/psisp)**. **Please use the following template.**
You can also specify the login names of the initial group members and the **owner** of the group.
The owner of the group is the person who will be allowed to modify the group.
* Please open an *Incident Request* with subject:
```
```text
Subject: Request for new unix group xxxx
```
* and base the text field of the request on this template
```
```text
Dear HelpDesk
I would like to request a new unix group.
@@ -63,6 +58,7 @@ The owner of the group is the person who will be allowed to modify the group.
Existing Merlin projects have already a Unix group assigned. To have access to a project, users must belong to the proper **Unix group** owning that project.
Supervisors should inform new users which extra groups are needed for their project(s). If this information is not known, one can check the permissions for that directory. In example:
```bash
(base) ❄ [caubet_m@merlin-l-001:/data/user/caubet_m]# ls -ltrhd /data/project/general/$projectname
(base) ❄ [caubet_m@merlin-l-001:/data/user/caubet_m]# ls -ltrhd /data/project/bio/$projectname
@@ -70,7 +66,7 @@ Supervisors should inform new users which extra groups are needed for their proj
Requesting membership for a specific Unix group *has to be done* with the corresponding **[Request Linux Group Membership](https://psi.service-now.com/psisp?id=psi_new_sc_cat_item&sys_id=84f2c0c81b04f110679febd9bb4bcbb1)** form, available in the [PSI Service Now Service Catalog](https://psi.service-now.com/psisp).
![Example: Requesting Unix Group membership]({{ "/images/Access/01-request-unx-group-membership.png" }})
![Example: Requesting Unix Group membership](/images/Access/01-request-unx-group-membership.png)
Once submitted, the responsible of the Unix group has to approve the request.
@@ -88,12 +84,14 @@ Once a Unix group is available, a Merlin project can be requested.
To request a project, please provide the following information in a **[PSI Service Now ticket](https://psi.service-now.com/psisp)**
* Please open an *Incident Request* with subject:
```
```text
Subject: [Merlin6] Project Request for project name xxxxxx
```
* and base the text field of the request on this template
```
```text
Dear HelpDesk
I would like to request a new Merlin6 project.
@@ -117,6 +115,7 @@ need to request this and provide a description of your storage needs.
## Further documentation
Further information it's also available in the Linux Central Documentation:
* [Unix Group / Group Management for users](https://linux.psi.ch/documentation/services/user-guide/unix_groups.html)
* [Unix Group / Group Management for group managers](https://linux.psi.ch/documentation/services/admin-guide/unix_groups.html)

View File

@@ -78,6 +78,13 @@ nav:
- merlin7/index.md
- Merlin6:
- merlin6/index.md
- Quick Start Guide:
- merlin6/quick-start-guide/introduction.md
- merlin6/quick-start-guide/code-of-conduct.md
- merlin6/quick-start-guide/requesting-accounts.md
- merlin6/quick-start-guide/requesting-projects.md
- merlin6/quick-start-guide/accessing-interactive-nodes.md
- merlin6/quick-start-guide/accessing-slurm.md
- PSI@CSCS:
- cscs-userlab/index.md
- cscs-userlab/transfer-data.md