create "connecting to merlin" section #20

Merged
viessm_h merged 2 commits from revamp_01 into master 2026-07-24 11:43:41 +02:00
55 changed files with 838 additions and 142 deletions
+2 -14
View File
@@ -1,19 +1,7 @@
/* Strong high-contrast section headers in MkDocs Material nav */
/* Top-level section blocks */
.md-nav__title {
font-weight: 800;
font-size: 0.65rem;
color: #ffffff !important; /* white text */
background-color: #586ACB !important; /* strong dark blue */
padding: 6px 10px;
margin-top: 14px;
}
/* For section-like items (common in newer Material versions) */
.md-nav--primary .md-nav__item--section > .md-nav__link {
.md-nav--primary .md-nav[data-md-level="1"] > .md-nav__list > .md-nav__item--section > .md-nav__link {
font-weight: 800;
font-size: 0.65rem;
@@ -25,7 +13,7 @@
}
/* For nested group headers (covers older/other layouts) */
.md-nav--primary .md-nav__item--nested > .md-nav__link {
.md-nav--primary .md-nav[data-md-level="1"] > .md-nav__list > .md-nav__item--nested > .md-nav__link {
font-weight: 800;
font-size: 0.65rem;
+1 -1
View File
@@ -6,5 +6,5 @@
Only a reduced footprint of Merlin6 is still maintained, and **exclusively** for a small number of stakeholders
who own dedicated nodes and cannot yet fully run their workloads on the new cluster.
**New users must request access to [Merlin7](../merlin7/01-Quick-Start-Guide/introduction.md) instead.**
**New users must request access to [Merlin7](../merlin7/01-quick-start-guide/introduction.md) instead.**
@@ -14,34 +14,34 @@ in the login nodes and X11 forwarding can be used for those users who have prope
### Accessing from a Linux client
Refer to [{How To Use Merlin -> Accessing from Linux Clients}](../02-How-To-Use-Merlin/connect-from-windows.md) for **Linux** SSH client and X11 configuration.
Refer to [{Connecting to Merlin7 -> Accessing from Linux Clients}](../02-connecting-to-merlin/connect-from-windows.md) for **Linux** SSH client and X11 configuration.
### Accessing from a Windows client
Refer to [{How To Use Merlin -> Accessing from Windows Clients}](../02-How-To-Use-Merlin/connect-from-windows.md) for **Windows** SSH client and X11 configuration.
Refer to [{Connecting to Merlin7 -> Accessing from Windows Clients}](../02-connecting-to-merlin/connect-from-windows.md) for **Windows** SSH client and X11 configuration.
### Accessing from a MacOS client
Refer to [{How To Use Merlin -> Accessing from MacOS Clients}](../02-How-To-Use-Merlin/connect-from-macos.md) for **MacOS** SSH client and X11 configuration.
Refer to [{Connecting to Merlin7 -> Accessing from MacOS Clients}](../02-connecting-to-merlin/connect-from-macos.md) for **MacOS** SSH client and X11 configuration.
## NoMachine Remote Desktop Access
X applications are supported in the login nodes and can run efficiently through a **NoMachine** client. This is the officially supported way to run more demanding X applications on Merlin7.
* For PSI Windows workstations, this can be installed from the Software Kiosk as 'NX Client'.
* For PSI Windows workstations, this can be installed from the Software Kiosk as 'NX Client'.
* If you have difficulties installing, please request support through **[PSI Service Now](https://psi.service-now.com/psisp)** as an *Incident Request*.
* For other workstations (e.g. macOS, Linux, or Windows workstations with administrator access) the client software can be downloaded from the [Nomachine Website](https://www.nomachine.com/product&p=NoMachine%20Enterprise%20Client).
### Accessing Interactive Nodes with NoMachine
Refer to [{How To Use Merlin -> Remote Desktop Access}](../02-How-To-Use-Merlin/nomachine.md) for further instructions of how to configure the NoMachine client and how to access it from PSI and from outside PSI.
Refer to [{How To Use Merlin -> Remote Desktop Access}](../02-connecting-to-merlin/nomachine.md) for further instructions of how to configure the NoMachine client and how to access it from PSI and from outside PSI.
## Terminal access through Open OnDemand
The [**Merlin7 Open OnDemand**](https://ondemand.psi.ch) service is currently in a ***preview / early-access phase***.
For terminal access to the login nodes through Open OnDemand, please refer to [Open OnDemand -> Merlin7 Shell Access](/merlin7/05-Open-OnDemand/open-ondemand/#merlin-7-shell-access).
For terminal access to the login nodes through Open OnDemand, please refer to [Open OnDemand -> Merlin7 Shell Access](../06-open-ondemand/open-ondemand.md#merlin-7-shell-access).
Currently, only `login002.merlin7.psi.ch` is configured as the only login node available for terminal access through Open OnDemand.
## Login nodes hardware description
@@ -24,10 +24,10 @@ In addition, any job *must be submitted from a high performance storage area vis
The **Merlin7 CPU cluster** (**`merlin7`**) is the default cluster configured in the login nodes. Any job submission will use by default this cluster, unless
the option `--cluster` is specified with another of the existing clusters.
For further information about how to use this cluster, please visit: [**Merlin7 CPU Slurm Cluster documentation**](../03-Slurm-General-Documentation/slurm-configuration.md#cpu-cluster-merlin7).
For further information about how to use this cluster, please visit: [**Merlin7 CPU Slurm Cluster documentation**](../04-slurm-general-documentation/slurm-configuration.md#cpu-cluster-merlin7).
### Merlin7 GPU cluster access
The **Merlin7 GPU cluster** (**`gmerlin7`**) is visible from the login nodes. However, to submit jobs to this cluster, one needs to specify the option `--cluster=gmerlin7` when submitting a job or allocation.
For further information about how to use this cluster, please visit: [**Merlin7 GPU Slurm Cluster documentation**](../03-Slurm-General-Documentation/slurm-configuration.md#gpu-cluster-gmerlin7).
For further information about how to use this cluster, please visit: [**Merlin7 GPU Slurm Cluster documentation**](../04-slurm-general-documentation/slurm-configuration.md#gpu-cluster-gmerlin7).
@@ -22,7 +22,7 @@ The basic principle is courtesy and consideration for other users.
* It is **forbidden** to use the ``/data/user`` or ``/data/project`` for that purpose.
* Always remove files you do not need any more (e.g. core dumps, temporary files) as early as possible. Keep the disk space clean on all nodes.
* Prefer ``/scratch`` over ``/data/scratch/shared`` and _use the latter only when you require the temporary files to be visible from multiple nodes_.
* Read the description in **[Merlin7 directory structure](../02-How-To-Use-Merlin/storage.md#merlin7-directories)** for learning about the correct usage of each partition type.
* Read the description in **[Merlin7 directory structure](../03-how-to-use-merlin/storage.md#merlin7-directories)** for learning about the correct usage of each partition type.
## User and project data
@@ -2,7 +2,7 @@
## SSH without X11 Forwarding
This is the standard method. Official X11 support is provided through [NoMachine](nomachine.md).
This is the standard method. Official X11 support is provided through [NoMachine](../02-connecting-to-merlin/nomachine.md).
For normal SSH sessions, use your SSH client as follows:
```bash
@@ -13,8 +13,8 @@ ssh $username@login002.merlin7.psi.ch
## SSH with X11 Forwarding
Official X11 Forwarding support is through NoMachine. Please follow the document
[Interactive Sessions](../03-Slurm-General-Documentation/interactive-sessions.md) and
[Remote Desktop Access](../02-How-To-Use-Merlin/nomachine.md) for more details. However,
[Interactive Sessions](../04-slurm-general-documentation/interactive-sessions.md) and
[Remote Desktop Access](../02-connecting-to-merlin/nomachine.md) for more details. However,
we provide a small recipe for enabling X11 Forwarding in Linux.
* For enabling client X11 forwarding, add the following to the start of ``~/.ssh/config``
@@ -2,7 +2,7 @@
## SSH without X11 Forwarding
This is the standard method. Official X11 support is provided through [NoMachine](nomachine.md).
This is the standard method. Official X11 support is provided through [NoMachine](../02-connecting-to-merlin/nomachine.md).
For normal SSH sessions, use your SSH client as follows:
```bash
@@ -21,8 +21,8 @@ you have it running before starting a SSH connection with X11 forwarding.
### SSH with X11 Forwarding in MacOS
Official X11 Forwarding support is through NoMachine. Please follow the document
[Interactive Sessions](../03-Slurm-General-Documentation/interactive-sessions.md) and
[Remote Desktop Access](../02-How-To-Use-Merlin/nomachine.md) for more details. However,
[Interactive Sessions](../04-slurm-general-documentation/interactive-sessions.md) and
[Remote Desktop Access](../02-connecting-to-merlin/nomachine.md) for more details. However,
we provide a small recipe for enabling X11 Forwarding in MacOS.
* Ensure that **[XQuartz](https://www.xquartz.org/)** is installed and running in your MacOS.
@@ -25,8 +25,8 @@ If they are missing, you can install them using the Software Kiosk icon on the D
## SSH with PuTTY with X11 Forwarding
Official X11 Forwarding support is through NoMachine. Please follow the document
[Interactive Sessions](../03-Slurm-General-Documentation/interactive-sessions.md) and
[Remote Desktop Access](../02-How-To-Use-Merlin/nomachine.md) for more details. However,
[Interactive Sessions](../04-slurm-general-documentation/interactive-sessions.md) and
[Remote Desktop Access](../02-connecting-to-merlin/nomachine.md) for more details. However,
we provide a small recipe for enabling X11 Forwarding in Windows.
Check, if the **Xming** is installed on the Windows workstation by inspecting the
@@ -0,0 +1,82 @@
# Host Keys
!!! danger "Be vigilant!"
If you have any doubt about the security of your SSH connection, please contact
the admins via one of the routes described the [support pages](../../support/introduction.md)!
**DO NOT COMPLETE THE CONNECTION OR ENTER IN YOUR PASSWORD IF IN DOUBT!!!**
When using SSH to connect to the cluster for the first time, you should be shown a text
prompt (or a popup) that looks similar to:
```text
The authenticity of host 'login001.merlin7.psi.ch (172.30.139.0)' can't be established.
ED25519 key fingerprint is: SHA256:JIRPllRmLDIRcufAQ5qrDCciGyaz/scj5vA0s9yzB48
This host key is known by the following other names/addresses:
~/.ssh/known_hosts:428: 172.30.137.0
Are you sure you want to continue connecting (yes/no/[fingerprint])?
```
This prompt is asking if you want to *trust* the remote system and allow the connection to proceed.
This trust is guaranteed by using a special kind of SSH key called the *host key*. This key is similar
to what is used in TLS/SSL connections, like accessing a website with an address starting with `https://...`.
The host key uniquely identifies the remote host. Unlike TLS/SSL, which uses an intermediate arbitrator
to validate the key (also known as a certificate authority), you as the user need to validate the host key
yourself. A unique **fingerprint** is shown which you can compare to a known value (see below).
The fingerprint is a unique hashed representation of the host key's public key. Typically the
hashing algorithms MD5 or SHA256[^1] are used to generate this string.
All of the login nodes have a unique *host key* which identifies the host, SSH
viessm_h marked this conversation as resolved Outdated
Outdated
Review

Sentence has a problem

Sentence has a problem
will always check this after the first connection to enforce that you are
connecting to the same host. This is en extra level of security to prevent
man-in-the-middle and similar attacks from happening.
If you answer `yes`, the host key will be stored locally (usually in `~/.ssh/known_hosts` but
this can be different depending on what client you are using).
## Currently Valid Host Keys
Below is a current list of our host key fingerprints for our entry nodes. *Users
are strongly encouraged to use this list to double check that for the given
host, it matches what is shown in the prompt above.*
| Node | Key Type | Fingerprints |
|--------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| login001.merlin7.psi.ch | ED25519 | SHA256: `JIRPllRmLDIRcufAQ5qrDCciGyaz/scj5vA0s9yzB48`<br> MD5: `e6:9a:eb:89:83:6e:96:2d:9d:d9:cd:b7:11:f4:03:1f`<br> SHA256 (alt): `1qLHYXRndp4SOJ0U9GHANFtFRkq0mBZyheCmpUK2RKI=`<br> MD5 (alt): `16:92:5d:84:cb:94:03:3f:c6:5e:36:3b:20:dd:36:b5` |
| login002.merlin7.psi.ch | ED25519 | SHA256: `nnP4IeemgKFGEVQd0WF2k7bpwQ+Xcy+XlK5iz6LZz1U`<br> MD5: `7b:85:31:77:83:87:38:c9:72:1c:44:e2:ac:0c:20:55`<br> SHA256 (alt): `58/8g78Y19JNzBYPDXShG9R/+T590rQci7j2E/P6yd4=`<br> MD5 (alt): `49:01:e0:6b:bc:00:be:56:7f:19:a7:9b:63:98:b4:7e` |
| service03.merlin7.psi.ch | ED25519 | SHA256: `CokdMlWmRrH5Yw5UcggbceLiI+cKMGm6oBYiuumLQHA`<br> MD5: `01:0d:4e:cf:17:49:b3:35:f7:79:73:74:0d:78:c8:b2`<br> SHA256 (alt): `47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=`<br> MD5 (alt): `78:ef:05:1d:e9:e2:d4:9d:6c:bb:7c:16:31:c7:e9:7a` |
??? info "Alternative Fingerprint"
Some SSH compatible tools do not use the standard SSH way of generating the
fingerprint string, this is true for tools like
[Cyberduck](https://cyberduck.io/) and instead show an alternative
fingerprint. We provide these as the `<HASH> (alt): ...` entries above.
## Something looks wrong?
If you get a prompt (or popup) that looks like:
```txt
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:nnP4IeemgKFGEVQd0WF2k7bpwQ+Xcy+XlK5iz6LZz1U.
Please contact your system administrator.
Add correct host key in /home/hans/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/hans/.ssh/known_hosts:412
Host key for login002.merlin7.psi.ch has changed and you have requested strict checking.
Host key verification failed.
```
This *can* indicate that the machine you are connecting to is not the one you
intended. If after double-checking your connection settings, you still get an
error message, **[please contact us!](../../support/introduction.md)**
Do not be tempted to modify the cached host keys (under `~/.ssh/known_hosts`)!
[^1]: The hash is additionally encoded into `base64` representation.
@@ -99,13 +99,13 @@ The Merlin7 NoMachine service is managed through a front-end server and back-end
### Architecture Overview
- **Front-End Server**: `merlin7-nx.psi.ch`
- Serves as the entry point for users connecting to the NoMachine service.
- Handles load-balancing and directs users to available back-end nodes.
- Serves as the entry point for users connecting to the NoMachine service.
- Handles load-balancing and directs users to available back-end nodes.
- **Back-End Nodes**:
- `login001.merlin7.psi.ch`
- `login002.merlin7.psi.ch`
- These nodes host the NoMachine desktop service and manage the individual desktop sessions.
- `login001.merlin7.psi.ch`
- `login002.merlin7.psi.ch`
- These nodes host the NoMachine desktop service and manage the individual desktop sessions.
Access to the login node desktops must be initiated through the `merlin7-nx.psi.ch` front-end. The front-end service will distribute sessions across available nodes in the back-end, ensuring optimal resource usage.
@@ -137,5 +137,5 @@ Preventing rescaling can help eliminate "blurriness" in your display, though it
- Display: Choose `Resize remote display` (forces 1:1 pixel sizes)
- Display > Change settings > Quality: Choose medium-best quality
- Display > Change settings > Modify the advanced display settings
- Check: Disable network-adaptive display quality (turns off lossy compression)
- Check: Disable client side image post-processing
- Check: Disable network-adaptive display quality (turns off lossy compression)
- Check: Disable client side image post-processing
@@ -1,11 +1,11 @@
# Configuring SSH Keys in Merlin
# Configuring SSH Key-pair in Merlin
Merlin users sometimes will need to access the different Merlin services without being constantly requested by a password.
One can achieve that with Kerberos authentication, however in some cases some software would require the setup of SSH Keys.
One example is ANSYS Fluent, which, when used interactively, the way of communication between the GUI and the different nodes
is through the SSH protocol, and the use of SSH Keys is enforced.
## Setting up SSH Keys on Merlin
## Setting up the SSH Key-pair
For security reason, users **must always protect SSH Keys with a passphrase**.
@@ -47,11 +47,11 @@ Below are the main steps for using the Data Catalog.
**``/data/project``**. It would be also necessary when the Merlin export server (**``merlin-archive.psi.ch``**)
is down for any reason.
* Archive the dataset:
* Visit [<https://discovery.psi.ch](https://discovery.psi.ch>)
* Visit [https://discovery.psi.ch](https://discovery.psi.ch)
* Click **``Archive``** for the dataset
* The system will now copy the data to the PetaByte Archive at CSCS
* Retrieve data from the catalog:
* Find the dataset on [<https://discovery.psi.ch](https://discovery.psi.ch>) and click **``Retrieve``**
* Find the dataset on [https://discovery.psi.ch](https://discovery.psi.ch) and click **``Retrieve``**
* Wait for the data to be copied to the PSI retrieval system
* Run **``datasetRetriever``** script
@@ -67,7 +67,7 @@ have been assigned a **``p-group``** (e.g. ``p12345``) for the experiment. Other
Groups are usually assigned to a PI, and then individual user accounts are added to the group. This must be done
under user request through PSI Service Now. For existing **a-groups** and **p-groups**, you can follow the standard
central procedures. Alternatively, if you do not know how to do that, follow the Merlin7
**[Requesting extra Unix groups](../01-Quick-Start-Guide/requesting-accounts.md)** procedure, or open
**[Requesting extra Unix groups](../01-quick-start-guide/requesting-accounts.md)** procedure, or open
a **[PSI Service Now](https://psi.service-now.com/psisp)** ticket.
### Documentation
@@ -152,9 +152,9 @@ section below. An example follows:
}
```
It is recommended to use the [ScicatEditor](https://bliven_s.gitpages.psi.ch/SciCatEditor/) for creating metadata files. This is a browser-based tool specifically for ingesting PSI data. Using the tool avoids syntax errors and provides templates for common data sets and options. The finished JSON file can then be downloaded to merlin or copied into a text editor.
It is recommended to use the [ScicatEditor](https://www.scicatproject.org/SciCatEditor/) for creating metadata files. This is a browser-based tool specifically for ingesting PSI data. Using the tool avoids syntax errors and provides templates for common data sets and options. The finished JSON file can then be downloaded to merlin or copied into a text editor.
Another option is to use the SciCat graphical interface from NoMachine. This provides a graphical interface for selecting data to archive. This is particularly useful for data associated with a DUO experiment and p-group. Type `SciCat`` to get started after loading the `datacatalog`` module. The GUI also replaces the the command-line ingestion described below.
Another option is to use the SciCat graphical interface from NoMachine. This provides a graphical interface for selecting data to archive. This is particularly useful for data associated with a DUO experiment and p-group. Type `SciCat` to get started after loading the `datacatalog` module. The GUI also replaces the the command-line ingestion described below.
The following steps can be run from wherever you saved your ``metadata.json``. First, perform a "dry-run" which will check the metadata for errors:
@@ -170,21 +170,20 @@ datasetIngestor --token $SCICAT_TOKEN --ingest --autoarchive metadata.json
You will be asked whether you want to copy the data to the central system:
* If you are on the Merlin cluster and you are archiving data from ``/data/user`` or ``/data/project``, answer 'no' since the data catalog can
directly read the data.
* If you are on a directory other than ``/data/user`` and ``/data/project, or you are on a desktop computer, answer 'yes'. Copying large datasets
to the PSI archive system may take quite a while (minutes to hours).
* If you are on the Merlin cluster and you are archiving data from `/data/user` or `/data/project`, answer 'no' since the data catalog can
directly read the data.
* If you are on a directory other than `/data/user` and `/data/project`, or you are on a desktop computer, answer 'yes'. Copying large datasets
to the PSI archive system may take quite a while (minutes to hours).
If there are no errors, your data has been accepted into the data catalog! From now on, no changes should be made to the ingested data.
This is important, since the next step is for the system to copy all the data to the CSCS Petabyte archive. Writing to tape is slow, so
this process may take several days, and it will fail if any modifications are detected.
If using the ``--autoarchive`` option as suggested above, your dataset should now be in the queue. Check the data catalog:
[<https://discovery.psi.ch](https://discovery.psi.ch>). Your job should have status 'WorkInProgress'. You will receive an email when the ingestion
If using the `--autoarchive` option as suggested above, your dataset should now be in the queue. Check the data catalog:
[https://discovery.psi.ch](https://discovery.psi.ch). Your job should have status 'WorkInProgress'. You will receive an email when the ingestion
is complete.
If you didn't use ``--autoarchive``, you need to manually move the dataset into the archive queue. From **discovery.psi.ch**, navigate to the 'Archive'
If you didn't use `--autoarchive`, you need to manually move the dataset into the archive queue. From **discovery.psi.ch**, navigate to the 'Archive'
tab. You should see the newly ingested dataset. Check the dataset and click **``Archive``**. You should see the status change from **``datasetCreated``** to
**``scheduleArchiveJob``**. This indicates that the data is in the process of being transferred to CSCS.
@@ -192,7 +191,7 @@ After a few days the dataset's status will change to **``datasetOnAchive``** ind
#### Useful commands
Running the datasetIngestor in dry mode (**without** ``--ingest``) finds most errors. However, it is sometimes convenient to find potential errors
Running the datasetIngestor in dry mode (**without** `--ingest`) finds most errors. However, it is sometimes convenient to find potential errors
yourself with simple unix commands.
Find problematic filenames
@@ -231,9 +230,10 @@ find . -name '*#autosave#' -delete
key_cert_check_authority: invalid certificate
Certificate invalid: name is not a listed principal
```
It indicates that no kerberos token was provided for authentication. You can avoid the warning by first running kinit (PSI linux systems).
* For decentral ingestion cases, the copy step is indicated by a message ``Running [/usr/bin/rsync -e ssh -avxz ...``. It is expected that this
* For decentral ingestion cases, the copy step is indicated by a message `Running [/usr/bin/rsync -e ssh -avxz ...`. It is expected that this
step will take a long time and may appear to have hung. You can check what files have been successfully transfered using rsync:
```bash
@@ -245,19 +245,20 @@ step will take a long time and may appear to have hung. You can check what files
* There is currently a limit on the number of files per dataset (technically, the limit is from the total length of all file paths). It is recommended to break up datasets into 300'000 files or less.
* If it is not possible or desirable to split data between multiple datasets, an alternate work-around is to package files into a tarball. For datasets which are already compressed, omit the -z option for a considerable speedup:
```
```bash
tar -f [output].tar [srcdir]
```
Uncompressed data can be compressed on the cluster using the following command:
```
```bash
sbatch /data/software/Slurm/Utilities/Parallel_TarGz.batch -s [srcdir] -t [output].tar -n
```
Run /data/software/Slurm/Utilities/Parallel_TarGz.batch -h for more details and options.
#### Sample ingestion output (datasetIngestor 1.1.11)
<details>
<summary>[Show Example]: Sample ingestion output (datasetIngestor 1.1.11)</summary>
<pre class="terminal code highlight js-syntax-highlight plaintext" lang="plaintext" markdown="false">
@@ -16,14 +16,14 @@ Key Features:
* **Custom Requests:** If a package, version, or feature is missing, users can contact the support team to explore feasibility for installation.
!!! tip
For further information about **PModules** on Merlin7 please refer to the [PSI Modules](../06-Software-Support/pmodules.md) chapter.
For further information about **PModules** on Merlin7 please refer to the [PSI Modules](../07-software-support/pmodules.md) chapter.
### Spack Modules
Merlin7 also provides Spack modules, offering a modern and flexible package management system. Spack supports a wide variety of software packages and versions. For more information, refer to the **external [PSI Spack](https://gitea.psi.ch/HPCE/spack-psi) documentation**.
!!! tip
For further information about **Spack** on Merlin7 please refer to the [Spack](../06-Software-Support/spack.md) chapter.
For further information about **Spack** on Merlin7 please refer to the [Spack](../07-software-support/spack.md) chapter.
### Cray Environment Modules
@@ -37,5 +37,5 @@ Recommendations:
* **General Use:** For most applications, prefer PModules, which ensure stability, backward compatibility, and long-term support.
!!! tip
For further information about **CPE** on Merlin7 please refer to the [Cray Modules](../06-Software-Support/cray-module.env.md) chapter.
For further information about **CPE** on Merlin7 please refer to the [Cray Modules](../07-software-support/cray-module.env.md) chapter.
@@ -103,12 +103,12 @@ The home directories are mounted in the login and computing nodes under the dire
Directory policies:
* Read **[Important: Code of Conduct](../01-Quick-Start-Guide/code-of-conduct.md)** for more information about Merlin7 policies.
* Is **forbidden** to use the home directories for IO-intensive tasks, instead use one of the **[scratch](storage.md#scratch-directories)** areas instead!
* Read **[Important: Code of Conduct](../01-quick-start-guide/code-of-conduct.md)** for more information about Merlin7 policies.
* Is **forbidden** to use the home directories for IO-intensive tasks, instead use one of the **[scratch](#scratch-directories)** areas instead!
* No backup policy is applied for the user home directories: **users are responsible for backing up their data**.
Home directory quotas are defined in a per Lustre project basis. The quota can be checked using the `merlin_quotas` command described
[above](storage.md#how-to-check-quotas).
[above](#how-to-check-quotas).
### Project data directory
@@ -117,7 +117,7 @@ shared by all members of the project (the project's corresponding UNIX group). W
project related storage spaces, since it allows users to coordinate. Also, project spaces have more flexible policies
regarding extending the available storage space.
Scientists can request a Merlin project space as described in **[[Accessing Merlin -> Requesting a Project]](../01-Quick-Start-Guide/requesting-projects.md)**.
Scientists can request a Merlin project space as described in **[[Accessing Merlin -> Requesting a Project]](../01-quick-start-guide/requesting-projects.md)**.
By default, Merlin can offer **general** project space, centrally covered, as long as it does not exceed 10TB (otherwise, it has to be justified).
General Merlin projects might need to be reviewed after one year of their creation.
@@ -140,7 +140,7 @@ lfs quota -h -p $projectid /data
Directory policies:
* Read **[Important: Code of Conduct](../01-Quick-Start-Guide/code-of-conduct.md)** for more information about Merlin7 policies.
* Read **[Important: Code of Conduct](../01-quick-start-guide/code-of-conduct.md)** for more information about Merlin7 policies.
* It is **forbidden** to use the data directories as `/scratch` area during a job's runtime, i.e. for high throughput I/O for a job's temporary files.
* Please Use `/scratch`, `/data/scratch/shared` for this purpose.
* No backups: users are responsible for managing the backups of their data directories.
@@ -178,7 +178,7 @@ and all tasks need to do I/O on the same temporary files.
Scratch directories policies:
* Read **[Important: Code of Conduct](../01-Quick-Start-Guide/code-of-conduct.md)** for more information about Merlin7 policies.
* Read **[Important: Code of Conduct](../01-quick-start-guide/code-of-conduct.md)** for more information about Merlin7 policies.
* By default, *always* use **local** first and only use **shared** if your specific use case requires it.
* Temporary files *must be deleted at the end of the job by the user*.
* Remaining files will be deleted by the system if detected.
@@ -34,7 +34,7 @@ The next chapters contain detailed information about the different transfer meth
## Transferring from within PSI
The following methods transfer data directly via the [login nodes](../01-Quick-Start-Guide/accessing-interactive-nodes.md#login-nodes-hardware-description). They are suitable for use from **within the PSI network**.
The following methods transfer data directly via the [login nodes](../01-quick-start-guide/accessing-interactive-nodes.md#login-nodes-hardware-description). They are suitable for use from **within the PSI network**.
### Secure FTP
@@ -121,7 +121,7 @@ Merlin7 is fully accessible from within the PSI network. To connect from outside
* [VPN](https://www.psi.ch/en/computing/vpn) ([alternate instructions](https://intranet.psi.ch/BIO/ComputingVPN))
* [SSH hopx](https://www.psi.ch/en/computing/ssh-hop)
* Please avoid transferring big amount data through **hop**
* [No Machine](nomachine.md)
* [No Machine](../02-connecting-to-merlin/nomachine.md)
* Remote Interactive Access through [**'nx.psi.ch'**](https://www.psi.ch/en/photon-science-data-services/remote-interactive-access)
* Please avoid transferring big amount of data through **NoMachine**
@@ -111,16 +111,16 @@ salloc: Relinquishing job allocation 165
#### Graphical access
[NoMachine](../02-How-To-Use-Merlin/nomachine.md) is the official supported service for graphical
[NoMachine](../02-connecting-to-merlin/nomachine.md) is the official supported service for graphical
access in the Merlin cluster. This service is running on the login nodes. Check the
document [{Accessing Merlin -> NoMachine}](../02-How-To-Use-Merlin/nomachine.md) for details about
document [{Accessing Merlin -> NoMachine}](../02-connecting-to-merlin/nomachine.md) for details about
how to connect to the **NoMachine** service in the Merlin cluster.
For other non officially supported graphical access (X11 forwarding):
* For Linux clients, please follow [{How To Use Merlin -> Accessing from Linux Clients}](../02-How-To-Use-Merlin/connect-from-linux.md)
* For Windows clients, please follow [{How To Use Merlin -> Accessing from Windows Clients}](../02-How-To-Use-Merlin/connect-from-windows.md)
* For MacOS clients, please follow [{How To Use Merlin -> Accessing from MacOS Clients}](../02-How-To-Use-Merlin/connect-from-macos.md)
* For Linux clients, please follow [{How To Use Merlin -> Accessing from Linux Clients}](../02-connecting-to-merlin/connect-from-linux.md)
* For Windows clients, please follow [{How To Use Merlin -> Accessing from Windows Clients}](../02-connecting-to-merlin/connect-from-windows.md)
* For MacOS clients, please follow [{How To Use Merlin -> Accessing from MacOS Clients}](../02-connecting-to-merlin/connect-from-macos.md)
### 'srun' with x11 support
@@ -34,7 +34,7 @@ queue. Additional customization can be implemented using the *'Optional user
defined line to be added to the batch launcher script'* option. This line is
added to the submission script at the end of other `#SBATCH` lines. Parameters can
be passed to SLURM by starting the line with `#SBATCH`, like in [Running Slurm
Scripts](../03-Slurm-General-Documentation/slurm-examples.md). Some ideas:
Scripts](../04-slurm-general-documentation/slurm-examples.md). Some ideas:
**Request additional memory**
@@ -1,4 +1,5 @@
# ondemand.psi.ch
# Open OnDemand @ PSI
[Open OnDemand](https://www.openondemand.org/) is an open-source, web-based portal that allows researchers, students, and engineers to access High-Performance Computing (HPC) resources from any device using a web browser.
!!! warning "Open OnDemand is currently in preview"
@@ -17,6 +18,7 @@
see the [official Open OnDemand documentation](https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html)
## Accessing Open OnDemand
**Open Ondemand** can be accessed from your browser at <https://ondemand.psi.ch/>.
!!! tip "Recommended browser"
@@ -0,0 +1,144 @@
# ANSYS - CFX
!!! tip
viessm_h marked this conversation as resolved Outdated
Outdated
Review

"It is...."

"It is...."
It is recommended to check which parameters are available in CFX and adapt
the below examples according to your needs. For that, run `cfx5solve
-help` to get a list of options.
## Running CFX jobs
### PModules
We strongly recommend that you use the latest version of ANSYS software available in PModules.
```bash
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2022R1
```
### Interactive: RSM from remote PSI Workstations
viessm_h marked this conversation as resolved
Review

It is

It is
It is possible to run CFX through RSM from remote PSI (Linux or Windows)
Workstation having a local installation of ANSYS CFX and RSM client. For that,
please refer to the **[ANSYS RSM](ansys-rsm.md)** documentation
for further information of how to setup a RSM client for submitting jobs to
Merlin.
### Non-interactive: sbatch
Running jobs with `sbatch` is always the recommended method. This makes the use
of the resources more efficient. Notice that for running non interactive
Mechanical APDL jobs one must specify the `-batch` option.
#### Serial example
This example shows a very basic serial job.
```bash
#!/bin/bash
#SBATCH --job-name=CFX # Job Name
#SBATCH --partition=hourly # Using 'daily' will grant higher priority than 'general'
#SBATCH --time=0-01:00:00 # Time needed for running the job. Must match with 'partition' limits.
#SBATCH --cpus-per-task=1 # Double if hyperthreading enabled
#SBATCH --ntasks-per-core=1 # Double if hyperthreading enabled
#SBATCH --hint=nomultithread # Disable Hyperthreading
#SBATCH --error=slurm-%j.err # Define your error file
module use unstable
module load ANSYS/2020R1-1
# [Optional:BEGIN] Specify your license server if this is not 'lic-ansys.psi.ch'
LICENSE_SERVER=<your_license_server>
export ANSYSLMD_LICENSE_FILE=1055@$LICENSE_SERVER
export ANSYSLI_SERVERS=2325@$LICENSE_SERVER
# [Optional:END]
SOLVER_FILE=/data/user/caubet_m/CFX5/mysolver.in
cfx5solve -batch -def "$JOURNAL_FILE"
```
One can enable hypertheading by defining `--hint=multithread`,
`--cpus-per-task=2` and `--ntasks-per-core=2`. However, this is in general not
recommended, unless one can ensure that can be beneficial.
#### MPI-based example
An example for running CFX using a Slurm batch script is the following:
```bash
#!/bin/bash
#SBATCH --job-name=CFX # Job Name
#SBATCH --partition=hourly # Using 'daily' will grant higher priority than 'general'
#SBATCH --time=0-01:00:00 # Time needed for running the job. Must match with 'partition' limits.
#SBATCH --nodes=1 # Number of nodes
#SBATCH --ntasks=44 # Number of tasks
#SBATCH --cpus-per-task=1 # Double if hyperthreading enabled
#SBATCH --ntasks-per-core=1 # Double if hyperthreading enabled
#SBATCH --hint=nomultithread # Disable Hyperthreading
#SBATCH --error=slurm-%j.err # Define a file for standard error messages
##SBATCH --exclusive # Uncomment if you want exclusive usage of the nodes
module use unstable
module load ANSYS/2020R1-1
# [Optional:BEGIN] Specify your license server if this is not 'lic-ansys.psi.ch'
LICENSE_SERVER=<your_license_server>
export ANSYSLMD_LICENSE_FILE=1055@$LICENSE_SERVER
export ANSYSLI_SERVERS=2325@$LICENSE_SERVER
# [Optional:END]
export HOSTLIST=$(scontrol show hostname | tr '\n' ',' | sed 's/,$//g')
JOURNAL_FILE=myjournal.in
# INTELMPI=no for IBM MPI
# INTELMPI=yes for INTEL MPI
INTELMPI=no
if [ "$INTELMPI" == "yes" ]
then
export I_MPI_DEBUG=4
export I_MPI_PIN_CELL=core
# Simple example: cfx5solve -batch -def "$JOURNAL_FILE" -par-dist "$HOSTLIST" \
# -part $SLURM_NTASKS \
# -start-method 'Intel MPI Distributed Parallel'
cfx5solve -batch -part-large -double -verbose -def "$JOURNAL_FILE" -par-dist "$HOSTLIST" \
-part $SLURM_NTASKS -par-local -start-method 'Intel MPI Distributed Parallel'
else
# Simple example: cfx5solve -batch -def "$JOURNAL_FILE" -par-dist "$HOSTLIST" \
# -part $SLURM_NTASKS \
# -start-method 'IBM MPI Distributed Parallel'
cfx5solve -batch -part-large -double -verbose -def "$JOURNAL_FILE" -par-dist "$HOSTLIST" \
-part $SLURM_NTASKS -par-local -start-method 'IBM MPI Distributed Parallel'
fi
```
In the above example, one can increase the number of *nodes* and/or *ntasks* if needed and combine it
with `--exclusive` whenever needed. In general, **no hypertheading** is recommended for MPI based jobs.
Also, one can combine it with `--exclusive` when necessary. Finally, one can change the MPI technology in `-start-method`
(check CFX documentation for possible values).
## CFX5 Launcher: CFD-Pre/Post, Solve Manager, TurboGrid
Some users might need to visualize or change some parameters when running calculations with the CFX Solver. For running
**TurboGrid**, **CFX-Pre**, **CFX-Solver Manager** or **CFD-Post** one should run it with the **`cfx5` launcher** binary:
```bash
cfx5
```
![CFX5 Launcher Example](../../images/ANSYS/cfx5launcher.png)
/// caption
///
Then, from the launcher, one can open the proper application (i.e. **CFX-Solver
Manager** for visualizing and modifying an existing job run)
For running CFX5 Launcher, is required a proper SSH + X11 Forwarding access
(`ssh -XY`) or *preferrible* **NoMachine**. If **ssh** does not work for you,
please use **NoMachine** instead (which is the supported X based access, and
simpler).
@@ -0,0 +1,167 @@
# ANSYS - Fluent
Is always recommended to check which parameters are available in Fluent and adapt the below example according to your needs.
For that, run `fluent -help` for getting a list of options. However, as when running Fluent one must specify one of the
following flags:
* **2d**: This is a 2D solver with single point precision.
* **3d**: This is a 3D solver with single point precision.
* **2dpp**: This is a 2D solver with double point precision.
* **3dpp**: This is a 3D solver with double point precision.
## Running Fluent jobs
### PModules
Is strongly recommended the use of the latest ANSYS software available in PModules.
```bash
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2024R2
```
### Interactive: RSM from remote PSI Workstations
Is possible to run Fluent through RSM from remote PSI (Linux or Windows)
Workstation having a local installation of ANSYS Fluent and RSM client. For
that, please refer to the [ANSYS RSM](ansys-rsm.md) in the Merlin
documentation for further information of how to setup a RSM client for
submitting jobs to Merlin.
### Non-interactive: sbatch
Running jobs with `sbatch` is always the recommended method. This makes the use
of the resources more efficient. For running it as a job, one needs to run in
no graphical mode (`-g` option).
#### Serial example
This example shows a very basic serial job.
```bash
#!/bin/bash
#SBATCH --job-name=Fluent # Job Name
#SBATCH --partition=hourly # Using 'daily' will grant higher priority than 'general'
#SBATCH --time=0-01:00:00 # Time needed for running the job. Must match with 'partition' limits.
#SBATCH --cpus-per-task=1 # Double if hyperthreading enabled
#SBATCH --hint=nomultithread # Disable Hyperthreading
#SBATCH --error=slurm-%j.err # Define your error file
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2024R2
# [Optional:BEGIN] Specify your license server if this is not 'lic-ansys.psi.ch'
LICENSE_SERVER=<your_license_server>
export ANSYSLMD_LICENSE_FILE=1055@$LICENSE_SERVER
export ANSYSLI_SERVERS=2325@$LICENSE_SERVER
# [Optional:END]
JOURNAL_FILE=/data/user/caubet_m/Fluent/myjournal.in
fluent 3ddp -g -i ${JOURNAL_FILE}
```
One can enable hypertheading by defining `--hint=multithread`, `--cpus-per-task=2` and `--ntasks-per-core=2`.
However, this is in general not recommended, unless one can ensure that can be beneficial.
#### MPI-based example
An example for running Fluent using a Slurm batch script is the following:
```bash
#!/bin/bash
#SBATCH --job-name=Fluent # Job Name
#SBATCH --partition=hourly # Using 'daily' will grant higher priority than 'general'
#SBATCH --time=0-01:00:00 # Time needed for running the job. Must match with 'partition' limits.
#SBATCH --nodes=1 # Number of nodes
#SBATCH --ntasks=44 # Number of tasks
#SBATCH --cpus-per-task=1 # Double if hyperthreading enabled
#SBATCH --ntasks-per-core=1 # Run one task per core
#SBATCH --hint=nomultithread # Disable Hyperthreading
#SBATCH --error=slurm-%j.err # Define a file for standard error messages
##SBATCH --exclusive # Uncomment if you want exclusive usage of the nodes
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2024R2
# [Optional:BEGIN] Specify your license server if this is not 'lic-ansys.psi.ch'
LICENSE_SERVER=<your_license_server>
export ANSYSLMD_LICENSE_FILE=1055@$LICENSE_SERVER
export ANSYSLI_SERVERS=2325@$LICENSE_SERVER
# [Optional:END]
JOURNAL_FILE=/data/user/caubet_m/Fluent/myjournal.in
fluent 3ddp -g -t ${SLURM_NTASKS} -i ${JOURNAL_FILE}
```
In the above example, one can increase the number of *nodes* and/or *ntasks* if needed. One can remove
`--nodes` for running on multiple nodes, but may lead to communication overhead. In general, **no
hyperthreading** is recommended for MPI based jobs. Also, one can combine it with `--exclusive` when necessary.
## Interactive: salloc
Running Fluent interactively is strongly not recommended and one should
whenever possible use `sbatch`. However, sometimes interactive runs are
needed. For jobs requiring only few CPUs (in example, 2 CPUs) **and** for a
short period of time, one can use the login nodes. Otherwise, one must use the
Slurm batch system using allocations:
* For short jobs requiring more CPUs, one can use the Merlin shortest partitions (`hourly`).
* For longer jobs, one can use longer partitions, however, interactive access
is not always possible (depending on the usage of the cluster).
Please refer to the documentation **[Running Interactive
Jobs](../04-slurm-general-documentation/interactive-jobs.md)** for firther information about different
ways for running interactive jobs in the Merlin6 cluster.
### Requirements
#### SSH Keys
Running Fluent interactively requires the use of SSH Keys. This is the way of
communication between the GUI and the different nodes. For doing that, one must
have a **passphrase protected** SSH Key. If the user does not have SSH Keys yet
(simply run **`ls $HOME/.ssh/`** to check whether **`id_rsa`** files exist or
not). For deploying SSH Keys for running Fluent interactively, one should
follow this documentation: **[Configuring SSH Keys](../02-connecting-to-merlin/ssh-keys.md)**
#### List of hosts
For running Fluent using Slurm computing nodes, one needs to get the list of
the reserved nodes. For getting that list, once you have the allocation, one
can run the following command:
```bash
scontrol show hostname
```
This list must be included in the settings as the list of hosts where to run
Fluent. Alternatively, one can give that list as parameter (`-cnf` option) when
running `fluent`, as follows:
??? example "Running Fluent with 'salloc'"
```console
$ salloc --nodes=2 --ntasks=88 --hint=nomultithread --time=0-01:00:00 --partition=test $SHELL
salloc: Pending job allocation 135030174
salloc: job 135030174 queued and waiting for resources
salloc: job 135030174 has been allocated resources
salloc: Granted job allocation 135030174
$ module use unstable
$ module load Pmodules/2.0.4
$ module use merlin
$ module load ANSYS/2024R2
module load: unstable module has been loaded -- ANSYS/2024R2
$ fluent 3ddp -t$SLURM_NPROCS -cnf=$(scontrol show hostname | tr '\n' ',')
$ exit
exit
salloc: Relinquishing job allocation 135030174
salloc: Job allocation 135030174 has been revoked.
```
@@ -0,0 +1,133 @@
# ANSYS HFSS (ElectroMagnetics)
This recipe is intended to show how to run ANSYS HFSS (ElectroMagnetics) in Slurm.
Having in mind that in general, running ANSYS HFSS means running **ANSYS Electronics Desktop**.
## Running HFSS / Electromagnetics jobs
### PModules
Is necessary to run at least ANSYS software **ANSYS/2022R1**, which is available in PModules:
```bash
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2024R2
```
## Remote job submission: HFSS RSM and SLURM
Running jobs through Remote RSM or Slurm is the recommended way for running ANSYS HFSS.
* **HFSS RSM** can be used from ANSYS HFSS installations running on Windows workstations at PSI (as long as are in the internal PSI network).
* **Slurm** can be used when submitting directly from a Merlin login node (i.e. `sbatch` command or interactively from **ANSYS Electronics Desktop**)
### HFSS RSM (from remote workstations)
Running jobs through Remote RSM is the way for running ANSYS HFSS when submitting from an ANSYS HFSS installation on a PSI Windows workstation.
A HFSS RSM service is running on each **Merlin login node**, and the listening port depends on the ANSYS EM version. Current support ANSYS EM RSM
release and associated listening ports are the following:
<table>
<thead>
<tr>
<th scope='col' style="vertical-align:middle;text-align:center;">ANSYS version</th>
<th scope='col' style="vertical-align:middle;text-align:center;">Login nodes</th>
<th scope='col' style="vertical-align:middle;text-align:center;">Listening port</th>
</tr>
</thead>
<tbody>
<tr align="center">
<td>2022R1</td>
<td><font size="2" face="Courier New">merlin-l-001 merlin-l-001 merlin-l-001</font></td>
<td>32958</td>
</tr>
<tr align="center">
<td>2022R2</td>
<td><font size="2" face="Courier New">merlin-l-001 merlin-l-001 merlin-l-001</font></td>
<td>32959</td>
</tr>
<tr align="center">
<td>2023R2</td>
<td><font size="2" face="Courier New">merlin-l-001 merlin-l-001 merlin-l-001</font></td>
<td>32960</td>
</tr>
</tbody>
</table>
Notice that by default ANSYS EM is listening on port **`32958`**, this is the default for **ANSYS/2022R1** only.
* Workstations connecting to the Merlin ANSYS EM service must ensure that **Electronics Desktop** is connecting to the proper port.
* In the same way, the ANSYS Workstation version must be the same as the version running on Merlin.
Notice that *HFSS RSM is not the same RSM provided for other ANSYS products*.
Therefore, the configuration is different from [ANSYS
RSM](ansys-rsm.md).
To setup HFSS RSM for using it with the Merlin cluster, it must be done from the following **ANSYS Electronics Desktop** menu:
1. **[Tools]->[Job Management]->[Select Scheduler]**:
![Select_Scheduler](../../images/ANSYS/HFSS/01_Select_Scheduler_Menu.png)
2. In the new **[Select scheduler]** window, setup the following settings and **Refresh**:
![RSM_Remote_Scheduler](../../images/ANSYS/HFSS/02_Select_Scheduler_RSM_Remote.png)
* **Select Scheduler**: `Remote RSM`.
* **Server**: Add a Merlin login node.
* **User name**: Add your Merlin username.
* **Password**: Add you Merlin username password.
Once *refreshed*, the **Scheduler info** box must provide **Slurm**
information of the server (see above picture). If the box contains that
information, then you can save changes (`OK` button).
3. **[Tools]->[Job Management]->[Submit Job...]**:
![Submit_Job](../../images/ANSYS/HFSS/04_Submit_Job_Menu.png)
4. In the new **[Submite Job]** window, you must specify the location of the **ANSYS Electronics Desktop** binary:
![Product_Path](../../images/ANSYS/HFSS/05_Submit_Job_Product_Path.png)
* In example, for **ANSYS/2022R1**, the location is `/data/software/pmodules/Tools/ANSYS/2021R1/v211/AnsysEM21.1/Linux64/ansysedt.exe`.
### HFSS Slurm (from login node only)
Running jobs through Slurm from **ANSYS Electronics Desktop** is the way for
running ANSYS HFSS when submitting from an ANSYS HFSS installation in a Merlin
login node. **ANSYS Electronics Desktop** usually needs to be run from the
**[Merlin NoMachine](../02-connecting-to-merlin/nomachine.md)** service, which currently runs
on:
* `merlin-l-001.psi.ch`
* `merlin-l-002.psi.ch`
Since the Slurm client is present in the login node (where **ANSYS Electronics
Desktop** is running), the application will be able to detect and to submit
directly to Slurm. Therefore, we only have to configure **ANSYS Electronics
Desktop** to submit to Slurm. This can set as follows:
1. **[Tools]->[Job Management]->[Select Scheduler]**:
![Select_Scheduler](../../images/ANSYS/HFSS/01_Select_Scheduler_Menu.png)
2. In the new **[Select scheduler]** window, setup the following settings and **Refresh**:
![RSM_Remote_Scheduler](../../images/ANSYS/HFSS/03_Select_Scheduler_Slurm.png)
* **Select Scheduler**: `Slurm`.
* **Server**: must point to `localhost`.
* **User name**: must be empty.
* **Password**: must be empty.
The **Server, User name** and **Password** boxes can't be modified, but if
value do not match with the above settings, they should be changed by
selecting another Scheduler which allows editig these boxes (i.e. **RSM
Remote**).
Once *refreshed*, the **Scheduler info** box must provide **Slurm**
information of the server (see above picture). If the box contains that
information, then you can save changes (`OK` button).
@@ -0,0 +1,164 @@
# ANSYS - MAPDL
# ANSYS - Mechanical APDL
Is always recommended to check which parameters are available in Mechanical APDL and adapt the below examples according to your needs.
For that, please refer to the official Mechanical APDL documentation.
## Running Mechanical APDL jobs
### PModules
Is strongly recommended the use of the latest ANSYS software available in PModules.
```bash
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2024R2
```
### Interactive: RSM from remote PSI Workstations
Is possible to run Mechanical through RSM from remote PSI (Linux or Windows)
Workstation having a local installation of ANSYS Mechanical and RSM client.
For that, please refer to the ***[ANSYS RSM](ansys-rsm.md)*** in the Merlin
documentation for further information of how to setup a RSM client for
submitting jobs to Merlin.
### Non-interactive: sbatch
Running jobs with `sbatch` is always the recommended method. This makes the use
of the resources more efficient. Notice that for running non interactive
Mechanical APDL jobs one must specify the `-b` option.
#### Serial example
This example shows a very basic serial job.
```bash
#!/bin/bash
#SBATCH --job-name=MAPDL # Job Name
#SBATCH --partition=hourly # Using 'daily' will grant higher priority than 'general'
#SBATCH --time=0-01:00:00 # Time needed for running the job. Must match with 'partition' limits.
#SBATCH --cpus-per-task=1 # Double if hyperthreading enabled
#SBATCH --ntasks-per-core=1 # Double if hyperthreading enabled
#SBATCH --hint=nomultithread # Disable Hyperthreading
#SBATCH --error=slurm-%j.err # Define your error file
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2024R2
# [Optional:BEGIN] Specify your license server if this is not 'lic-ansys.psi.ch'
LICENSE_SERVER=<your_license_server>
export ANSYSLMD_LICENSE_FILE=1055@$LICENSE_SERVER
export ANSYSLI_SERVERS=2325@$LICENSE_SERVER
# [Optional:END]
SOLVER_FILE=/data/user/caubet_m/MAPDL/mysolver.in
mapdl -b -i "$SOLVER_FILE"
```
One can enable hypertheading by defining `--hint=multithread`,
`--cpus-per-task=2` and `--ntasks-per-core=2`. However, this is in general not
recommended, unless one can ensure that can be beneficial.
#### SMP-based example
This example shows how to running Mechanical APDL in Shared-Memory Parallelism
mode. It limits the use to 1 single node, but by using many cores. In the
example below, we use a node by using all his cores and the whole memory.
```bash
#!/bin/bash
#SBATCH --job-name=MAPDL # Job Name
#SBATCH --partition=hourly # Using 'daily' will grant higher priority than 'general'
#SBATCH --time=0-01:00:00 # Time needed for running the job. Must match with 'partition' limits.
#SBATCH --nodes=1 # Number of nodes
#SBATCH --ntasks=1 # Number of tasks
#SBATCH --cpus-per-task=44 # Double if hyperthreading enabled
#SBATCH --hint=nomultithread # Disable Hyperthreading
#SBATCH --error=slurm-%j.err # Define a file for standard error messages
#SBATCH --exclusive # Uncomment if you want exclusive usage of the nodes
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2024R2
# [Optional:BEGIN] Specify your license server if this is not 'lic-ansys.psi.ch'
LICENSE_SERVER=<your_license_server>
export ANSYSLMD_LICENSE_FILE=1055@$LICENSE_SERVER
export ANSYSLI_SERVERS=2325@$LICENSE_SERVER
# [Optional:END]
SOLVER_FILE=/data/user/caubet_m/MAPDL/mysolver.in
mapdl -b -np ${SLURM_CPUS_PER_TASK} -i "$SOLVER_FILE"
```
In the above example, one can reduce the number of **cpus per task**. Here
usually `--exclusive` is recommended if one needs to use the whole memory.
For **SMP** runs, one might try the hyperthreading mode by doubling the proper
settings (`--cpus-per-task`), in some cases it might be beneficial.
Please notice that `--ntasks-per-core=1` is not defined here, this is because
we want to run 1 task on many cores! As an alternative, one can explore
`--ntasks-per-socket` or `--ntasks-per-node` for fine grained configurations.
#### MPI-based example
This example enables Distributed ANSYS for running Mechanical APDL using a Slurm batch script.
```bash
#!/bin/bash
#SBATCH --job-name=MAPDL # Job Name
#SBATCH --partition=hourly # Using 'daily' will grant higher priority than 'general'
#SBATCH --time=0-01:00:00 # Time needed for running the job. Must match with 'partition' limits.
#SBATCH --nodes=1 # Number of nodes
#SBATCH --ntasks=44 # Number of tasks
#SBATCH --cpus-per-task=1 # Double if hyperthreading enabled
#SBATCH --ntasks-per-core=1 # Run one task per core
#SBATCH --hint=nomultithread # Disable Hyperthreading
#SBATCH --error=slurm-%j.err # Define a file for standard error messages
##SBATCH --exclusive # Uncomment if you want exclusive usage of the nodes
module use unstable
module load Pmodules/2.0.4
module use merlin
module load ANSYS/2024R2
# [Optional:BEGIN] Specify your license server if this is not 'lic-ansys.psi.ch'
LICENSE_SERVER=<your_license_server>
export ANSYSLMD_LICENSE_FILE=1055@$LICENSE_SERVER
export ANSYSLI_SERVERS=2325@$LICENSE_SERVER
# [Optional:END]
SOLVER_FILE=input.dat
# INTELMPI=no for IBM MPI
# INTELMPI=yes for INTEL MPI
INTELMPI=no
if [ "$INTELMPI" == "yes" ]
then
# When using -mpi=intelmpi, KMP Affinity must be disabled
export KMP_AFFINITY=disabled
# INTELMPI is not aware about distribution of tasks.
# - We need to define tasks distribution.
HOSTLIST=$(srun hostname | sort | uniq -c | awk '{print $2 ":" $1}' | tr '\n' ':' | sed 's/:$/\n/g')
mapdl -b -dis -mpi intelmpi -machines $HOSTLIST -np ${SLURM_NTASKS} -i "$SOLVER_FILE"
else
# IBMMPI (default) will be aware of the distribution of tasks.
# - In principle, no need to force tasks distribution
mapdl -b -dis -mpi ibmmpi -np ${SLURM_NTASKS} -i "$SOLVER_FILE"
fi
```
In the above example, one can increase the number of *nodes* and/or *ntasks* if
needed and combine it with `--exclusive` when necessary. In general, **no
hypertheading** is recommended for MPI based jobs. Also, one can combine it
with `--exclusive` when necessary.
@@ -1,11 +1,11 @@
<!-- docs/merlin7/06-Software-Support/includes/open-ondemand-alternative.md -->
As an alternative to creating an `interactive` session with or without X11 forwarding, you can also use [Open OnDemand on Merlin7](../../05-Open-OnDemand/open-ondemand.md).
As an alternative to creating an `interactive` session with or without X11 forwarding, you can also use [Open OnDemand on Merlin7](../../06-open-ondemand/open-ondemand.md).
[Open OnDemand](https://www.openondemand.org/) provides web-based access to HPC resources and can be used to start graphical applications such as **VS Code** or **MATLAB** directly from a browser. Sometimes, this can be more convenient than setting up local SSH configuration, X11 forwarding, or VS Code Remote SSH manually.
!!! tip
If you mainly need a graphical VS Code or MATLAB session, [Open OnDemand on Merlin7](../../05-Open-OnDemand/open-ondemand.md) is often the easiest option.
If you mainly need a graphical VS Code or MATLAB session, [Open OnDemand on Merlin7](../../06-open-ondemand/open-ondemand.md) is often the easiest option.
!!! warning
The same login-node rule still applies: **do not run heavy interactive applications directly on login nodes.**
@@ -3,14 +3,17 @@
## Best practices when running MATLAB
* Use `matlab -batch` for non-interactive production jobs. It is cleaner and better suited for batch execution than starting the full GUI.
* Prefer `-nodesktop -nosplash` for interactive terminal-based sessions on compute nodes in case of Slurm allocations (e.g. `salloc` or [interactive Slurm sessions](../03-Slurm-General-Documentation/interactive-sessions.md).
* Prefer `-nodesktop -nosplash` for interactive terminal-based sessions on compute nodes in case of Slurm allocations (e.g. `salloc` or [interactive Slurm sessions](../04-slurm-general-documentation/interactive-sessions.md).
* If using MATLAB parallel features (`parpool`), make sure the number of workers matches the CPUs requested from Slurm.
* Write temporary files to appropriate scratch storage if large I/O is expected.
* For batch jobs, test first with a small input and short walltime before scaling up.
## MATLAB via interactive sessions
MATLAB for interactive usage **should be always started on a compute node through** an [interactive Slurm session](../03-Slurm-General-Documentation/interactive-sessions.md), and **never directly on a login node**.
MATLAB for interactive usage **should be always started on a compute node
through** an [interactive Slurm
session](../04-slurm-general-documentation/interactive-sessions.md), and
**never directly on a login node**.
!!! danger "Do not run MATLAB on login nodes"
Do **not** start MATLAB directly on Merlin7 login nodes.
@@ -20,13 +23,13 @@ MATLAB for interactive usage **should be always started on a compute node throug
process **being terminated by administrators**.
For any real computation, start an interactive or batch job with Slurm and run MATLAB inside that allocation.
Start an [interactive session](../03-Slurm-General-Documentation/interactive-sessions.md) first, then run
Start an [interactive session](../04-slurm-general-documentation/interactive-sessions.md) first, then run
MATLAB on the allocated compute node.
There are two common ways to use MATLAB interactively via `interactive` sessions:
- **Option 1:** [with X11-forwarding](#option-1-matlab-with-x11-forwarding), for the full graphical MATLAB desktop
- **Option 2:** [without X11-forwarding](#option-2-matlab-without-x11), for terminal-based MATLAB sessions or batch-like interactive testing
* **Option 1:** [with X11-forwarding](#option-1-matlab-with-x11-forwarding), for the full graphical MATLAB desktop
* **Option 2:** [without X11-forwarding](#option-2-matlab-without-x11), for terminal-based MATLAB sessions or batch-like interactive testing
Choosing between X11 and non-X11 mode:
@@ -226,5 +229,5 @@ In general, the number of MATLAB workers should not exceed the number of CPUs al
## MATLAB via Open OnDemand
<!-- Include open-ondemand-alternative information -->
{% include-markdown "merlin7/06-Software-Support/includes/open-ondemand-alternative.md" %}
{% include-markdown "merlin7/07-software-support/includes/open-ondemand-alternative.md" %}
@@ -1,15 +1,19 @@
# Visual Studio Code
Visual Studio Code can be used on Merlin7 with the **Remote - SSH** extension. The recommended workflow is to start a normal non-X11 interactive Slurm session and connect VS Code to the allocated compute node.
Visual Studio Code can be used on Merlin7 with the **Remote - SSH** extension.
The recommended workflow is to start a normal non-X11 interactive Slurm session
and connect VS Code to the allocated compute node.
!!! danger "Do not run VS Code on login nodes"
Do **not** start VS Code directly on Merlin7 login nodes.
Login nodes are not meant for CPU-intensive, memory-intensive, or long-running VS Code workloads.
Running VS Code there **can slow down the system for all users** and may lead to your session or
process **being terminated by administrators**.
Login nodes are not meant for CPU-intensive, memory-intensive, or
long-running VS Code workloads. Running VS Code there **can slow down the
system for all users** and may lead to your session or process **being
terminated by administrators**.
Start an [interactive session](../03-Slurm-General-Documentation/interactive-sessions.md) first,
Start an [interactive
session](../04-slurm-general-documentation/interactive-sessions.md) first,
then connect VS Code to the allocated compute node.
!!! warning "Do not use X11 mode for VS Code"
@@ -19,7 +23,8 @@ Visual Studio Code can be used on Merlin7 with the **Remote - SSH** extension. T
interactive
```
Do **not** use `interactive --x11` for VS Code Remote SSH. X11 sessions are not detachable and are intended for foreground graphical applications.
Do **not** use `interactive --x11` for VS Code Remote SSH. X11 sessions are
not detachable and are intended for foreground graphical applications.
## VS Code via interactive sessions
@@ -40,7 +45,8 @@ interactive -- --cpus-per-task=8 --mem=16G
!!! tip "Request resources with --cpus-per-task"
For most VS Code workflows, request CPUs with `--cpus-per-task`.
Avoid requesting more CPUs or memory than needed for an editor, terminal, notebook, or compile session.
Avoid requesting more CPUs or memory than needed for an editor, terminal,
notebook, or compile session.
The allocation is detachable. You may leave the shell with:
@@ -76,7 +82,8 @@ Copy the printed `Host` block into the SSH config file on your **local computer*
!!! tip "Use --print-vscode after each new allocation"
The allocated compute node may change whenever you create a new session.
Run this command again after creating a new allocation and update your local SSH config:
Run this command again after creating a new allocation and update your
local SSH config:
```bash
interactive --print-vscode
@@ -140,4 +147,4 @@ If it has ended, create a new allocation and regenerate the VS Code SSH config.
## VS Code via Open OnDemand
<!-- Include open-ondemand-alternative information -->
{% include-markdown "merlin7/06-Software-Support/includes/open-ondemand-alternative.md" %}
{% include-markdown "merlin7/07-software-support/includes/open-ondemand-alternative.md" %}
@@ -61,7 +61,7 @@ Before starting the migration, make sure you:
* are **registered on Merlin7**.
* If not yet registered, please do so following [these instructions](../01-Quick-Start-Guide/requesting-accounts.md)
* If not yet registered, please do so following [these instructions](../01-quick-start-guide/requesting-accounts.md)
* **have cleaned up your data to reduce migration time and space usage**.
+1 -1
View File
@@ -10,6 +10,6 @@ tags:
The Merlin7 cluster is officially in preproduction. This phase will be tested by a few users
and slowly we will contact other users to be part of it. Keep in mind that access is restricted.
Merlin7 documentation is now available at [Slurm configuration](../../merlin7/03-Slurm-General-Documentation/slurm-configuration.md).
Merlin7 documentation is now available at [Slurm configuration](../../merlin7/04-slurm-general-documentation/slurm-configuration.md).
More complete documentation will be coming shortly.
+3 -3
View File
@@ -6,7 +6,7 @@ title: "FAQ"
## How do I register for Merlin?
See [Requesting Merlin Access](../merlin7/01-Quick-Start-Guide/requesting-accounts.md).
See [Requesting Merlin Access](../merlin7/01-quick-start-guide/requesting-accounts.md).
## How do I get information about downtimes and updates?
@@ -17,7 +17,7 @@ See [Get updated through the Merlin User list!](introduction.md#merlin-user-mail
Merlin projects are placed in the `/data/project` directory. Access to each
project is controlled by Unix group membership. If you require access to an
existing project, please request group membership as described in
[Requesting Unix Group Membership](../merlin7/01-Quick-Start-Guide/requesting-projects.md#requesting-unix-group-membership).
[Requesting Unix Group Membership](../merlin7/01-quick-start-guide/requesting-projects.md#requesting-unix-group-membership).
Your project leader or project colleagues will know what Unix group you should
belong to. Otherwise, you can check what Unix group is allowed to access that
@@ -29,7 +29,7 @@ Most software can be installed in user directories without any special
permissions. We recommend using `/data/user/$USER/bin` for software since home
directories are fairly small. For software that will be used by multiple
groups/users you can also [request the admins](introduction.md) install it as a
[module](../merlin7/06-Software-Support/pmodules.md).
[module](../merlin7/07-software-support/pmodules.md).
How to install depends a bit on the software itself. There are three common
installation procedures:
+2 -2
View File
@@ -133,9 +133,9 @@ module load paraview
vglrun paraview
```
Officially, the supported method for running `vglrun` is by using the [NoMachine remote desktop](../merlin7/02-How-To-Use-Merlin/nomachine.md).
Officially, the supported method for running `vglrun` is by using the [NoMachine remote desktop](../merlin7/02-connecting-to-merlin/nomachine.md).
Running `vglrun` it's also possible using SSH with X11 Forwarding. However, it's very slow and it's only recommended when running
in Slurm (from [NoMachine](../merlin7/02-How-To-Use-Merlin/nomachine.md)). Please, avoid running `vglrun` over SSH from a desktop or laptop.
in Slurm (from [NoMachine](../merlin7/02-connecting-to-merlin/nomachine.md)). Please, avoid running `vglrun` over SSH from a desktop or laptop.
## Software
+47 -42
View File
@@ -66,6 +66,8 @@ plugins:
- 'cscs*/*.md'
Support:
- 'support/*.md'
News:
- 'news/*.md'
# Extensions
markdown_extensions:
@@ -103,53 +105,56 @@ nav:
- news/index.md
- Merlin 7:
- Quick Start Guide:
- merlin7/01-Quick-Start-Guide/introduction.md
- merlin7/01-Quick-Start-Guide/code-of-conduct.md
- merlin7/01-Quick-Start-Guide/requesting-accounts.md
- merlin7/01-Quick-Start-Guide/requesting-projects.md
- merlin7/01-Quick-Start-Guide/accessing-interactive-nodes.md
- merlin7/01-Quick-Start-Guide/accessing-slurm.md
- merlin7/01-quick-start-guide/introduction.md
- merlin7/01-quick-start-guide/code-of-conduct.md
- merlin7/01-quick-start-guide/requesting-accounts.md
- merlin7/01-quick-start-guide/requesting-projects.md
- merlin7/01-quick-start-guide/accessing-interactive-nodes.md
- merlin7/01-quick-start-guide/accessing-slurm.md
- Connecting to Merlin7:
- merlin7/02-connecting-to-merlin/host-keys.md
- SSH:
- merlin7/02-connecting-to-merlin/connect-from-linux.md
- merlin7/02-connecting-to-merlin/connect-from-windows.md
- merlin7/02-connecting-to-merlin/connect-from-macos.md
- SSH Key-pairs: merlin7/02-connecting-to-merlin/ssh-keys.md
- Remote Desktop: merlin7/02-connecting-to-merlin/nomachine.md
- How To Use Merlin7:
- merlin7/02-How-To-Use-Merlin/connect-from-linux.md
- merlin7/02-How-To-Use-Merlin/connect-from-windows.md
- merlin7/02-How-To-Use-Merlin/connect-from-macos.md
- merlin7/02-How-To-Use-Merlin/storage.md
- merlin7/02-How-To-Use-Merlin/transfer-data.md
- merlin7/02-How-To-Use-Merlin/archive.md
- merlin7/02-How-To-Use-Merlin/nomachine.md
- merlin7/02-How-To-Use-Merlin/merlin-rmount.md
- merlin7/02-How-To-Use-Merlin/merlin_tools.md
- merlin7/02-How-To-Use-Merlin/ssh-keys.md
- merlin7/02-How-To-Use-Merlin/kerberos.md
- merlin7/02-How-To-Use-Merlin/software-repositories.md
- merlin7/03-how-to-use-merlin/storage.md
- merlin7/03-how-to-use-merlin/transfer-data.md
- merlin7/03-how-to-use-merlin/archive.md
- merlin7/03-how-to-use-merlin/merlin-rmount.md
- merlin7/03-how-to-use-merlin/merlin_tools.md
- merlin7/03-how-to-use-merlin/kerberos.md
- merlin7/03-how-to-use-merlin/software-repositories.md
- Slurm General Documentation:
- merlin7/03-Slurm-General-Documentation/merlin7-configuration.md
- merlin7/03-Slurm-General-Documentation/slurm-configuration.md
- merlin7/03-Slurm-General-Documentation/interactive-sessions.md
- merlin7/03-Slurm-General-Documentation/interactive-jobs.md
- merlin7/03-Slurm-General-Documentation/slurm-examples.md
- merlin7/03-Slurm-General-Documentation/slurm-mail.md
- merlin7/04-slurm-general-documentation/merlin7-configuration.md
- merlin7/04-slurm-general-documentation/slurm-configuration.md
- merlin7/04-slurm-general-documentation/interactive-sessions.md
- merlin7/04-slurm-general-documentation/interactive-jobs.md
- merlin7/04-slurm-general-documentation/slurm-examples.md
- merlin7/04-slurm-general-documentation/slurm-mail.md
- Jupyterhub:
- merlin7/04-Jupyterhub/jupyterhub.md
- merlin7/05-jupyterhub/jupyterhub.md
- Open OnDemand:
- merlin7/05-Open-OnDemand/open-ondemand.md
- Introduction: merlin7/06-open-ondemand/open-ondemand.md
- Software Support:
- merlin7/06-Software-Support/pmodules.md
- merlin7/06-Software-Support/openmpi.md
- merlin7/06-Software-Support/mpich.md
- merlin7/06-Software-Support/cray-module.env.md
- merlin7/06-Software-Support/ansys.md
- merlin7/06-Software-Support/ansys-rsm.md
- merlin7/06-Software-Support/cp2k.md
- merlin7/06-Software-Support/dirac.md
- merlin7/06-Software-Support/gromacs.md
- merlin7/06-Software-Support/ippl.md
- merlin7/06-Software-Support/lammps.md
- merlin7/06-Software-Support/matlab.md
- merlin7/06-Software-Support/opal-x.md
- merlin7/06-Software-Support/quantum-espresso.md
- merlin7/06-Software-Support/spack.md
- merlin7/06-Software-Support/vscode.md
- merlin7/07-software-support/pmodules.md
- merlin7/07-software-support/openmpi.md
- merlin7/07-software-support/mpich.md
- merlin7/07-software-support/cray-module.env.md
- merlin7/07-software-support/ansys.md
- merlin7/07-software-support/ansys-rsm.md
- merlin7/07-software-support/cp2k.md
- merlin7/07-software-support/dirac.md
- merlin7/07-software-support/gromacs.md
- merlin7/07-software-support/ippl.md
- merlin7/07-software-support/lammps.md
- merlin7/07-software-support/matlab.md
- merlin7/07-software-support/opal-x.md
- merlin7/07-software-support/quantum-espresso.md
- merlin7/07-software-support/spack.md
- merlin7/07-software-support/vscode.md
- Merlin 6:
- Quick Start Guide:
- merlin6/introduction.md