diff --git a/_data/sidebars/merlin7_sidebar.yml b/_data/sidebars/merlin7_sidebar.yml index 0ae1d18..6a6baa4 100644 --- a/_data/sidebars/merlin7_sidebar.yml +++ b/_data/sidebars/merlin7_sidebar.yml @@ -28,10 +28,20 @@ entries: url: /merlin7/connect-from-macos.html - title: Merlin6 Storage url: /merlin7/storage.html - - title: Cray systems modules + - title: Transferring Data + url: /merlin7/transfer-data.html + # - title: Archive & PSI Data Catalog + # url: /merlin7/archive.html + - title: Remote Desktop Access + url: /merlin7/nomachine.html + - title: Configuring SSH Keys + url: /merlin7/ssh-keys.html + - title: Kerberos and AFS authentication + url: /merlin7/kerberos.html + - title: Software repository - Cray Modules url: /merlin7/cray-module-env.html - - title: Transferring files - url: /merlin7/file-transfers.html + - title: Software repository - PModules + url: /merlin7/using-modules.html - title: Slurm General Documentation folderitems: - title: Merlin7 Infrastructure diff --git a/pages/merlin7/02-How-To-Use-Merlin/cray-module-env.md b/pages/merlin7/02-How-To-Use-Merlin/cray-module-env.md index 59c52d3..9906e0b 100644 --- a/pages/merlin7/02-How-To-Use-Merlin/cray-module-env.md +++ b/pages/merlin7/02-How-To-Use-Merlin/cray-module-env.md @@ -15,38 +15,40 @@ Please do not use or rely on this documentation until this becomes official. This applies to any page under https://lsm-hpce.gitpages.psi.ch/merlin7/ {{site.data.alerts.end}} -To switch from the PSI Module Environment to the provided Cray Programming Environment (CPE), please do the following -Switch to Cray PrgEnv +Cray Programming Environment, with Cray's compilers and MPI, are loaded by default. You can check with module list command or use the short form as shown below: - $ source /etc/bash.bashrc.local.2023-04-26T164631 +```bash +🔥 [caubet_m@login001:~/slurm/ior]# module list +Currently Loaded Modules: + 1) craype-x86-rome 2) libfabric/1.15.2.0 + 3) craype-network-ofi + 4) xpmem/2.8.2-1.0_5.1__g84a27a5.shasta 5) PrgEnv-cray/8.5.0 + 6) cce/17.0.0 7) cray-dsmml/0.2.2 + 8) cray-libsci/23.12.5 9) cray-mpich/8.1.28 +10) craype/2.7.30 11) perftools-base/23.12.0 +12) cpe/23.12 13) cray/23.12 +``` +You will notice an unfamiliar PrgEnv-cray/8.5.0 that was loaded. This is a meta-module that Cray provides to simplify the switch of compilers and their associated dependencies and libraries, +as a whole called Programming Environment. In the Cray Programming Environment, there are 4 key modules. -Cray Programming Environment, with Cray's compilers and MPI are loaded by default. You can check with module list command or use the short form as shown below. -ml +* `cray-libsci` is a collection of numerical routines tuned for performance on Cray systems. +* `libfabric` is an important low-level library that allows you to take advantage of the high performance Slingshot11 network. +* `cray-mpich` is a CUDA-aware MPI implementation. +* `cce` is the compiler from Cray. C/C++ compilers are based on Clang/LLVM while Fortran supports Fortran 2018 standard. More info: https://user.cscs.ch/computing/compilation/cray/ - $ ml #means: module list - Currently Loaded Modules: - 1) craype-x86-rome 4) perftools-base/21.12.0 7) craype/2.7.13 10) cray-libsci/21.08.1.2 - 2) libfabric/1.15.2.0 5) xpmem/2.4.4-2.3_13.8__gff0e1d9.shasta 8) cray-dsmml/0.2.2 11) PrgEnv-cray/8.3.0 - 3) craype-network-ofi 6) cce/13.0.0 9) cray-mpich/8.1.12 +You can switch between different programming environments. You can check the available module with the `module avail` command, as follows: -You will notice an unfamiliar PrgEnv-cray/8.3.0 that was loaded. This is meta-module that Cray provides to simplify the switch of compilers and their associated dependencies and libraries, as a whole called Programming Environment. In the Cray Programming Environment, there are 4 key modules. - - cray-libsci is a collection of numerical routines tuned for performance on Cray systems. - libfabric is an important low-level library that allows you to take advantage of the high performance Slingshot11 network. - cray-mpich is a CUDA-aware MPI implementation. - cce is the compiler from Cray. C/C++ compilers are based on Clang/LLVM while Fortran supports Fortran 2018 standard. More info: https://user.cscs.ch/computing/compilation/cray/ - -You can switch between different programming environments. You can check the available module with module avail command or the short form (ml av)as shown below. -ml av - $ ml av PrgEnv - - PrgEnv-aocc/8.3.0 (D) PrgEnv-cray/8.3.3 PrgEnv-intel/8.3.0 (D) PrgEnv-nvidia/8.3.0 (D) - PrgEnv-aocc/8.3.3 PrgEnv-gnu/8.3.0 (D) PrgEnv-intel/8.3.3 PrgEnv-nvidia/8.3.3 - PrgEnv-cray/8.3.0 (L,D) PrgEnv-gnu/8.3.3 PrgEnv-nvhpc/8.3.3 +```bash +🔥 [caubet_m@login001:~/slurm/ior]# module avail PrgEnv +--------------------- /opt/cray/pe/lmod/modulefiles/core --------------------- +PrgEnv-aocc/8.5.0 PrgEnv-cray/8.5.0 +PrgEnv-gnu/8.5.0 PrgEnv-intel/8.5.0 +``` +{% comment %} If you want an in-depth information on the different programming environments, you can use the module spider command. This command allows you to explore the hierarchical structure of Lmod module environment that is in use here. -module spider +```bash $ module spider PrgEnv-cray --------------------------------------------------------------------------------------------------------------------------------------------------- PrgEnv-cray: @@ -63,35 +65,45 @@ module spider $ module spider PrgEnv-cray/8.3.3 --------------------------------------------------------------------------------------------------------------------------------------------------- +``` +{% endcomment %} ## Switching compiler suites Compiler suites can be exchanged with PrgEnv (Programming Environments) provided by HPE-Cray. The wrappers call the correct compiler with appropriate options to build and link applications with relevant libraries, as required by the loaded modules (only dynamic linking is supported) and therefore should replace direct calls to compiler drivers in Makefiles and build scripts. -To swap the the compiler suite from the default Cray to Intel, you can do the following. If there are no PrgEnv loaded, then you can load the one you prefer with ml command. -Swapping PrgEnv +To swap the the compiler suite from the default Cray to Intel, you can do the following. - $ module swap PrgEnv-cray PrgEnv-intel +```bash +🔥 [caubet_m@login001:~]# module swap PrgEnv-cray/8.5.0 PrgEnv-gnu/8.5.0 -Please note that in a Cray provided PrgEnv, cray-mpich will always be used by default. This is because this MPI library has been strongly optimised for a Cray system. In the case of Intel, cray-mpich has been compiled with Intel compiler to improve interoperability. Please note that the same condition applies when you use the GNU and AMD (AOCC) programming environments. +Lmod is automatically replacing "cce/17.0.0" with "gcc-native/12.3". + +🔥 [caubet_m@login001:~]# module list +Currently Loaded Modules: + 1) craype-x86-rome 2) libfabric/1.15.2.0 + 3) craype-network-ofi + 4) xpmem/2.8.2-1.0_5.1__g84a27a5.shasta 5) perftools-base/23.12.0 + 6) cpe/23.12 7) cray/23.12 + 8) gcc-native/12.3 9) craype/2.7.30 +10) cray-dsmml/0.2.2 11) cray-mpich/8.1.28 +12) cray-libsci/23.12.5 13) PrgEnv-gnu/8.5.0 +``` + +Please note that in a Cray provided `PrgEnv`, `cray-mpich` will always be used by default. This is because this MPI library has been strongly optimised for a Cray system. In the case of Intel, `cray-mpich` has been compiled with Intel compiler to improve interoperability. Please note that the same condition applies when you use the GNU and AMD (AOCC) programming environments. If you would like to use a pure Intel-MPI, please refer to the the advanced guide on how to install and set up Intel such that the optimised Slingshot11 network is used. -You can switch versions of intel compilers by using the module swap command. If you want to know the available versions of intel compilers, you can use the module avail or module spider commands. +You can switch versions of intel compilers by using the module swap command. If you want to know the available versions of intel compilers, you can use the `module avail` command. -An example to switch the version of Cray compiler while you are in the PrgEnv-cray environment -module swap - - $ module swap cce/13.0.0 cce/14.0.0 - -Due to the use of wrapper, you can use the same commands when compiling with say cce or gnu compilers. In the case of intel compilers, you have to use the original commands, e.g. icc. - - C compiler : cc - C++ compiler : CC - Fortran compiler : ftn - MPI C compiler: mpicc - MPI C++ compiler: mpic++ - MPI Fortran compiler: mpif90 - -When using gnu compiler, you will need to specify the architecture (-march or -mtune or --offload-arch) you would like to optimise your code for, in this case Milan, you need to use craype-x86-milan. +Due to the use of wrapper, you can use the same commands when compiling with say `cce` or `gnu` compilers. In the case of `intel` compilers, you have to use the original commands, e.g. `icc`. +```bash +C compiler : cc +C++ compiler : CC +Fortran compiler : ftn +MPI C compiler: mpicc +MPI C++ compiler: mpic++ +MPI Fortran compiler: mpif90 +``` +When using `gnu` compiler, you will need to specify the architecture (`-march` or `-mtune` or `--offload-arch`) you would like to optimise your code for, in this case Milan, you need to use `craype-x86-milan`. diff --git a/pages/merlin7/02-How-To-Use-Merlin/file-transfers.md b/pages/merlin7/02-How-To-Use-Merlin/file-transfers.md deleted file mode 100644 index fa7be4d..0000000 --- a/pages/merlin7/02-How-To-Use-Merlin/file-transfers.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: Transferring files between systems -#tags: -keywords: files, transfer, scp -last_updated: 24 Mai 2023 -summary: "This document describes some possibilities to transfer files from Merlin7 to Merlin7." -sidebar: merlin7_sidebar -permalink: /merlin7/file-transfers.html ---- - -![Work In Progress](/images/WIP/WIP1.webp){:style="display:block; margin-left:auto; margin-right:auto"} - -{{site.data.alerts.warning}}The Merlin7 documentation is Work In Progress. -Please do not use or rely on this documentation until this becomes official. -This applies to any page under https://lsm-hpce.gitpages.psi.ch/merlin7/ -{{site.data.alerts.end}} - -From Merlin7 to Merlin7 you can use (ideally SSH keys should be set beforehand): - - $ rsync -avAHXS ~/merlin7_localdata $USER@psi-dev.cscs.ch:/scratch/home/$USER/ diff --git a/pages/merlin7/02-How-To-Use-Merlin/transfer-data.md b/pages/merlin7/02-How-To-Use-Merlin/transfer-data.md index 369f8b6..dd17171 100644 --- a/pages/merlin7/02-How-To-Use-Merlin/transfer-data.md +++ b/pages/merlin7/02-How-To-Use-Merlin/transfer-data.md @@ -68,6 +68,7 @@ The purpose of the software is to send a large file to someone, have that file a **[SWITCHfilesender](https://filesender.switch.ch/filesender2/?s=upload)** is fully integrated with PSI, therefore, PSI employees can log in by using their PSI account (through Authentication and Authorization Infrastructure / AAI, by selecting PSI as the institution to be used for log in). +{% comment %} ## PSI Data Transfer From August 2024, Merlin is connected to the **[PSI Data Transfer](https://www.psi.ch/en/photon-science-data-services/data-transfer)** service, @@ -168,3 +169,4 @@ provides a helpful wrapper over the Gnome storage utilities, and provides suppor [More instruction on using `merlin_rmount`](/merlin7/merlin-rmount.html) +{% endcomment %} diff --git a/pages/merlin7/02-How-To-Use-Merlin/using-modules.md b/pages/merlin7/02-How-To-Use-Merlin/using-modules.md index f5368af..8f945c3 100644 --- a/pages/merlin7/02-How-To-Use-Merlin/using-modules.md +++ b/pages/merlin7/02-How-To-Use-Merlin/using-modules.md @@ -8,7 +8,13 @@ sidebar: merlin7_sidebar permalink: /merlin7/using-modules.html --- -## Environment Modules +## Cray Environment Modules + +By default, Merlin7 uses Cray Environment Modules. How to use it is explained in the [Cray Module Environment](/merlin7/cray-module-env.html) section. In addition, PSI Environment Modules, which contains a long list of software which is validated and can be used to run on Merlin7. + +For compiling, we strongly recommend using the software provided by the [Cray Module Environment](/merlin7/cray-module-env.html), which contains compilers, MPI and libraries optimized to run on Cray, and to use the PSI Environment Modules (a.k.a. PModules) for other software (ANSYS, Matlab, etc.). + +## PSI Environment Modules On top of the operating system stack we provide different software using the PSI developed PModule system. diff --git a/pages/merlin7/03-Slurm-General-Documentation/slurm-examples.md b/pages/merlin7/03-Slurm-General-Documentation/slurm-examples.md index 42bc92c..9fdbfd9 100644 --- a/pages/merlin7/03-Slurm-General-Documentation/slurm-examples.md +++ b/pages/merlin7/03-Slurm-General-Documentation/slurm-examples.md @@ -30,3 +30,46 @@ module purge module load $MODULE_NAME # where $MODULE_NAME is a software in PModules srun $MYEXEC # where $MYEXEC is a path to your binary file ``` + +## Multi-core based jobs example + +### Pure MPI + +```bash +#!/bin/bash +#SBATCH --job-name=purempi +#SBATCH --partition=daily # Using 'daily' will grant higher priority +#SBATCH --time=24:00:00 # Define max time job will run +#SBATCH --output=%x-%j.out # Define your output file +#SBATCH --error=%x-%j.err # Define your error file +#SBATCH --exclusive +#SBATCH --nodes=1 +#SBATCH --ntasks=128 +#SBATCH --hint=nomultithread +##SBATCH --cpus-per-task=1 + +module purge +module load $MODULE_NAME # where $MODULE_NAME is a software in PModules +srun $MYEXEC # where $MYEXEC is a path to your binary file +``` + +### Hybrid + +```bash +#!/bin/bash +#SBATCH --job-name=hybrid +#SBATCH --partition=daily # Using 'daily' will grant higher priority +#SBATCH --time=24:00:00 # Define max time job will run +#SBATCH --output=%x-%j.out # Define your output file +#SBATCH --error=%x-%j.err # Define your error file +#SBATCH --exclusive +#SBATCH --nodes=1 +#SBATCH --ntasks=128 +#SBATCH --hint=multithread +#SBATCH --cpus-per-task=2 + +module purge +module load $MODULE_NAME # where $MODULE_NAME is a software in PModules +srun $MYEXEC # where $MYEXEC is a path to your binary file +``` + diff --git a/pages/merlin7/slurm-configuration.md b/pages/merlin7/slurm-configuration.md index b742fe0..4efbeda 100644 --- a/pages/merlin7/slurm-configuration.md +++ b/pages/merlin7/slurm-configuration.md @@ -21,36 +21,15 @@ This documentation shows basic Slurm configuration and options needed to run job #### Hardware -The current configuration for the _test_ phase is made up as: +The current configuration for the _preproduction_ phase is made up as: -* 9 nodes for the _PSI-Dev_ development system - * 8 nodes were meant for baremetal and k8s - * 1 login node +* nodes for the _PSI-Dev_ development system + * 2 CPU-only login nodes + * 77 CPU-only compute nodes + * 4 GPU nodes | Node | CPU | RAM | GRES | Notes | | ---- | --- | --- | ---- | ----- | -| Compute node | _2x_ AMD EPYC 7713 (x86_64 Milan, 64 Cores, 3.2GHz) | 512GB DDR4 3200Mhz | _4x_ NVidia A100 (Ampere, 80GB) | | | Login node | _2x_ AMD EPYC 7742 (x86_64 Rome, 64 Cores, 3.2GHz) | 512GB DRR4 3200Mhz | | | - -#### Storage - -* CephFS only for `/home` -- 1 TB -* ClusterStor L300 for `/scratch` -- 224 TB usable space -* CephRBD `/local` -- 100GB - -#### Node IDs - -Cray user various identifies to uniquely label each node, details on this can be found on the [Crayism page](cray-conventions.html). -The table below collates these together for the current configuration: - -| Node ID | Cray XNAME | Notes | -| ---------- | ---------- | - | -| nid003204 | x1500c4s7b0n0 | login node, to which **psi-dev.cscs.ch** points | -| nid002808 | x1007c0s4b0n0 | | -| nid002809 | x1007c0s4b0n1 | | -| nid002812 | x1007c0s5b0n0 | | -| nid002813 | x1007c0s5b0n1 | | -| nid002824 | x1007c1s0b0n0 | | -| nid002825 | x1007c1s0b0n1 | | -| nid002828 | x1007c1s1b0n0 | | -| nid002829 | x1007c1s1b0n1 | | +| CPU node | _2x_ AMD EPYC 7742 (x86_64 Rome, 64 Cores, 3.2GHz) | 512GB DRR4 3200Mhz | | | +| GPU node | _2x_ AMD EPYC 7713 (x86_64 Milan, 64 Cores, 3.2GHz) | 512GB DDR4 3200Mhz | _4x_ NVidia A100 (Ampere, 80GB) | |