264 lines
11 KiB
Markdown
264 lines
11 KiB
Markdown
---
|
|
layout: default
|
|
title: Using Merlin6
|
|
parent: Merlin6 User Guide
|
|
nav_order: 3
|
|
---
|
|
|
|
# Using Merlin6
|
|
{: .no_toc }
|
|
|
|
## Table of contents
|
|
{: .no_toc .text-delta }
|
|
|
|
1. TOC
|
|
{:toc}
|
|
|
|
---
|
|
|
|
## Important: Code of Conduct
|
|
|
|
The basic principle is courtesy and consideration for other users.
|
|
|
|
* Merlin6 is a shared resource, not your laptop, therefore you are kindly requested to behave in a way you would be happy to see other users behaving towards you.
|
|
* Basic shell programming skills in Linux/UNIX environment is a must-have requirement for HPC users; a proficiency in shell programming would be greatly beneficial.
|
|
* The login nodes are for development and quick testing:
|
|
* Is **strictly forbidden to run production jobs** on the login nodes.
|
|
* Is **forbidden to run long processes** occupying big part of the resources.
|
|
* *Any miss-behaving running processes according to these rules will be killed.*
|
|
* All production jobs should be submitted using the batch system.
|
|
* Make sure that no broken or run-away processes are left when your job is done. Keep the process space clean on all nodes.
|
|
* Remove files you do not need any more (e.g. core dumps, temporary files) as early as possible. Keep the disk space clean on all nodes.
|
|
|
|
The system administrator has the right to block the access to Merlin6 for an account violating the Code of Conduct, in which case the issue will be escalated to the user's supervisor.
|
|
|
|
---
|
|
|
|
## Merlin6 Access
|
|
|
|
### HowTo: Request Access to Merlin6
|
|
|
|
* PSI users with their Linux accounts belonging to the *svc-cluster_merlin6* group are allowed to use Merlin6.
|
|
|
|
* Registration for Merlin6 access must be done through [PSI Service Now](https://psi.service-now.com/psisp)
|
|
* Please open it as an Incident request, with subject: ``[Merlin6] Access Request for user '<username>'``
|
|
|
|
|
|
### HowTo: Access to Merlin6
|
|
|
|
Use SSH to access the login nodes:
|
|
* <tt><b>merlin-l-01.psi.ch</b></tt> ("merlin" '-' 'el' '-' 'zero' 'one')
|
|
* <tt><b>merlin-l-02.psi.ch</b></tt> ("merlin" '-' 'el' '-' 'zero' 'two')
|
|
|
|
Examples:
|
|
<pre>
|
|
ssh -Y merlin-l-01
|
|
ssh -Y bond_j@merlin-l-02.psi.ch
|
|
</pre>
|
|
|
|
<!--
|
|
### Home and Data Directories
|
|
|
|
Default quota for the home directory */gpfs/home/$USER* is 10GB.
|
|
Until a service for an automatic backup of the home directories is announced
|
|
to be in production, the users are responsible for managing the backups
|
|
of their home directories.
|
|
|
|
The data directories */gpfs/data/$USER* have much larger quotas per user (default is 1TB, extendible on request) then the home directories,
|
|
but there will be no automatic backup of the data directories.
|
|
The users are fully responsible for backup and restore operations
|
|
in the data directories.
|
|
|
|
Command to see your quota on merlin5:
|
|
<pre>
|
|
/usr/lpp/mmfs/bin/mmlsquota -u $USER --block-size auto merlin5
|
|
</pre>
|
|
|
|
### Scratch disk and Temporary Files
|
|
|
|
A */scratch* partition of ~50GB is available on each computing node. This partition should be the one used by the users for creating temporary files and/or
|
|
directories that are needed by running jobs. Temporary files *must be deleted at the end of the job*.
|
|
|
|
Example of how to use the */scratch* disk:
|
|
|
|
<pre>
|
|
#!/bin/bash
|
|
#SBATCH --partition=merlin # name of slurm partition to submit
|
|
#SBATCH --time=2:00:00 # limit the execution of this job to 2 hours, see sinfo for the max. allowance
|
|
#SBATCH --nodes=4 # you request a 4 nodes
|
|
|
|
<b># Create scratch directory</b>
|
|
<i>SCRATCHDIR="/scratch/$(id -un)/${SLURM_JOB_ID}"
|
|
mkdir -p ${SCRATCHDIR}</i>
|
|
...
|
|
<b># Core code, generating temporary files in $SCRATCHDIR</b>
|
|
...
|
|
<b># Copy final results (whenever needed)</b>
|
|
<i>mkdir /gpfs/home/$(id -un)/${SLURM_JOB_ID}</i>
|
|
<i>cp -pr /scratch/$(id -un)/${SLURM_JOB_ID}/my_results /gpfs/home/$(id -un)/${SLURM_JOB_ID}</i>
|
|
<b># Cleanup temporary data and directories</b>
|
|
<i>rm -rf /scratch/$(id -un)/${SLURM_JOB_ID}</i>
|
|
<i>rmdir /scratch/$(id -un)</i>
|
|
</pre>
|
|
|
|
### Using Batch System to Submit Jobs to Merlin5
|
|
|
|
The Slurm Workload Manager is used on Merlin5 to manage and schedule jobs.
|
|
Please see "man slurm" and references therein for more details.
|
|
There are many tutorials and howtos on Slurm elsewhere, e.g. at CSCS.
|
|
We shall provide some typical examples for submitting different types of jobs.
|
|
|
|
Useful commands for the slurm:
|
|
<pre>
|
|
sinfo # to see the name of nodes, their occupancy, name of slurm partitions, limits (try out with "-l" option)
|
|
squeue # to see the currently running/waiting jobs in slurm (additional "-l" option may also be useful)
|
|
sbatch Script.sh # to submit a script (example below) to the slurm
|
|
scancel job_id # to cancel slurm job, job id is the numeric id, seen by the squeue
|
|
</pre>
|
|
|
|
Other advanced commands:
|
|
<pre>
|
|
sinfo -N -l # list nodes, state, resources (number of CPUs, memory per node, etc.), and other information
|
|
sshare -a # to list shares of associations to a cluster
|
|
sprio -l # to view the factors that comprise a job's scheduling priority (add -u <username> for filtering user)
|
|
</pre>
|
|
|
|
###+ Simple slurm test script (copy-paste the following example in file Script.sh):
|
|
<pre>
|
|
#!/bin/bash
|
|
#SBATCH --partition=merlin # name of slurm partition to submit
|
|
#SBATCH --time=2:00:00 # limit the execution of this job to 2 hours, see sinfo for the max. allowance
|
|
#SBATCH --nodes=4 # you request a 4 nodes
|
|
|
|
hostname # will print one name, since executed on one node
|
|
echo
|
|
module load gcc/6.2.0 openmpi/1.10.2 hdf5/1.8.17
|
|
mpirun hostname # will be executed on all 4 nodes (see above --nodes)
|
|
echo
|
|
sleep 60 # useless work occupying 4 merlin nodes
|
|
module list
|
|
</pre>
|
|
|
|
Submit job to slurm and check it's status:
|
|
<pre>
|
|
sbatch Script.sh # submit this job to slurm
|
|
squeue # check it's status
|
|
</pre>
|
|
|
|
###+ Advanced slurm test script (copy-paste the following example in file Script.sh):
|
|
<pre>
|
|
#!/bin/bash
|
|
#SBATCH --partition=merlin # name of slurm partition to submit
|
|
#SBATCH --time=2:00:00 # limit the execution of this job to 2 hours, see sinfo for the max. allowance
|
|
#SBATCH --nodes=2 # number of nodes
|
|
#SBATCH --ntasks=24 # number of tasks
|
|
|
|
hostname # will print one name, since executed on one node
|
|
echo
|
|
module load gcc/6.2.0 openmpi/1.10.2 hdf5/1.8.17
|
|
mpirun hostname # will be executed on all 4 nodes (see above --nodes)
|
|
echo
|
|
sleep 60 # useless work occupying 4 merlin nodes
|
|
module list
|
|
</pre>
|
|
|
|
In the above example are specified the options *--nodes=2* and *--ntasks=24*. This means that 2 nodes are requested,
|
|
and is expected to run 24 tasks. Hence, 24 cores are needed for running that job. Slurm will try to allocate 2 nodes
|
|
with similar resources, having at least 12 cores/node.
|
|
|
|
Usually, 2 nodes with 12 cores/node would fit in the allocation decision. However, other combinations may be possible
|
|
(i.e, 2 nodes with 16 cores/node). In this second case, could happen that other users are running jobs in the allocated
|
|
nodes (in this example, up to 4 cores per node could be used by other user jobs, having at least 12 cores per node
|
|
available, which are the minimum number of tasks/cores required by our job).
|
|
|
|
In order to ensure exclusivity of the node, an option *--exclusive* can be used (see below). This will ensure that
|
|
the requested nodes are exclusive for the job (no other users jobs will interact with this node, and only completely
|
|
free nodes will be allocated).
|
|
|
|
<pre>
|
|
#SBATCH --exclusive
|
|
</pre>
|
|
|
|
More advanced configurations can be defined and can be combined with the previous examples. More information about advanced
|
|
options can be found in the following link: https://slurm.schedmd.com/sbatch.html (or run 'man sbatch').
|
|
|
|
If you have questions about how to properly execute your jobs, please contact us through merlin-admins@lists.psi.ch. Do not run
|
|
advanced configurations unless your are sure of what you are doing.
|
|
|
|
### Environment Modules
|
|
|
|
On top of the operating system stack we provide different software using the PSI developed
|
|
pmodule system. Useful commands:
|
|
<pre>
|
|
module avail # to see the list of available software provided via pmodules
|
|
module load gnuplot/5.2.0 # to load specific version of gnuplot package
|
|
module search hdf # try it out to see which version of hdf5 package is provided and with which dependencies
|
|
module load gcc/6.2.0 openmpi/1.10.2 hdf5/1.8.17 # load the specific version of hdf5, compiled with specific version of gcc and openmpi
|
|
module use unstable # to get access to the packages which are not considered to be fully stable by module provider (may be very fresh version, or not yet tested by community)
|
|
module list # to see which software is loaded in your environment
|
|
</pre>
|
|
|
|
###+ Requests for New Software
|
|
|
|
If you miss some package/version, contact us
|
|
|
|
### Known Problems and Troubleshooting
|
|
|
|
###+ Paraview, ANSYS and openGL
|
|
|
|
Try to use X11(mesa) driver for paraview and ANSYS instead of OpenGL:
|
|
<pre>
|
|
module load ANSYS
|
|
fluent -driver x11
|
|
</pre>
|
|
|
|
<pre>
|
|
module load paraview
|
|
paraview --mesa
|
|
</pre>
|
|
|
|
###+ Illegal instructions
|
|
|
|
It may happened that your code, compiled on one machine will not be executed on another throwing exception like "(Illegal instruction)".
|
|
Check (with "hostname" command) on which of the node you are and compare it with the names from first item. We observe few applications
|
|
that can't be run on merlin-c-01..16 because of this problem (notice that these machines are more then 5 years old). Hint: you may
|
|
choose the particular flavour of the machines for your slurm job, check the "--cores-per-node" option for sbatch:
|
|
<pre>
|
|
sbatch --cores-per-socket=8 Script.sh # will filter the selection of the machine and exclude the oldest one, merlin-c-01..16
|
|
</pre>
|
|
|
|
###+ Troubleshooting SSH
|
|
|
|
Use the ssh command with the "-vvv" option and copy and paste (no screenshot please)
|
|
the output to your request in Service-Now. Example
|
|
|
|
<pre>
|
|
ssh -Y -vvv bond_j@merlin-l-01
|
|
</pre>
|
|
|
|
###+ Troubleshooting SLURM
|
|
|
|
If one copies Slurm commands or batch scripts from another cluster,
|
|
they may need some changes (often minor) to run successfully on Merlin5.
|
|
Examine carefully the error message, especially concerning the options
|
|
used in the slurm commands.
|
|
|
|
Try to submit jobs using the examples given in the section "Using Batch System to Submit Jobs to Merlin5".
|
|
If you can run successfully an example for a type of job (!OpenMP, MPI) similar to your one,
|
|
try to edit the example to run your application.
|
|
|
|
If the problem remains, then, in your request in Service-Now, describe the problem in details that
|
|
are needed to reproduce it. Include the output of the following commands:
|
|
|
|
<pre>
|
|
date
|
|
hostname
|
|
pwd
|
|
module list
|
|
# All slurm commands used with the corresponding output
|
|
</pre>
|
|
|
|
Do not delete any output and error files generated by Slurm.
|
|
Make a copy of the failed job script if you like to edit it meanwhile.
|
|
-->
|