HPCE/gitea-pages
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Activity

Add GPU example

Browse Source
This commit is contained in:
Spencer Bliven
2022-06-21 11:42:53 +02:00
parent 300b1950a4
commit 9356e3554c
1 changed files with 51 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
pages/merlin6/03-Slurm-General-Documentation/slurm-examples.md
View File

@ -32,7 +32,7 @@ srun $MYEXEC # where $MYEXEC is a path to your binary file
### Example 2: Non-hyperthreaded job
In this example we do not want hyper-threading (``--ntasks-per-core=1`` and ``--hint=nomultithread``). In our Merlin6 configuration,
the default memory per cpu (a CPU is equivalent to a core thread) is 4000MB. If we do not specify anything else, our
the default memory per cpu (a CPU is equivalent to a core thread) is 4000MB. If we do not specify anything else, our
single core task will use a default of 4000MB. However, one could double it with ``--mem-per-cpu=8000`` if you require more memory
(remember, the second thread will not be used so we can safely assign +4000MB to the unique active thread).
@ -56,7 +56,7 @@ srun $MYEXEC # where $MYEXEC is a path to your binary file
In this example we run a job that will run 88 tasks. Merlin6 Apollo nodes have 44 cores each one with hyper-threading
enabled. This means that we can run 2 threads per core, in total 88 threads. To accomplish that, users should specify
``--ntasks-per-core=2`` and ``--hint=multithread``.
``--ntasks-per-core=2`` and ``--hint=multithread``.
Use `--nodes=1` if you want to use a node exclusively (88 hyperthreaded tasks would fit in a Merlin6 node).
@ -77,10 +77,10 @@ srun $MYEXEC # where $MYEXEC is a path to your binary file
### Example 2: MPI without Hyper-Threading
In this example, we want to run a job that will run 44 tasks, and due to performance reasons we want to disable hyper-threading.
Merlin6 Apollo nodes have 44 cores, each one with hyper-threading enabled. For ensuring that only 1 thread will be used per task,
In this example, we want to run a job that will run 44 tasks, and due to performance reasons we want to disable hyper-threading.
Merlin6 Apollo nodes have 44 cores, each one with hyper-threading enabled. For ensuring that only 1 thread will be used per task,
users should specify ``--ntasks-per-core=1`` and ``--hint=nomultithread``. With this configuration, we tell Slurm to run only 1
tasks per core and no hyperthreading should be used. Hence, each tasks will be assigned to an independent core.
tasks per core and no hyperthreading should be used. Hence, each tasks will be assigned to an independent core.
Use `--nodes=1` if you want to use a node exclusively (44 non-hyperthreaded tasks would fit in a Merlin6 node).
@ -90,7 +90,7 @@ Use `--nodes=1` if you want to use a node exclusively (44 non-hyperthreaded task
#SBATCH --ntasks=44 # Job will run 44 tasks
#SBATCH --ntasks-per-core=1 # Request the max ntasks be invoked on each core
#SBATCH --hint=nomultithread # Don't use extra threads with in-core multi-threading
#SBATCH --time=00:30:00 # Define max time job will run
#SBATCH --time=00:30:00 # Define max time job will run
#SBATCH --output=myscript.out # Define your output file
#SBATCH --error=myscript.err # Define your output file
@ -101,8 +101,8 @@ srun $MYEXEC # where $MYEXEC is a path to your binary file
### Example 3: Hyperthreaded Hybrid MPI/OpenMP job
In this example, we want to run a Hybrid Job using MPI and OpenMP using hyperthreading. In this job, we want to run 4 MPI
tasks by using 8 CPUs per task. Each task in our example requires 128GB of memory. Then we specify 16000MB per CPU
In this example, we want to run a Hybrid Job using MPI and OpenMP using hyperthreading. In this job, we want to run 4 MPI
tasks by using 8 CPUs per task. Each task in our example requires 128GB of memory. Then we specify 16000MB per CPU
(8 x 16000MB = 128000MB). Notice that since hyperthreading is enabled, Slurm will use 4 cores per task (with hyperthreading
2 threads -a.k.a. Slurm CPUs- fit into a core).
@ -130,24 +130,24 @@ srun $MYEXEC # where $MYEXEC is a path to your binary file
### Example 4: Non-hyperthreaded Hybrid MPI/OpenMP job
In this example, we want to run a Hybrid Job using MPI and OpenMP without hyperthreading. In this job, we want to run 4 MPI
tasks by using 8 CPUs per task. Each task in our example requires 128GB of memory. Then we specify 16000MB per CPU
In this example, we want to run a Hybrid Job using MPI and OpenMP without hyperthreading. In this job, we want to run 4 MPI
tasks by using 8 CPUs per task. Each task in our example requires 128GB of memory. Then we specify 16000MB per CPU
(8 x 16000MB = 128000MB). Notice that since hyperthreading is disabled, Slurm will use 8 cores per task (disabling hyperthreading
we force the use of only 1 thread -a.k.a. 1 CPU- per core).
```bash
#!/bin/bash -l
#SBATCH --clusters=merlin6
#!/bin/bash -l
#SBATCH --clusters=merlin6
#SBATCH --job-name=test
#SBATCH --ntasks=4
#SBATCH --ntasks-per-socket=1
#SBATCH --ntasks=4
#SBATCH --ntasks-per-socket=1
#SBATCH --mem-per-cpu=16000
#SBATCH --cpus-per-task=8
#SBATCH --cpus-per-task=8
#SBATCH --partition=hourly
#SBATCH --time=01:00:00
#SBATCH --output=srun_%j.out
#SBATCH --error=srun_%j.err
#SBATCH --hint=nomultithread
#SBATCH --time=01:00:00
#SBATCH --output=srun_%j.out
#SBATCH --error=srun_%j.err
#SBATCH --hint=nomultithread
module purge
module load $MODULE_NAME # where $MODULE_NAME is a software in PModules
@ -157,6 +157,31 @@ srun $MYEXEC # where $MYEXEC is a path to your binary file
{{site.data.alerts.tip}} Also, always consider that **`'--mem-per-cpu' x '--cpus-per-task'`** can **never** exceed the maximum amount of memory per node (352000MB).
{{site.data.alerts.end}}
## GPU examples
Using GPUs requires two major changes. First, the cluster needs to be specified
to `gmerlin6`. This should also be added to later commands pertaining to the
job, e.g. `scancel --cluster=gmerlin6 <jobid>`. Second, the number of GPUs
should be specified using `--gpus`, `--gpus-per-task`, or similar parameters.
Here's an example for a simple test job:
```bash
#!/bin/bash
#SBATCH --partition=gpu # Or 'gpu-short' for higher priority but 2-hour limit
#SBATCH --cluster=gmerlin6 # Required for GPU
#SBATCH --gpus=2 # Total number of GPUs
#SBATCH --cpus-per-gpu=5 # Request CPU resources
#SBATCH --time=1-00:00:00 # Define max time job will run
#SBATCH --output=myscript.out # Define your output file
#SBATCH --error=myscript.err # Define your error file
module purge
module load cuda # load any needed modules here
srun $MYEXEC # where $MYEXEC is a path to your binary file
```
Slurm will automatically set the gpu visibility (eg `$CUDA_VISIBLE_DEVICES`).
## Advanced examples
### Array Jobs: launching a large number of related jobs
@ -190,7 +215,7 @@ have their own output file.
* Do not use such jobs if you have very short tasks, since each array sub job will incur the full overhead for launching an independent Slurm job. For such cases you should used a **packed job** (see below).
* If you want to control how many of these jobs can run in parallel, you can use the `#SBATCH --array=1-100%5` syntax. The `%5` will define
that only 5 sub jobs may ever run in parallel.
You also can use an array job approach to run over all files in a directory, substituting the payload with
``` bash
@ -220,7 +245,7 @@ strategy:
#SBATCH --time=7-00:00:00 # each job can run for 7 days
#SBATCH --cpus-per-task=1
#SBATCH --array=1-10%1 # Run a 10-job array, one job at a time.
if test -e checkpointfile; then
if test -e checkpointfile; then
# There is a checkpoint file;
$MYEXEC --read-checkp checkpointfile
else
@ -250,7 +275,7 @@ arguments passed from 1 to 1000. But with the =-N1 -n1 -c1
instances are effectively running, each being allocated one CPU. You
can at this point decide to allocate several CPUs or tasks by adapting
the corresponding parameters.
``` bash
#! /bin/bash
#SBATCH --job-name=test-checkpoint
@ -289,11 +314,11 @@ echo "Example MPI:" ; srun hostname # will print one hostname per ntask
```
In the above example are specified the options ``--nodes=2`` and ``--ntasks=44``. This means that up 2 nodes are requested,
and is expected to run 44 tasks. Hence, 44 cores are needed for running that job. Slurm will try to allocate a maximum of
2 nodes, both together having at least 44 cores. Since our nodes have 44 cores / each, if nodes are empty (no other users
and is expected to run 44 tasks. Hence, 44 cores are needed for running that job. Slurm will try to allocate a maximum of
2 nodes, both together having at least 44 cores. Since our nodes have 44 cores / each, if nodes are empty (no other users
have running jobs there), job can land on a single node (it has enough cores to run 44 tasks).
If we want to ensure that job is using at least two different nodes (i.e. for boosting CPU frequency, or because the job
If we want to ensure that job is using at least two different nodes (i.e. for boosting CPU frequency, or because the job
requires more memory per core) you should specify other options.
A good example is ``--ntasks-per-node=22``. This will equally distribute 22 tasks on 2 nodes.
@ -304,7 +329,7 @@ A good example is ``--ntasks-per-node=22``. This will equally distribute 22 task
A different example could be by specifying how much memory per core is needed. For instance ``--mem-per-cpu=32000`` will reserve
~32000MB per core. Since we have a maximum of 352000MB per Apollo node, Slurm will be only able to allocate 11 cores (32000MB x 11cores = 352000MB) per node.
It means that 4 nodes will be needed (max 11 tasks per node due to memory definition, and we need to run 44 tasks), in this case we need to change ``--nodes=4``
It means that 4 nodes will be needed (max 11 tasks per node due to memory definition, and we need to run 44 tasks), in this case we need to change ``--nodes=4``
(or remove ``--nodes``). Alternatively, we can decrease ``--mem-per-cpu`` to a lower value which can allow the use of at least 44 cores per node (i.e. with ``16000``
should be able to use 2 nodes)