Document common statuses
This commit is contained in:
Spencer Bliven
@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Running Jobs
|
||||
title: Running Jobs
|
||||
#tags:
|
||||
#keywords:
|
||||
last_updated: 18 June 2019
|
||||
@ -12,7 +12,7 @@ permalink: /merlin6/running-jobs.html
|
||||
|
||||
* ``sbatch``: to submit a batch script to Slurm. Use ``squeue`` for checking jobs status and ``scancel`` for deleting a job from the queue.
|
||||
* ``srun``: to run parallel jobs in the batch system
|
||||
* ``salloc``: to obtain a Slurm job allocation (a set of nodes), execute command(s), and then release the allocation when the command is finished.
|
||||
* ``salloc``: to obtain a Slurm job allocation (a set of nodes), execute command(s), and then release the allocation when the command is finished.
|
||||
This is equivalent to interactive run.
|
||||
|
||||
## Running on Merlin5
|
||||
@ -24,7 +24,7 @@ but they will need to specify a couple of extra options to their scripts.
|
||||
#SBATCH --clusters=merlin5
|
||||
```
|
||||
|
||||
By adding ``--clusters=merlin5`` it will send the jobs to the old Merlin5 computing nodes. Also, ``--partition=<merlin|gpu>`` can be specified in
|
||||
By adding ``--clusters=merlin5`` it will send the jobs to the old Merlin5 computing nodes. Also, ``--partition=<merlin|gpu>`` can be specified in
|
||||
order to use the old Merlin5 partitions.
|
||||
|
||||
## Running on Merlin6
|
||||
@ -35,12 +35,12 @@ In order to run on the **Merlin6** cluster, users have to add the following opti
|
||||
#SBATCH --clusters=merlin6
|
||||
```
|
||||
|
||||
By adding ``--clusters=merlin6`` it will send the jobs to the old Merlin6 computing nodes.
|
||||
By adding ``--clusters=merlin6`` it will send the jobs to the old Merlin6 computing nodes.
|
||||
|
||||
## Shared nodes and exclusivity
|
||||
|
||||
The **Merlin6** cluster has been designed in a way that should allow running MPI/OpenMP processes as well as single core based jobs. For allowing
|
||||
co-existence, nodes are configured by default in a shared mode. It means, that multiple jobs from multiple users may land in the same node. This
|
||||
co-existence, nodes are configured by default in a shared mode. It means, that multiple jobs from multiple users may land in the same node. This
|
||||
behaviour can be changed by a user if they require exclusive usage of nodes.
|
||||
|
||||
By default, Slurm will try to allocate jobs on nodes that are already occupied by processes not requiring exclusive usage of a node. In this way,
|
||||
@ -57,8 +57,8 @@ Exclusivity of a node can be setup by specific the ``--exclusive`` option as fol
|
||||
By default, Slurm script will generate standard output and errors files in the directory from where
|
||||
you submit the batch script:
|
||||
|
||||
* standard output will be written into a file ``slurm-$SLURM_JOB_ID.out``.
|
||||
* standard error will be written into a file ``slurm-$SLURM_JOB_ID.err``.
|
||||
* standard output will be written into a file ``slurm-$SLURM_JOB_ID.out``.
|
||||
* standard error will be written into a file ``slurm-$SLURM_JOB_ID.err``.
|
||||
|
||||
If you want to the default names it can be done with the options ``--output`` and ``--error``. In example:
|
||||
|
||||
@ -72,8 +72,8 @@ Use **man sbatch** (``man sbatch | grep -A36 '^filename pattern'``) for getting
|
||||
## Partitions
|
||||
|
||||
Merlin6 contains 6 partitions for general purpose:
|
||||
|
||||
* For the CPU these are ``general``, ``daily`` and ``hourly``.
|
||||
|
||||
* For the CPU these are ``general``, ``daily`` and ``hourly``.
|
||||
* For the GPU these are ``gpu``.
|
||||
|
||||
If no partition is defined, ``general`` will be the default. Partition can be defined with the ``--partition`` option as follows:
|
||||
@ -82,7 +82,7 @@ If no partition is defined, ``general`` will be the default. Partition can be de
|
||||
#SBATCH --partition=<partition_name> # Partition to use. 'general' is the 'default'.
|
||||
```
|
||||
|
||||
Please check the section [Slurm Configuration#Merlin6 Slurm Partitions] for more information about Merlin6 partition setup.
|
||||
Please check the section [Slurm Configuration#Merlin6 Slurm Partitions] for more information about Merlin6 partition setup.
|
||||
|
||||
## CPU-based Jobs Settings
|
||||
|
||||
@ -115,7 +115,7 @@ The following template should be used by any user submitting jobs to CPU nodes:
|
||||
|
||||
## Advanced options example
|
||||
##SBATCH --nodes=1 # Uncomment and specify #nodes to use
|
||||
##SBATCH --ntasks=44 # Uncomment and specify #nodes to use
|
||||
##SBATCH --ntasks=44 # Uncomment and specify #nodes to use
|
||||
##SBATCH --ntasks-per-node=44 # Uncomment and specify #tasks per node
|
||||
##SBATCH --ntasks-per-core=2 # Uncomment and specify #tasks per core (a.k.a. threads)
|
||||
##SBATCH --cpus-per-task=44 # Uncomment and specify the number of cores per task
|
||||
@ -139,8 +139,8 @@ The following options are mandatory settings that **must be included** in your b
|
||||
|
||||
### Slurm GPU Recommended Settings
|
||||
|
||||
GPUs are also a shared resource. Hence, multiple users can run jobs on a single node, but only one GPU per user process
|
||||
must be used. Users can define which GPUs resources they need with the ``--gres`` option.
|
||||
GPUs are also a shared resource. Hence, multiple users can run jobs on a single node, but only one GPU per user process
|
||||
must be used. Users can define which GPUs resources they need with the ``--gres`` option.
|
||||
Valid ``gres`` options are: ``gpu[[:type]:count]`` where ``type=GTX1080|GTX1080Ti`` and ``count=<number of gpus to use>``
|
||||
This would be according to the following rules:
|
||||
|
||||
@ -170,3 +170,30 @@ The following template should be used by any user submitting jobs to GPU nodes:
|
||||
##SBATCH --ntasks-per-node=44 # Uncomment and specify number of tasks per node
|
||||
##SBATCH --cpus-per-task=44 # Uncomment and specify the number of cores per task
|
||||
```
|
||||
|
||||
|
||||
## Job status
|
||||
|
||||
The status of submitted jobs can be check with the `squeue` command:
|
||||
|
||||
```
|
||||
~ $ squeue -u bliven_s
|
||||
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
|
||||
134507729 gpu test_scr bliven_s PD 0:00 3 (AssocGrpNodeLimit)
|
||||
134507768 general test_scr bliven_s PD 0:00 19 (AssocGrpCpuLimit)
|
||||
134507729 gpu test_scr bliven_s PD 0:00 3 (Resources)
|
||||
134506301 gpu test_scr bliven_s PD 0:00 1 (Priority)
|
||||
134506288 gpu test_scr bliven_s R 9:16 1 merlin-g-008
|
||||
```
|
||||
|
||||
Common Statuses:
|
||||
- *merlin-\** Running on the specified host
|
||||
- *(Priority)* Waiting in the queue
|
||||
- *(Resources)* At the head of the queue, waiting for machines to become available
|
||||
- *(AssocGrpCpuLimit), (AssocGrpNodeLimit)* Job would exceed per-user limitations on
|
||||
the number of simultaneous CPUs/Nodes. Use `scancel` to remove the job and
|
||||
resubmit with fewer resources, or else wait for your other jobs to finish.
|
||||
- *(PartitionNodeLimit)* Exceeds all resources available on this partition.
|
||||
Run `scancel` and resubmit to a different partition (`-p`) or with fewer
|
||||
resources.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user