--- title: GOTHIC #tags: keywords: software, gothic, slurm, interactive, batch job last_updated: 07 September 2022 summary: "This document describes how to run Gothic in the Merlin cluster" sidebar: merlin6_sidebar permalink: /merlin6/gothic.html --- This document describes generic information of how to run Gothic in the Merlin cluster ## Gothic installation Gothic is locally installed in Merlin in the following directory: ```bash /data/project/general/software/gothic ``` Multiple versions are available. As of August 22, 2022, the latest installed version is **Gothic 8.3 QA**. Future releases will be placed in the PSI Modules system, therefore, loading it through PModules will be possible at some point. However, in the meantime one has to use the existing installations present in `/data/project/general/software/gothic`. ## Running Gothic ### General requirements When running Gothic in interactive or batch mode, one has to consider the following requirements: * **Use always one node only**: Gothic runs a single instance. Therefore, it can not run on multiple nodes. Adding option `--nodes=1-1` or `-N 1-1` is strongly recommended: this will prevent Slurm to allocate multiple nodes if the Slurm allocation definition is ambiguous. * **Use one task only**: Gothic spawns one main process, which then will spawn multiple threads depending on the number of available cores. Therefore, one has to specify 1 task (`--ntasks=1` or `-n 1`). * **Use multiple CPUs**: since Gothic will spawn multiple threads, then multiple CPUs can be used. Adding `--cpus-per-task=` or `-c ` is in general recommended. Notice that `` must never exceed the maximum number of CPUS in a compute node (usually *88*). * **Use multithread**: Gothic is an OpenMP based software, therefore, running in hyper-threading mode is strongly recommended. Use the option `--hint=multithread` for enforcing hyper-threading. * **[Optional]** *Memory setup*: The default memory per CPU (4000MB) is usually enough for running Gothic. If you require more memory, you can always set the `--mem=` option. This is in general *not necessary*. ### Interactive **Is not allowed to run CPU intensive interactive jobs in the login nodes**. Only applications capable to limit the number of cores are allowed to run for longer time. Also, **running in the login nodes is not efficient**, since resources are shared with other processes and users. Is possible to submit interactive jobs to the cluster by allocating a full compute node, or even by allocating a few cores only. This will grant dedicated CPUs and resources and in general it will not affect other users. For interactive jobs, is strongly recommended to use the `hourly` partition, which usually has a good availability of nodes. For longer runs, one should use the `daily` (or `general`) partition. However, getting interactive access to nodes on these partitions is sometimes more difficult if the cluster is pretty full. To submit an interactive job, consider the following requirements: * **X11 forwarding must be enabled**: Gothic spawns an interactive window which requires X11 forwarding when using it remotely, therefore using the Slurm option `--x11` is necessary. * **Ensure that the scratch area is accessible**: For running Gothic, one has to define a scratch area with the `GTHTMP` environment variable. There are two options: 1. **Use local scratch**: Each compute node has its own `/scratch` area. This area is independent to any other node, therefore not visible by other nodes. Using the top directory `/scratch` for interactive jobs is the simplest way, and it can be defined before or after the allocation creation, as follows: ```bash # Example 1: Define GTHTMP before the allocation export GTHTMP=/scratch salloc ... # Example 2: Define GTHTMP after the allocation salloc ... export GTHTMP=/scratch ``` Notice that if you want to create a custom sub-directory (i.e. `/scratch/$USER`, one has to create the sub-directory on every new allocation! In example: ```bash # Example 1: export GTHTMP=/scratch/$USER salloc ... mkdir -p $GTHTMP # Example 2: salloc ... export GTHTMP=/scratch/$USER mkdir -p $GTHTMP ``` Creating sub-directories makes the process more complex, therefore using just `/scratch` is simpler and recommended. 2. **Shared scratch**: Using shared scratch allows to have a directory visible from all compute nodes and login nodes. Therefore, one can use `/shared-scratch` to achieve the same as in **1.**, but creating a sub-directory needs to be done just once. Please, consider that `/scratch` usually provides better performance and, in addition, will offload the main storage. Therefore, using **local scratch** is strongly recommended. Use the shared scratch only when strongly necessary. * **Use the `hourly` partition**: Using the `hourly` partition is recommended for running interactive jobs (latency is in general lower). However, `daily` and `general` are also available if you expect longer runs, but in these cases you should expect longer waiting times. These requirements are in addition to the requirements previously described in the [General requirements](/merlin6/gothic.html#general-requirements) section. #### Interactive allocations: examples * Requesting a full node, ```bash salloc --partition=hourly -N 1 -n 1 -c 88 --hint=multithread --x11 --exclusive --mem=0 ``` * Requesting 22 CPUs from a node, with default memory per CPU (4000MB/CPU): ```bash num_cpus=22 salloc --partition=hourly -N 1 -n 1 -c $num_cpus --hint=multithread --x11 ``` ### Batch job The Slurm cluster is mainly used by non interactive batch jobs: Users submit a job, which goes into a queue, and waits until Slurm can assign resources to it. In general, the longer the job, the longer the waiting time, unless there are enough free resources to inmediately start running it. Running Gothic in a Slurm batch script is pretty simple. One has to mainly consider the requirements described in the [General requirements](/merlin6/gothic.html#general-requirements) section, and: * **Use local scratch** for running batch jobs. In general, defining `GTHTMP` in a batch script is simpler than on an allocation. If you plan to run multiple jobs in the same node, you can even create a second sub-directory level based on the Slurm Job ID: ```bash mkdir -p /scratch/$USER/$SLURM_JOB_ID export GTHTMP=/scratch/$USER/$SLURM_JOB_ID ... # Run Gothic here rm -rf /scratch/$USER/$SLURM_JOB_ID ``` Temporary data generated by the job in `GTHTMP` must be removed at the end of the job, as showed above. #### Batch script: examples * Requesting a full node: ```bash #!/bin/bash -l #SBATCH --job-name=Gothic #SBATCH --time=3-00:00:00 #SBATCH --partition=general #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=88 #SBATCH --hint=multithread #SBATCH --exclusive #SBATCH --mem=0 #SBATCH --clusters=merlin6 INPUT_FILE='MY_INPUT.SIN' mkdir -p /scratch/$USER/$SLURM_JOB_ID export GTHTMP=/scratch/$USER/$SLURM_JOB_ID /data/project/general/software/gothic/gothic8.3qa/bin/gothic_s.sh $INPUT_FILE -m -np $SLURM_CPUS_PER_TASK gth_exit_code=$? # Clean up data in /scratch rm -rf /scratch/$USER/$SLURM_JOB_ID # Return exit code from GOTHIC exit $gth_exit_code ``` * Requesting 22 CPUs from a node, with default memory per CPU (4000MB/CPU): ```bash #!/bin/bash -l #SBATCH --job-name=Gothic #SBATCH --time=3-00:00:00 #SBATCH --partition=general #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=22 #SBATCH --hint=multithread #SBATCH --clusters=merlin6 INPUT_FILE='MY_INPUT.SIN' mkdir -p /scratch/$USER/$SLURM_JOB_ID export GTHTMP=/scratch/$USER/$SLURM_JOB_ID /data/project/general/software/gothic/gothic8.3qa/bin/gothic_s.sh $INPUT_FILE -m -np $SLURM_CPUS_PER_TASK gth_exit_code=$? # Clean up data in /scratch rm -rf /scratch/$USER/$SLURM_JOB_ID # Return exit code from GOTHIC exit $gth_exit_code ```