---
title: GOTHIC
#tags:
keywords: software, gothic, slurm, interactive, batch job
last_updated: 07 September 2022
summary: "This document describes how to run Gothic in the Merlin cluster"
sidebar: merlin6_sidebar
permalink: /merlin6/gothic.html
---

This document describes generic information of how to run Gothic in the
Merlin cluster

## Gothic installation

Gothic is locally installed in Merlin in the following directory:
```bash
/data/project/general/software/gothic
```

Multiple versions are available. As of August 22, 2022, the latest 
installed version is **Gothic 8.3 QA**.

Future releases will be placed in the PSI Modules system, therefore,
loading it through PModules will be possible at some point. However, in the
meantime one has to use the existing installations present in 
`/data/project/general/software/gothic`.

## Running Gothic

### General requirements

When running Gothic in interactive or batch mode, one has to consider
the following requirements:

* **Use always one node only**: Gothic runs a single instance. 
Therefore, it can not run on multiple nodes. Adding option `--nodes=1-1` 
or `-N 1-1` is strongly recommended: this will prevent Slurm to allocate
multiple nodes if the Slurm allocation definition is ambiguous.
* **Use one task only**: Gothic spawns one main process, which then will 
spawn multiple threads depending on the number of available cores. 
Therefore, one has to specify 1 task (`--ntasks=1` or `-n 1`).
* **Use multiple CPUs**: since Gothic will spawn multiple threads, then
multiple CPUs can be used. Adding `--cpus-per-task=<num_cpus>` 
or `-c <num_cpus>` is in general recommended. 
Notice that `<num_cpus>` must never exceed the maximum number of CPUS 
in a compute node (usually *88*).
* **Use multithread**: Gothic is an OpenMP based software, therefore, 
running in hyper-threading mode is strongly recommended. Use the option
`--hint=multithread` for enforcing hyper-threading.
* **[Optional]** *Memory setup*: The default memory per CPU (4000MB) 
is usually enough for running Gothic. If you require more memory, you 
can always set the `--mem=<mem_in_MB>` option. This is in general 
*not necessary*.

### Interactive

**Is not allowed to run CPU intensive interactive jobs in the
login nodes**. Only applications capable to limit the number of cores are
allowed to run for longer time. Also, **running in the login nodes is not 
efficient**, since resources are shared with other processes and users.

Is possible to submit interactive jobs to the cluster by allocating a
full compute node, or even by allocating a few cores only. This will grant
dedicated CPUs and resources and in general it will not affect other users.

For interactive jobs, is strongly recommended to use the `hourly` partition, 
which usually has a good availability of nodes. 

For longer runs, one should use the `daily` (or `general`) partition. 
However, getting interactive access to nodes on these partitions is 
sometimes more difficult if the cluster is pretty full.

To submit an interactive job, consider the following requirements:
* **X11 forwarding must be enabled**: Gothic spawns an interactive 
window which requires X11 forwarding when using it remotely, therefore 
using the Slurm option `--x11` is necessary.
* **Ensure that the scratch area is accessible**: For running Gothic,
one has to define a scratch area with the `GTHTMP` environment variable.
There are two options:
   1. **Use local scratch**: Each compute node has its own `/scratch` area.
This area is independent to any other node, therefore not visible by other nodes.
Using the top directory `/scratch` for interactive jobs is the simplest way,
and it can be defined before or after the allocation creation, as follows:
      ```bash
      # Example 1: Define GTHTMP before the allocation
      export GTHTMP=/scratch
      salloc ...
      
      # Example 2: Define GTHTMP after the allocation
      salloc ...
      export GTHTMP=/scratch
      ```
   Notice that if you want to create a custom sub-directory (i.e. 
`/scratch/$USER`, one has to create the sub-directory on every new 
allocation! In example:
      ```bash
      # Example 1: 
      export GTHTMP=/scratch/$USER
      salloc ...
      mkdir -p $GTHTMP

      # Example 2:
      salloc ...
      export GTHTMP=/scratch/$USER
      mkdir -p $GTHTMP
      ``` 
   Creating sub-directories makes the process more complex, therefore 
using just `/scratch` is simpler and recommended.
   2. **Shared scratch**: Using shared scratch allows to have a 
directory visible from all compute nodes and login nodes. Therefore,
one can use `/shared-scratch` to achieve the same as in **1.**, but 
creating a sub-directory needs to be done just once.

   Please, consider that `/scratch` usually provides better performance and,
in addition, will offload the main storage. Therefore, using **local scratch** 
is strongly recommended. Use the shared scratch only when strongly necessary. 
* **Use the `hourly` partition**: Using the `hourly` partition is 
recommended for running interactive jobs (latency is in general 
lower). However, `daily` and `general` are also available if you expect 
longer runs, but in these cases you should expect longer waiting times.

These requirements are in addition to the requirements previously described
in the [General requirements](/merlin6/gothic.html#general-requirements) 
section.

#### Interactive allocations: examples
* Requesting a full node, 
```bash
salloc --partition=hourly  -N 1 -n 1 -c 88 --hint=multithread --x11 --exclusive --mem=0 
```
* Requesting 22 CPUs from a node, with default memory per CPU (4000MB/CPU):
```bash
num_cpus=22
salloc --partition=hourly -N 1 -n 1 -c $num_cpus --hint=multithread --x11
```

### Batch job

The Slurm cluster is mainly used by non interactive batch jobs: Users
submit a job, which goes into a queue, and waits until Slurm can assign 
resources to it. In general, the longer the job, the longer the waiting time,
unless there are enough free resources to inmediately start running it.

Running Gothic in a Slurm batch script is pretty simple. One has to mainly 
consider the requirements described in the [General requirements](/merlin6/gothic.html#general-requirements)
section, and:
* **Use local scratch** for running batch jobs. In general, defining 
`GTHTMP` in a batch script is simpler than on an allocation. If you plan
to run multiple jobs in the same node, you can even create a second sub-directory
level based on the Slurm Job ID:
   ```bash
   mkdir -p /scratch/$USER/$SLURM_JOB_ID
   export GTHTMP=/scratch/$USER/$SLURM_JOB_ID
   ... # Run Gothic here
   rm -rf /scratch/$USER/$SLURM_JOB_ID
   ```
   Temporary data generated by the job in `GTHTMP` must be removed at the end of
the job, as showed above.

#### Batch script: examples

* Requesting a full node:
   ```bash
   #!/bin/bash -l
   #SBATCH --job-name=Gothic
   #SBATCH --time=3-00:00:00
   #SBATCH --partition=general
   #SBATCH --nodes=1
   #SBATCH --ntasks=1
   #SBATCH --cpus-per-task=88
   #SBATCH --hint=multithread
   #SBATCH --exclusive
   #SBATCH --mem=0
   #SBATCH --clusters=merlin6
   
   INPUT_FILE='MY_INPUT.SIN'
   
   mkdir -p /scratch/$USER/$SLURM_JOB_ID
   export GTHTMP=/scratch/$USER/$SLURM_JOB_ID
   
   /data/project/general/software/gothic/gothic8.3qa/bin/gothic_s.sh $INPUT_FILE -m -np $SLURM_CPUS_PER_TASK
   gth_exit_code=$?

   # Clean up data in /scratch   
   rm -rf /scratch/$USER/$SLURM_JOB_ID

   # Return exit code from GOTHIC
   exit $gth_exit_code
   ```
* Requesting 22 CPUs from a node, with default memory per CPU (4000MB/CPU):
   ```bash
   #!/bin/bash -l
   #SBATCH --job-name=Gothic
   #SBATCH --time=3-00:00:00
   #SBATCH --partition=general
   #SBATCH --nodes=1
   #SBATCH --ntasks=1
   #SBATCH --cpus-per-task=22
   #SBATCH --hint=multithread
   #SBATCH --clusters=merlin6
   
   INPUT_FILE='MY_INPUT.SIN'
   
   mkdir -p /scratch/$USER/$SLURM_JOB_ID
   export GTHTMP=/scratch/$USER/$SLURM_JOB_ID
   
   /data/project/general/software/gothic/gothic8.3qa/bin/gothic_s.sh $INPUT_FILE -m -np $SLURM_CPUS_PER_TASK
   gth_exit_code=$?

   # Clean up data in /scratch   
   rm -rf /scratch/$USER/$SLURM_JOB_ID

   # Return exit code from GOTHIC
   exit $gth_exit_code
   ```
