HPCE/gitea-pages
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Activity

Details UCX examples

Browse Source
This commit is contained in:
caubet_m
2020-07-15 10:53:58 +02:00
parent 0f0ba76057
commit 69f182627b
1 changed files with 27 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
pages/merlin6/05 Software Support/openmpi.md
View File

@ -34,23 +34,41 @@ srun ./app
**UCX** should be used only by advanced users, as it requires to run it with **mpirun** (needs advanced knowledge) and is an exception for running MPI
without **srun** (**UCX** is not integrated at PSI within **srun**).
For running UCX, one should add the following options to **mpirun**:
For running UCX, one should:
```bash
-mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1
```
* add the following options to **mpirun**:
```bash
-mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1
```
* or alternatively, add the following options **before mpirun**
```bash
export OMPI_MCA_pml="ucx"
export OMPI_MCA_btl="^vader,tcp,openib,uct"
export UCX_NET_DEVICES=mlx5_0:1
```
Alternatively, one can add the following options for debugging purposes (visit [UCX Logging](https://github.com/openucx/ucx/wiki/Logging) for possible `UCX_LOG_LEVEL` values):
In addition, one can add the following options for debugging purposes (visit [UCX Logging](https://github.com/openucx/ucx/wiki/Logging) for possible `UCX_LOG_LEVEL` values):
```bash
-x UCX_LOG_LEVEL=<data|debug|warn|info|...> -x UCX_LOG_FILE=<filename>
```
Full example:
This can be also added externally before the **mpirun** call (see below example). Full example:
```bash
mpirun -np $SLURM_NTASKS -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_LOG_LEVEL=data -x UCX_LOG_FILE=UCX-$SLURM_JOB_ID.loga ./app
```
* Within the **mpirun** command:
```bash
mpirun -np $SLURM_NTASKS -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_LOG_LEVEL=data -x UCX_LOG_FILE=UCX-$SLURM_JOB_ID.log ./app
```
* Outside the **mpirun** command:
```bash
export OMPI_MCA_pml="ucx"
export OMPI_MCA_btl="^vader,tcp,openib,uct"
export UCX_NET_DEVICES=mlx5_0:1
export UCX_LOG_LEVEL=data
export UCX_LOG_FILE=UCX-$SLURM_JOB_ID.log
mpirun -no $SLURM_NTASKS ./app
```
## Supported OpenMPI versions