From 69f182627b2b19a42d21b2af9aa435dde10e1f40 Mon Sep 17 00:00:00 2001
From: caubet_m <marc.caubet@psi.ch>
Date: Wed, 15 Jul 2020 10:53:58 +0200
Subject: [PATCH] Details UCX examples

---
 pages/merlin6/05 Software Support/openmpi.md | 36 +++++++++++++++-----
 1 file changed, 27 insertions(+), 9 deletions(-)

diff --git a/pages/merlin6/05 Software Support/openmpi.md b/pages/merlin6/05 Software Support/openmpi.md
index 53d35a1..4647f2f 100644
--- a/pages/merlin6/05 Software Support/openmpi.md	
+++ b/pages/merlin6/05 Software Support/openmpi.md	
@@ -34,23 +34,41 @@ srun ./app
 **UCX** should be used only by advanced users, as it requires to run it with **mpirun** (needs advanced knowledge) and is an exception for running MPI 
 without **srun** (**UCX** is not integrated at PSI within **srun**).
 
-For running UCX, one should add the following options to **mpirun**:
+For running UCX, one should:
 
-```bash
--mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1
-```
+* add the following options to **mpirun**:
+   ```bash
+   -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1
+   ```
+* or alternatively, add the following options **before mpirun**
+   ```bash
+   export OMPI_MCA_pml="ucx"
+   export OMPI_MCA_btl="^vader,tcp,openib,uct"
+   export UCX_NET_DEVICES=mlx5_0:1
+   ```
 
-Alternatively, one can add the following options for debugging purposes (visit [UCX Logging](https://github.com/openucx/ucx/wiki/Logging) for possible `UCX_LOG_LEVEL` values):
+In addition, one can add the following options for debugging purposes (visit [UCX Logging](https://github.com/openucx/ucx/wiki/Logging) for possible `UCX_LOG_LEVEL` values):
 
 ```bash
 -x UCX_LOG_LEVEL=<data|debug|warn|info|...> -x UCX_LOG_FILE=<filename>
 ```
 
-Full example:
+This can be also added externally before the **mpirun** call (see below example). Full example:
 
-```bash
-mpirun -np $SLURM_NTASKS -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_LOG_LEVEL=data -x UCX_LOG_FILE=UCX-$SLURM_JOB_ID.loga ./app
-```
+* Within the **mpirun** command:
+   ```bash
+   mpirun -np $SLURM_NTASKS -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_LOG_LEVEL=data -x UCX_LOG_FILE=UCX-$SLURM_JOB_ID.log ./app
+   ```
+* Outside the **mpirun** command:
+  ```bash
+  export OMPI_MCA_pml="ucx"
+  export OMPI_MCA_btl="^vader,tcp,openib,uct"
+  export UCX_NET_DEVICES=mlx5_0:1
+  export UCX_LOG_LEVEL=data
+  export UCX_LOG_FILE=UCX-$SLURM_JOB_ID.log
+
+  mpirun -no $SLURM_NTASKS ./app
+  ```
 
 ## Supported OpenMPI versions