13
Conda
assman_g edited this page 2024-06-11 10:38:31 +02:00

Anaconda (in short: conda) is a package and environment managing tool, mostly used for python. There is two major options how to use conda: Use the centralized conda installation or install conda by yourself:

centralized conda: module load anaconda

One option is to use the centralized conda module anaconda/2019.07 which is installed as a module on merlin. As soon as a user loads the module, conda will be available. There is no need to initialize it! To try do:

module load anaconda
which conda
conda --version 

This should show a path to the conda instance and the version that is currently installed (4.10.3). Remark: This conda instance is rather old and will be updated soon, might cause some problems for some difficult dependencies.

If you want to create a conda env with this conda instance, it is crucial to generate/edit your ~/.condarc file, as the default will put both your environments and packages into your home (only 4GB!)

Generate/Edit your file in ~/.condarc with you favorite editor as follows:

always_copy: true

envs_dirs:
  - /data/user/$USER/conda/envs

pkgs_dirs:
  - /data/user/$USER/conda/pkgs
  - $ANACONDA_PREFIX/conda/pkgs

channels:
  - conda-forge
  - defaults

This will write the envs and packages to /data/user/$USER/conda/... (max 1TB !). Always verify this with the command conda info displaying your current configuration! Double-check if something is not according to your expectations!

Remark: If you want to use this conda instance to activate one of your environments in a SLURM script (sbatch myscript.sh), you need to load the module anaconda inside the script. In the following simple example, the script only activates a personal environment and prints out the conda info and which conda command.

#!/bin/bash
 
 #SBATCH --cluster=merlin6
 #SBATCH --partition=hourly
 #SBATCH --nodes=1
 #SBATCH --job-name="test"
 
 module load anaconda
 conda activate /data/user/assman_g/conda/envs/test_env
 conda info
 which conda

personal conda: miniconda installation

If you do not want to use the central anaconda module, you can also install your own, personal conda instance. The most common solution for that is miniconda. Again, you want to install your miniconda and the associated environments + packages in /data/user/$USER in order to prevent filling up your 4Gb home directory.

miniconda installation

Further miniconda info: installation

Follow the steps below to install miniconda. Follow the process and check for error messages indicating that something went wrong with the installation!

cd /data/user/$USER
mkdir miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda3/miniconda.sh
bash miniconda/miniconda.sh -b -u -p miniconda3

# In order to add your conda installation to your ~/.bashrc (= initializing), do: 
conda init bash

To verify your installation, check again with the following commands: conda info , which conda, and open your ~/.bashrc to check if a new miniconda section appears.

REMARK: It is important to undestand/keep in mind, that other conda instances (such as loading the anaconda module (see above) might cause conflicts ! Always verify which conda instance is currently active with which conda and conda info

conda important commands

This quick steps will only be a very basic usage of conda, always have a look into the official documentation

generate a conda env

To generate a simple conda environment with a specific python version, do:

conda create -n myenv python=3.9

remove a conda env

To remove a conda env do:

conda remove --name myenv --all

general commands

conda env list             : list all environments
conda info                 : show the current config
conda activate myenv       : activate the env myenv
conda deactivate           : deactivate the active env

Some Remarks:

  • the currently active env will be shown in front of your username in the prompt
  • to install packages, always activate the target env
  • if you want to provide your env to other users, best is to generate an environment.yml file, which can be used for conda env creation.
  • do not install packages into your "base" env, this is created by default upon a miniconda installation and should stay clean of packages, as this can cause problems.