Hans-Nikolai Viessmann
bde174b726
refactor CSCS and Meg content add merlin6 quick start update merlin6 nomachine docs give the userdoc its own color scheme we use the Materials default one refactored slurm general docs merlin6 add merlin6 JB docs add software support m6 docs add all files to nav vibed changes #1 add missing pages further vibing #2 vibe #3 further fixes
160 lines
4.9 KiB
Markdown
160 lines
4.9 KiB
Markdown
# Python
|
|
|
|
PSI provides a variety of ways to execute python code.
|
|
|
|
1. **Anaconda** - Custom environments for using installation and development
|
|
1. **Jupyterhub** - Execute Jupyter notebooks on the cluster
|
|
1. **System Python** - Do not use! Only for OS applications.
|
|
|
|
## Anaconda
|
|
|
|
[Anaconda](https://www.anaconda.com/) ("conda" for short) is a package manager
|
|
with excellent python integration. Using it you can create isolated
|
|
environments for each of your python applications, containing exactly the
|
|
dependencies needed for that app. It is similar to the
|
|
[virtualenv](http://virtualenv.readthedocs.org/) python package, but can also
|
|
manage non-python requirements.
|
|
|
|
### Loading conda
|
|
|
|
Conda is loaded from the module system:
|
|
|
|
```bash
|
|
module load anaconda
|
|
```
|
|
|
|
### Using pre-made environments
|
|
|
|
Loading the module provides the `conda` command, but does not otherwise change your
|
|
environment. First an environment needs to be activated. Available environments can
|
|
be seen with `conda info --envs` and include many specialized environments for
|
|
software installs. After activating you should see the environment name in your
|
|
prompt:
|
|
|
|
```bash
|
|
conda activate datascience_py37
|
|
```
|
|
|
|
### CondaRC file
|
|
|
|
Creating a `~/.condarc` file is recommended if you want to create new environments on
|
|
merlin. Environments can grow quite large, so you will need to change the default
|
|
storage location from the default (your home directory) to a larger volume (usually
|
|
`/data/user/$USER`).
|
|
|
|
Save the following as `$HOME/.condarc`:
|
|
|
|
```yaml
|
|
always_copy: true
|
|
|
|
envs_dirs:
|
|
- /data/user/$USER/conda/envs
|
|
|
|
pkgs_dirs:
|
|
- /data/user/$USER/conda/pkgs
|
|
- $ANACONDA_PREFIX/conda/pkgs
|
|
|
|
channels:
|
|
- conda-forge
|
|
- nodefaults
|
|
```
|
|
|
|
Run `conda info` to check that the variables are being set correctly.
|
|
|
|
### Creating environments
|
|
|
|
We will create an environment named `myenv` which uses an older version of
|
|
numpy, e.g. to test for backwards compatibility of our code (the `-q` and
|
|
`--yes` switches are just for not getting prompted and disabling the progress
|
|
bar). The environment will be created in the default location as defined by the
|
|
`.condarc` configuration file (see above).
|
|
|
|
```console
|
|
$ conda create -q --yes -n 'myenv1' numpy=1.8 scipy ipython
|
|
|
|
Fetching package metadata: ...
|
|
Solving package specifications: .
|
|
Package plan for installation in environment /gpfs/home/feichtinger/conda-envs/myenv1:
|
|
|
|
The following NEW packages will be INSTALLED:
|
|
|
|
ipython: 2.3.0-py27_0
|
|
numpy: 1.8.2-py27_0
|
|
openssl: 1.0.1h-1
|
|
pip: 1.5.6-py27_0
|
|
python: 2.7.8-1
|
|
readline: 6.2-2
|
|
scipy: 0.14.0-np18py27_0
|
|
setuptools: 5.8-py27_0
|
|
sqlite: 3.8.4.1-0
|
|
system: 5.8-1
|
|
tk: 8.5.15-0
|
|
zlib: 1.2.7-0
|
|
|
|
To activate this environment, use:
|
|
$ source activate myenv1
|
|
|
|
To deactivate this environment, use:
|
|
$ source deactivate
|
|
```
|
|
|
|
The created environment contains **just the packages that are needed to satisfy the
|
|
requirements** and it is local to your installation. The python installation is even
|
|
independent of the central installation, i.e. your code will still work in such an
|
|
environment, even if you are offline or AFS is down. However, you need the central
|
|
installation if you want to use the `conda` command itself.
|
|
|
|
Packages for your new environment will be either copied from the central one into
|
|
your new environment, or if there are newer packages available from anaconda and you
|
|
did not specify exactly the version from our central installation, they may get
|
|
downloaded from the web. **This will require significant space in the `envs_dirs`
|
|
that you defined in `.condarc`**. If you create other environments on the same local
|
|
disk, they will share the packages using hard links.
|
|
|
|
We can switch to the newly created environment with the `conda activate` command.
|
|
|
|
```bash
|
|
conda activate myenv1
|
|
```
|
|
|
|
!!! info
|
|
Note that anaconda's activate/deactivate scripts are compatible with the
|
|
bash and zsh shells but not with [t]csh.
|
|
|
|
Let's test whether we indeed got the desired numpy version:
|
|
|
|
```console
|
|
$ python -c 'import numpy as np; print np.version.version'
|
|
|
|
1.8.2
|
|
```
|
|
|
|
You can install additional packages into the active environment using the `conda
|
|
install` command.
|
|
|
|
```console
|
|
$ conda install --yes -q bottle
|
|
|
|
Fetching package metadata: ...
|
|
Solving package specifications: .
|
|
Package plan for installation in environment /gpfs/home/feichtinger/conda-envs/myenv1:
|
|
|
|
The following NEW packages will be INSTALLED:
|
|
|
|
bottle: 0.12.5-py27_0
|
|
```
|
|
|
|
## Jupyterhub
|
|
|
|
Jupyterhub is a service for running code notebooks on the cluster, particularly
|
|
in python. It is a powerful tool for data analysis and prototyping. For more
|
|
infomation see the [Jupyterhub documentation](../jupyterhub/jupyterhub.md).
|
|
|
|
## Pythons to avoid
|
|
|
|
Avoid using the system python (`/usr/bin/python`). It is intended for OS
|
|
software and may not be up to date.
|
|
|
|
Also avoid the 'python' module (`module load python`). This is a minimal
|
|
install of python intended for embedding in other modules.
|