HPCE/gitea-pages
0
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Activity

Doc changes

Browse Source
This commit is contained in:
caubet_m
2021-05-21 12:34:19 +02:00
parent 42d8f38934
commit fcfdbf1344
46 changed files with 447 additions and 528 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
pages/merlin6/04-Jupyterhub/jupyter-examples.md Normal file
View File

@ -0,0 +1,29 @@
---
title: Jupyter examples on merlin6
#tags:
#keywords:
last_updated: 1 October 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/jupyter-examples.html
---
These examples demonstrate the use of certain python libraries and modules in the merlin6 environment. They are provided to get you started fast. You can check out a repository of the examples from
<https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples>
A number of standard data sets for the tutorials of the libraries are hosted on merlin6 centrally under `/data/project/general/public`, so you do not need to store them in your user space.
# Dask
[Dask](https://dask.org/) is a flexible library for parallel computing in Python. It provides the abstraction of a dask dataframe that can reside on multiple machines and can be manipulated by an API designed to be as close as possible to [pandas](https://pandas.pydata.org/). The example shows how to start up dask workers on merlin6 through slurm.
* [Link to example](https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples/blob/master/dask-example.ipynb)
* The data sets for the [dask tutorial](https://github.com/dask/dask-tutorial) are hosted at `/data/project/general/public/dask-tutorial`.
# Plotly
[Plotly](https://plot.ly/python/getting-started/) is an interactive open source plotting library
* [Link to example](https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples/blob/master/plotly-example.ipynb)

32
pages/merlin6/04-Jupyterhub/jupyter-extensions.md Normal file
View File

@ -0,0 +1,32 @@
---
title: Jupyter Extensions
#tags:
#keywords:
last_updated: 30 September 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/jupyter-extensions.html
---
## Using nbextensions for adding features to your notebook
There exist a number of useful [contributed but unofficial
extensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/index.html)
that add useful features to your notebooks.
From the classic Notebook UI you can access the available extensions in a separate tab as displayed in the screenshot, below. You may have to unselect the *disable configuration for nbextensions without explicit copatibility*. The extensions we tested still worked fine with this jupyterhub version of 1.0.0.
{% include image.html file="jupyter-nbextensions.png" caption="Launch Classic Notebook" max-width=586 %}
## Extensions for working with large notebooks
Especially the following extensions make working with larger notebooks easier
* **Table of Contents**: Displays a TOC on the left and you can also configure it
to add and update a TOC at the head of the document.
* **Collapsible Headings**: allows you to fold all the cells below a heading
It may also be interesting for you to explore the [Jupytext](jupytext.html) server extension.
## Variable Inspector
The `variable inspector` extension provides a constantly updated window in which you can see the value and type of your notebook's variables.

63
pages/merlin6/04-Jupyterhub/jupyterhub-trouble.md Normal file
View File

@ -0,0 +1,63 @@
---
title: Jupyterhub Troubleshooting
#tags:
#keywords:
last_updated: 18 February 2020
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/jupyterhub-trouble.html
---
In case of problems or requests, please either submit a **[PSI Service Now](https://psi.service-now.com/psisp)** incident containing *"Merlin Jupyterhub"* as part of the subject, or contact us by mail through <merlin-admins@lists.psi.ch>.
## General steps for troubleshooting
### Investigate the Slurm output file
Your jupyterhub session runs as a normal batch job on the cluster, and each launch will create a slurm output file in your HOME directory named like `jupyterhub_batchspawner_{$JOBID}.log`, where the $JOBID part is the slurm job ID of your job. After a failed launch, investigate the contents of that file. An error message will usually be found towards the end of the file, often including a python backtrace.
### Investigate python environment interferences
Jupyterhub just runs a jupyter notebook executable as your user inside the batch job. A frequent source of errors consists of a user's local python environment definitions getting mixed up with the environment that jupyter needs to launch.
- setting PYTHONPATH inside of the ~/.bash_profile or any other startup script
- having installed packages to your local user area (e.g. using `pip install --user <some-package>`). Such installation will interfere with the environment offered by the `module` system on our cluster (based on anaconda). You can list such packages by executing
`pip list user`. They are usually located in `~/.local/lib/pythonX.Y/...`.
You can investigate the launching of a notebook interactively, by logging in to Merlin6 and running a jupyter command in the correct environment.
```
module use unstable
module load anaconda/2019.07
conda activate jupyterhub-1.0.0_py36
jupyter --paths
```
## Known Problems and workarounds
### Spawner times out
If the cluster is very full, it may be difficult to launch a session. We always reserve some slots for interactive Jupyterhub use, but it may be that these slots have been taken or that the resources you requested are currently not available.
Inside of a Merlin6 terminal shell, you can run the standard commands like `sinfo` and `squeue` to get an overview of how full the cluster is.
### Your user environment is not among the kernels offered for choice
Refer to our documentation about [using your own custom made
environments with jupyterhub](/merlin6/jupyterhub.html).
### Cannot save notebook - _xsrf argument missing_
You cannot save your notebook anymore and you get this error:
**'_xsrf' argument missing from POST**
This issue occurs very seldomly. There exists the following workaround:
Go to the jupyterhub file browsing window and just open another
notebook using the same kernel in another browser window. The issue
should then go away. For more information refer to [this github
thread](https://github.com/nteract/hydrogen/issues/922#issuecomment-405456346)
<!-- ## Error HTTP 500 when starting the spawner -->
<!-- The spawner screen shows after launching an error message like the following: -->
<!-- `Internal server error (Spawner failed to start [status=..]. The logs may contain details)` -->

83
pages/merlin6/04-Jupyterhub/jupyterhub.md Normal file
View File

@ -0,0 +1,83 @@
---
title: Jupyterhub on Merlin
#tags:
#keywords:
last_updated: 31 July 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/jupyterhub.html
---
Jupyterhub provides [jupyter notebooks](https://jupyter.org/) that are launched on
cluster nodes of merlin and can be accessed through a web portal.
## Accessing Jupyterhub and launching a session
The service is available inside of PSI (or through a VPN connection) at
**<https://merlin-jupyter.psi.ch:8000>**
1. **Login**: You will be presented with a **Login** web page for
authenticating with your PSI account.
1. **Spawn job**: The **Spawner Options** page allows you to
specify the properties (Slurm partition, running time,...) of
the batch jobs that will be running your jupyter notebook. Once
you click on the `Spawn` button, your job will be sent to the
Slurm batch system. If the cluster is not currently overloaded
and the resources you requested are available, your job will
usually start within 30 seconds.
## Jupyter software environments - running different kernels
Your notebooks can run within different software environments which are offered by a number of available **Jupyter kernels**.
E.g. in this test installation we provide two environments targeted at data science
* **tensorflow-1.13.1_py37**: contains Tensorflow, Keras, scikit-learn, Pandas, numpy, dask, and dependencies. Stable
* **talos_py36**: also contains the Talos package. This
environment is experimental and subject to updates and changes.
When you create a new notebook you will be asked to specify which kernel you want to use. It is also possible to switch the kernel of a running notebook, but you will lose the state of the current kernel, so you will have to recalculate the notebook cells with this new kernel.
These environments are also available for standard work in a shell session. You can activate an environment in a normal merlin terminal session by using the `module` (q.v. [using Pmodules](using-modules.html)) command to load anaconda python, and from there using the `conda` command to switch to the desired environment
```
module use unstable
module load anaconda/2019.07
conda activate tensorflow-1.13.1_py36
```
When the `anaconda` module has been loaded, you can list the available environments by executing
```
conda info -e
```
You can get more info on the use of the `conda` package management tool at its official [https://conda.io/projects/conda/en/latest/commands.html](documentation site).
## Using your own custom made environments with jupyterhub
Python environments can take up a lot of space due to the many dependencies that will be installed. You should always install your extra environments to the data area belonging to your account, e.g. `/data/user/${YOUR-USERNAME}/conda-envs`
In order for jupyterhub (and jupyter in general) to recognize the provided environment as a valid kernel, make sure that you include the `nb_conda_kernels` package in your environment. This package provides the necessary activation and the dependencies.
Example:
```
conda create -c conda-forge -p /data/user/${USER}/conda-envs/my-test-env python=3.7 nb_conda_kernels
```
After this, your new kernel will be visible as `my-test-env` inside of your jupyterhub session.
## Contact
In case of problems or requests, please either submit a **[PSI Service Now](https://psi.service-now.com/psisp)** incident containing *"Merlin Jupyterhub"* as part of the subject, or contact us by mail through <merlin-admins@lists.psi.ch>.

28
pages/merlin6/04-Jupyterhub/jupyterlab.md Normal file
View File

@ -0,0 +1,28 @@
---
title: Jupyterlab User interface
#tags:
#keywords:
last_updated: 31 July 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/jupyterlab.html
---
## Testing out Jupyterlab
**Jupyterlab** is a new interface to interact with your Jupyter notebooks. However, it is in very active development and undergoing constant changes. You can read about its features [on the official website](https://jupyterlab.readthedocs.io/en/stable/user/interface.html).
You can test it out on our server by using the following kind of URL, where *$YOUR-USER* must be replaced by your PSI username. You must already have an active session on the jupyterhub.
https://merlin-jupyter.psi.ch:8000/user/$YOUR-USER/lab
## Switching to the Classic Notebook user interface
You can switch to the classical notebook UI by using the **"Launch Classic Notebook"** command from the left sidebar of JupyterLab.
{% include image.html file="jupyter-launch-classic.png" caption="Launch Classic Notebook" max-width=501 %}
## Jupyterlab does not support the older nbextensions
These regrettably are not yet supported from within the JupyterLab UI,
but you can activate them through the Classic Notebook interface (see
above)

37
pages/merlin6/04-Jupyterhub/jupytext.md Normal file
View File

@ -0,0 +1,37 @@
---
title: Jupytext - efficient editing
#tags:
#keywords:
last_updated: 30 September 2019
#summary: ""
sidebar: merlin6_sidebar
permalink: /merlin6/jupytext.html
---
[Jupytext](https://github.com/mwouts/jupytext) is a Jupyter serverextension that allows creating a text file from a notebook that can be kept in sync with it **with the aim of using more efficient editors or IDEs on it**. The file can be created in a number of formats, e.g. *markdown, *.py (light Script)*, and others. `Jupytext` will keep the both the notebook and this **paired** file in sync: If you save the paired file, changes will be carried over into the notebook, and vice versa. This pairing will persist also in new sessions of your notebook until you explicitely remove it again.
The paired file contains only the cell contents and not the output. Therefore it also is **much better suited for revision control**, since the differences between versions are limited to the cells and these file formats yield more meaningful text differences than the default notebook storage format.
## Creating a paired file in python format for efficient refactoring
From your notebook, go to the `file` menu and navigate to the `jupytext` submenu. Select the **light script** pairing option. This will create a `*.py` file version with the same basename as your notebook file.
{% include image.html file="jupytext_menu.png" caption="Jupytext menu" max-width=501 %}
You can edit that file separately in your favourite python editor. The markdown text parts will be conserved in the file in the form of python comments.
When you save the file and do a browser page reload of your jupyter notebook, you will see all the changes carried over into your jupyter notebook.
## Creating a paired file in mardown format for efficient text authoring
If you want to efficiently work on the descriptive text base of your notebook, just pair it using the `Pair notebook with Markdown` menu item and edit the generated `*.md` file with your favourite Markdown editor.
## Disable autosaving when working on the paired file
Your notebooks usually auto save every 2 min (default). Turn this feature off when working with the paired file. Otherwise Jupyter will continue to save the state while you are editing the paired file, and the changes will be synced to the disk version of the paired file. You can disable the autosave by unchecking the `Autosave notebook` menu item in the Juytext menu (see above image).
## Further information
Please refer to
* [the Jupytext FAQ](https://jupytext.readthedocs.io/en/latest/faq.html)
* [the Jupytext documentation](https://jupytext.readthedocs.io/en/latest/index.html)