add merlin6 JB docs

docs/merlin6/jupyterhub/jupyter-examples.md

@@ -0,0 +1,20 @@
+# Jupyter examples on merlin6
+
+These examples demonstrate the use of certain python libraries and modules in the merlin6 environment. They are provided to get you started fast. You can check out a repository of the examples from
+
+<https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples>
+
+A number of standard data sets for the tutorials of the libraries are hosted on merlin6 centrally under `/data/project/general/public`, so you do not need to store them in your user space.
+
+## Dask
+
+  [Dask](https://dask.org/) is a flexible library for parallel computing in Python. It provides the abstraction of a dask dataframe that can reside on multiple machines and can be manipulated by an API designed to be as close as possible to [pandas](https://pandas.pydata.org/). The example shows how to start up dask workers on merlin6 through slurm.
+
+* [Link to example](https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples/blob/master/dask-example.ipynb)
+* The data sets for the [dask tutorial](https://github.com/dask/dask-tutorial) are hosted at `/data/project/general/public/dask-tutorial`.
+
+## Plotly
+
+[Plotly](https://plot.ly/python/getting-started/) is an interactive open source plotting library
+
+* [Link to example](https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples/blob/master/plotly-example.ipynb)

docs/merlin6/jupyterhub/jupyter-extensions.md

@@ -0,0 +1,30 @@
+# Jupyter Extensions
+
+## Using nbextensions for adding features to your notebook
+
+There exist a number of useful [contributed but unofficial
+extensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/index.html)
+that add useful features to your notebooks.
+
+From the classic Notebook UI you can access the available extensions in a
+separate tab as displayed in the screenshot, below. You may have to unselect
+the *disable configuration for nbextensions without explicit copatibility*. The
+extensions we tested still worked fine with this jupyterhub version of 1.0.0.
+
+![Launch Classic Notebook](../../images/jupyter/jupyter-launch-classic.png)
+/// caption
+///
+
+## Extensions for working with large notebooks
+
+Especially the following extensions make working with larger notebooks easier
+
+* **Table of Contents**: Displays a TOC on the left and you can also configure it
+  to add and update a TOC at the head of the document.
+* **Collapsible Headings**: allows you to fold all the cells below a heading
+
+It may also be interesting for you to explore the [Jupytext](jupytext.html) server extension.
+
+## Variable Inspector
+
+The `variable inspector` extension provides a constantly updated window in which you can see the value and type of your notebook's variables.

docs/merlin6/jupyterhub/jupyterhub-trouble.md

@@ -0,0 +1,78 @@
+# Jupyterhub Troubleshooting
+
+In case of problems or requests, please either submit a **[PSI Service
+Now](https://psi.service-now.com/psisp)** incident containing *"Merlin
+Jupyterhub"* as part of the subject, or contact us by mail through
+<merlin-admins@lists.psi.ch>.
+
+## General steps for troubleshooting
+
+### Investigate the Slurm output file
+
+Your jupyterhub session runs as a normal batch job on the cluster, and each
+launch will create a slurm output file in your *home* directory named like
+`jupyterhub_batchspawner_{$JOBID}.log`, where the `$JOBID` part is the slurm job
+ID of your job. After a failed launch, investigate the contents of that file.
+An error message will usually be found towards the end of the file, often
+including a python backtrace.
+
+### Investigate python environment interferences
+
+Jupyterhub just runs a jupyter notebook executable as your user inside the
+batch job. A frequent source of errors consists of a user's local python
+environment definitions getting mixed up with the environment that jupyter
+needs to launch.
+
+- setting PYTHONPATH inside of the ~/.bash_profile or any other startup script
+- having installed packages to your local user area (e.g. using `pip install
+  --user <some-package>`). Such installation will interfere with the
+  environment offered by the `module` system on our cluster (based on
+  anaconda). You can list such packages by executing `pip list user`.
+  They are usually located in `~/.local/lib/pythonX.Y/...`.
+
+You can investigate the launching of a notebook interactively, by logging in to
+Merlin6 and running a jupyter command in the correct environment.
+
+```bash
+module use unstable
+module load anaconda/2019.07
+conda activate jupyterhub-1.0.0_py36
+jupyter --paths
+```
+
+## Known Problems and workarounds
+
+### Spawner times out
+
+If the cluster is very full, it may be difficult to launch a session. We always
+reserve some slots for interactive Jupyterhub use, but it may be that these
+slots have been taken or that the resources you requested are currently not
+available.
+
+Inside of a Merlin6 terminal shell, you can run the standard commands like
+`sinfo` and `squeue` to get an overview of how full the cluster is.
+
+### Your user environment is not among the kernels offered for choice
+
+Refer to our documentation about [using your own custom made
+environments with jupyterhub](/merlin6/jupyterhub.html).
+
+### Cannot save notebook - *xsrf argument missing*
+
+You cannot save your notebook anymore and you get this error:
+
+```text
+'_xsrf' argument missing from POST
+```
+
+This issue occurs very seldomly. There exists the following workaround:
+
+Go to the jupyterhub file browsing window and just open another
+notebook using the same kernel in another browser window. The issue
+should then go away. For more information refer to [this github
+thread](https://github.com/nteract/hydrogen/issues/922#issuecomment-405456346)
+
+<!-- ## Error HTTP 500 when starting the spawner -->
+
+<!--    The spawner screen shows after launching an error message like the following: -->
+<!--    `Internal server error (Spawner failed to start [status=..]. The logs may contain details)` -->

docs/merlin6/jupyterhub/jupyterhub.md

@@ -0,0 +1,109 @@
+# Jupyterhub on Merlin
+
+Jupyterhub provides [jupyter notebooks](https://jupyter.org/) that are launched on
+cluster nodes of merlin and can be accessed through a web portal.
+
+## Accessing Jupyterhub and launching a session
+
+The service is available inside of PSI (or through a VPN connection) at
+
+**<https://merlin-jupyter.psi.ch:8000>**
+
+1. **Login**: You will be presented with a **Login** web page for
+  authenticating with your PSI account.
+1. **Spawn job**: The **Spawner Options** page allows you to
+  specify the properties (Slurm partition, running time,...) of
+  the batch jobs that will be running your jupyter notebook. Once
+  you click on the `Spawn` button, your job will be sent to the
+  Slurm batch system. If the cluster is not currently overloaded
+  and the resources you requested are available, your job will
+  usually start within 30 seconds.
+
+## Jupyter software environments - running different kernels
+
+Your notebooks can run within different software environments which are offered by a number of available **Jupyter kernels**.
+
+E.g. in this test installation we provide two environments targeted at data science
+
+* **tensorflow-1.13.1_py37**: contains Tensorflow, Keras, scikit-learn, Pandas, numpy, dask, and dependencies. Stable
+* **talos_py36**: also contains the Talos package. This
+  environment is experimental and subject to updates and changes.
+
+When you create a new notebook you will be asked to specify which kernel you
+want to use. It is also possible to switch the kernel of a running notebook,
+but you will lose the state of the current kernel, so you will have to
+recalculate the notebook cells with this new kernel.
+
+These environments are also available for standard work in a shell session. You
+can activate an environment in a normal merlin terminal session by using the
+`module` (q.v. [using Pmodules](using-modules.html)) command to load anaconda
+python, and from there using the `conda` command to switch to the desired
+environment:
+
+```bash
+module use unstable
+module load anaconda/2019.07
+conda activate tensorflow-1.13.1_py36
+```
+
+When the `anaconda` module has been loaded, you can list the available environments by executing:
+
+```bash
+conda info -e
+```
+
+You can get more info on the use of the `conda` package management tool at its official [https://conda.io/projects/conda/en/latest/commands.html](documentation site).
+
+## Using your own custom made environments with jupyterhub
+
+Python environments can take up a lot of space due to the many dependencies that will be installed. You should always install your extra environments to the data area belonging to your account, e.g. `/data/user/${YOUR-USERNAME}/conda-envs`
+
+In order for jupyterhub (and jupyter in general) to recognize the provided
+environment as a valid kernel, make sure that you include the
+`nb_conda_kernels` package in your environment. This package provides the
+necessary activation and the dependencies.
+
+Example:
+
+```bash
+conda create -c conda-forge -p /data/user/${USER}/conda-envs/my-test-env python=3.7 nb_conda_kernels
+```
+
+After this, your new kernel will be visible as `my-test-env` inside of your jupyterhub session.
+
+## Requesting additional resources
+
+The **Spawner Options** page covers the most common options. These are used to
+create a submission script for the jupyterhub job and submit it to the slurm
+queue. Additional customization can be implemented using the *'Optional user
+defined line to be added to the batch launcher script'* option. This line is
+added to the submission script at the end of other `#SBATCH` lines. Parameters can
+be passed to SLURM by starting the line with `#SBATCH`, like in [Running Slurm
+Scripts](/merlin6/running-jobs.html). Some ideas:
+
+### Request additional memory
+
+```bash
+#SBATCH --mem=100G
+```
+
+### Request multiple GPUs (gpu partition only)
+
+```bash
+#SBATCH --gpus=2
+```
+
+### Log additional information
+
+```bash
+hostname; date; echo $USER
+```
+
+Output is found in `~/jupyterhub_batchspawner_<jobid>.log`.
+
+## Contact
+
+In case of problems or requests, please either submit a **[PSI Service
+Now](https://psi.service-now.com/psisp)** incident containing *"Merlin
+Jupyterhub"* as part of the subject, or contact us by mail through
+<merlin-admins@lists.psi.ch>.

docs/merlin6/jupyterhub/jupyterlab.md

@@ -0,0 +1,30 @@
+# Jupyterlab User interface
+
+## Testing out Jupyterlab
+
+**Jupyterlab** is a new interface to interact with your Jupyter notebooks.
+However, it is in very active development and undergoing constant changes. You
+can read about its features [on the official
+website](https://jupyterlab.readthedocs.io/en/stable/user/interface.html).
+
+You can test it out on our server by using the following kind of URL, where
+`$YOUR-USER` must be replaced by your PSI username. You must already have an
+active session on the jupyterhub.
+
+```text
+https://merlin-jupyter.psi.ch:8000/user/$YOUR-USER/lab
+```
+
+## Switching to the Classic Notebook user interface
+
+You can switch to the classical notebook UI by using the **"Launch Classic Notebook"** command from the left sidebar of JupyterLab.
+
+![Launch Classic Notebook](../../images/jupyter/jupyter-launch-classic.png)
+/// caption
+///
+
+## Jupyterlab does not support the older nbextensions
+
+These regrettably are not yet supported from within the JupyterLab UI,
+but you can activate them through the Classic Notebook interface (see
+above)

docs/merlin6/jupyterhub/jupytext.md

@@ -0,0 +1,52 @@
+# Jupytext - efficient editing
+
+[Jupytext](https://github.com/mwouts/jupytext) is a Jupyter serverextension
+that allows creating a text file from a notebook that can be kept in sync with
+it **with the aim of using more efficient editors or IDEs on it**. The file can
+be created in a number of formats, e.g. *markdown, *.py (light Script)*, and
+others. `Jupytext` will keep the both the notebook and this **paired** file in
+sync: If you save the paired file, changes will be carried over into the
+notebook, and vice versa. This pairing will persist also in new sessions of
+your notebook until you explicitely remove it again.
+
+The paired file contains only the cell contents and not the output. Therefore
+it also is **much better suited for revision control**, since the differences
+between versions are limited to the cells and these file formats yield more
+meaningful text differences than the default notebook storage format.
+
+## Creating a paired file in python format for efficient refactoring
+
+From your notebook, go to the `file` menu and navigate to the `jupytext`
+submenu. Select the **light script** pairing option. This will create a `*.py`
+file version with the same basename as your notebook file.
+
+![Jupytext menu](../../images/jupyter/jupytext_menu.png)
+/// caption
+///
+
+You can edit that file separately in your favourite python editor. The markdown
+text parts will be conserved in the file in the form of python comments.
+
+When you save the file and do a browser page reload of your jupyter notebook,
+you will see all the changes carried over into your jupyter notebook.
+
+## Creating a paired file in mardown format for efficient text authoring
+
+If you want to efficiently work on the descriptive text base of your notebook,
+just pair it using the `Pair notebook with Markdown` menu item and edit the
+generated `*.md` file with your favourite Markdown editor.
+
+## Disable autosaving when working  on the paired file
+
+Your notebooks usually auto save every 2 min (default). Turn this feature off
+when working with the paired file. Otherwise Jupyter will continue to save the
+state while you are editing the paired file, and the changes will be synced to
+the disk version of the paired file. You can disable the autosave by unchecking
+the `Autosave notebook` menu item in the Juytext menu (see above image).
+
+## Further information
+
+Please refer to:
+
+* [the Jupytext FAQ](https://jupytext.readthedocs.io/en/latest/faq.html)
+* [the Jupytext documentation](https://jupytext.readthedocs.io/en/latest/index.html)