first stab at mkdocs migration

refactor CSCS and Meg content

add merlin6 quick start

update merlin6 nomachine docs

give the userdoc its own color scheme

we use the Materials default one

refactored slurm general docs merlin6

add merlin6 JB docs

add software support m6 docs

add all files to nav

vibed changes #1

add missing pages

further vibing #2

vibe #3

further fixes

docs/merlin6/jupyterhub/jupyter-examples.md

@@ -0,0 +1,20 @@
+# Jupyter examples on merlin6
+
+These examples demonstrate the use of certain python libraries and modules in the merlin6 environment. They are provided to get you started fast. You can check out a repository of the examples from
+
+<https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples>
+
+A number of standard data sets for the tutorials of the libraries are hosted on merlin6 centrally under `/data/project/general/public`, so you do not need to store them in your user space.
+
+## Dask
+
+  [Dask](https://dask.org/) is a flexible library for parallel computing in Python. It provides the abstraction of a dask dataframe that can reside on multiple machines and can be manipulated by an API designed to be as close as possible to [pandas](https://pandas.pydata.org/). The example shows how to start up dask workers on merlin6 through slurm.
+
+* [Link to example](https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples/blob/master/dask-example.ipynb)
+* The data sets for the [dask tutorial](https://github.com/dask/dask-tutorial) are hosted at `/data/project/general/public/dask-tutorial`.
+
+## Plotly
+
+[Plotly](https://plot.ly/python/getting-started/) is an interactive open source plotting library
+
+* [Link to example](https://git.psi.ch/lsm-hpce/merlin6-jupyterhub-examples/blob/master/plotly-example.ipynb)

docs/merlin6/jupyterhub/jupyter-extensions.md

@@ -0,0 +1,30 @@
+# Jupyter Extensions
+
+## Using nbextensions for adding features to your notebook
+
+There exist a number of useful [contributed but unofficial
+extensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/index.html)
+that add useful features to your notebooks.
+
+From the classic Notebook UI you can access the available extensions in a
+separate tab as displayed in the screenshot, below. You may have to unselect
+the *disable configuration for nbextensions without explicit copatibility*. The
+extensions we tested still worked fine with this jupyterhub version of 1.0.0.
+
+![Launch Classic Notebook](../../images/jupyter/jupyter-launch-classic.png)
+/// caption
+///
+
+## Extensions for working with large notebooks
+
+Especially the following extensions make working with larger notebooks easier
+
+* **Table of Contents**: Displays a TOC on the left and you can also configure it
+  to add and update a TOC at the head of the document.
+* **Collapsible Headings**: allows you to fold all the cells below a heading
+
+It may also be interesting for you to explore the [Jupytext](jupytext.md) server extension.
+
+## Variable Inspector
+
+The `variable inspector` extension provides a constantly updated window in which you can see the value and type of your notebook's variables.

docs/merlin6/jupyterhub/jupyterhub-trouble.md

@@ -0,0 +1,78 @@
+# Jupyterhub Troubleshooting
+
+In case of problems or requests, please either submit a **[PSI Service
+Now](https://psi.service-now.com/psisp)** incident containing *"Merlin
+Jupyterhub"* as part of the subject, or contact us by mail through
+<merlin-admins@lists.psi.ch>.
+
+## General steps for troubleshooting
+
+### Investigate the Slurm output file
+
+Your jupyterhub session runs as a normal batch job on the cluster, and each
+launch will create a slurm output file in your *home* directory named like
+`jupyterhub_batchspawner_{$JOBID}.log`, where the `$JOBID` part is the slurm job
+ID of your job. After a failed launch, investigate the contents of that file.
+An error message will usually be found towards the end of the file, often
+including a python backtrace.
+
+### Investigate python environment interferences
+
+Jupyterhub just runs a jupyter notebook executable as your user inside the
+batch job. A frequent source of errors consists of a user's local python
+environment definitions getting mixed up with the environment that jupyter
+needs to launch.
+
+- setting PYTHONPATH inside of the ~/.bash_profile or any other startup script
+- having installed packages to your local user area (e.g. using `pip install
+  --user <some-package>`). Such installation will interfere with the
+  environment offered by the `module` system on our cluster (based on
+  anaconda). You can list such packages by executing `pip list user`.
+  They are usually located in `~/.local/lib/pythonX.Y/...`.
+
+You can investigate the launching of a notebook interactively, by logging in to
+Merlin6 and running a jupyter command in the correct environment.
+
+```bash
+module use unstable
+module load anaconda/2019.07
+conda activate jupyterhub-1.0.0_py36
+jupyter --paths
+```
+
+## Known Problems and workarounds
+
+### Spawner times out
+
+If the cluster is very full, it may be difficult to launch a session. We always
+reserve some slots for interactive Jupyterhub use, but it may be that these
+slots have been taken or that the resources you requested are currently not
+available.
+
+Inside of a Merlin6 terminal shell, you can run the standard commands like
+`sinfo` and `squeue` to get an overview of how full the cluster is.
+
+### Your user environment is not among the kernels offered for choice
+
+Refer to our documentation about [using your own custom made
+environments with jupyterhub](jupyterhub.md).
+
+### Cannot save notebook - *xsrf argument missing*
+
+You cannot save your notebook anymore and you get this error:
+
+```text
+'_xsrf' argument missing from POST
+```
+
+This issue occurs very seldomly. There exists the following workaround:
+
+Go to the jupyterhub file browsing window and just open another
+notebook using the same kernel in another browser window. The issue
+should then go away. For more information refer to [this github
+thread](https://github.com/nteract/hydrogen/issues/922#issuecomment-405456346)
+
+<!-- ## Error HTTP 500 when starting the spawner -->
+
+<!--    The spawner screen shows after launching an error message like the following: -->
+<!--    `Internal server error (Spawner failed to start [status=..]. The logs may contain details)` -->

docs/merlin6/jupyterhub/jupyterhub.md

@@ -0,0 +1,109 @@
+# Jupyterhub on Merlin
+
+Jupyterhub provides [jupyter notebooks](https://jupyter.org/) that are launched on
+cluster nodes of merlin and can be accessed through a web portal.
+
+## Accessing Jupyterhub and launching a session
+
+The service is available inside of PSI (or through a VPN connection) at
+
+**<https://merlin-jupyter.psi.ch:8000>**
+
+1. **Login**: You will be presented with a **Login** web page for
+  authenticating with your PSI account.
+1. **Spawn job**: The **Spawner Options** page allows you to
+  specify the properties (Slurm partition, running time,...) of
+  the batch jobs that will be running your jupyter notebook. Once
+  you click on the `Spawn` button, your job will be sent to the
+  Slurm batch system. If the cluster is not currently overloaded
+  and the resources you requested are available, your job will
+  usually start within 30 seconds.
+
+## Jupyter software environments - running different kernels
+
+Your notebooks can run within different software environments which are offered by a number of available **Jupyter kernels**.
+
+E.g. in this test installation we provide two environments targeted at data science
+
+* **tensorflow-1.13.1_py37**: contains Tensorflow, Keras, scikit-learn, Pandas, numpy, dask, and dependencies. Stable
+* **talos_py36**: also contains the Talos package. This
+  environment is experimental and subject to updates and changes.
+
+When you create a new notebook you will be asked to specify which kernel you
+want to use. It is also possible to switch the kernel of a running notebook,
+but you will lose the state of the current kernel, so you will have to
+recalculate the notebook cells with this new kernel.
+
+These environments are also available for standard work in a shell session. You
+can activate an environment in a normal merlin terminal session by using the
+`module` (q.v. [using Pmodules](../how-to-use-merlin/using-modules.md)) command to load anaconda
+python, and from there using the `conda` command to switch to the desired
+environment:
+
+```bash
+module use unstable
+module load anaconda/2019.07
+conda activate tensorflow-1.13.1_py36
+```
+
+When the `anaconda` module has been loaded, you can list the available environments by executing:
+
+```bash
+conda info -e
+```
+
+You can get more info on the use of the `conda` package management tool at its official [documentation site](https://conda.io/projects/conda/en/latest/commands.html).
+
+## Using your own custom made environments with jupyterhub
+
+Python environments can take up a lot of space due to the many dependencies that will be installed. You should always install your extra environments to the data area belonging to your account, e.g. `/data/user/${YOUR-USERNAME}/conda-envs`
+
+In order for jupyterhub (and jupyter in general) to recognize the provided
+environment as a valid kernel, make sure that you include the
+`nb_conda_kernels` package in your environment. This package provides the
+necessary activation and the dependencies.
+
+Example:
+
+```bash
+conda create -c conda-forge -p /data/user/${USER}/conda-envs/my-test-env python=3.7 nb_conda_kernels
+```
+
+After this, your new kernel will be visible as `my-test-env` inside of your jupyterhub session.
+
+## Requesting additional resources
+
+The **Spawner Options** page covers the most common options. These are used to
+create a submission script for the jupyterhub job and submit it to the slurm
+queue. Additional customization can be implemented using the *'Optional user
+defined line to be added to the batch launcher script'* option. This line is
+added to the submission script at the end of other `#SBATCH` lines. Parameters can
+be passed to SLURM by starting the line with `#SBATCH`, like in [Running Slurm
+Scripts](../slurm-general-docs/running-jobs.md). Some ideas:
+
+### Request additional memory
+
+```bash
+#SBATCH --mem=100G
+```
+
+### Request multiple GPUs (gpu partition only)
+
+```bash
+#SBATCH --gpus=2
+```
+
+### Log additional information
+
+```bash
+hostname; date; echo $USER
+```
+
+Output is found in `~/jupyterhub_batchspawner_<jobid>.log`.
+
+## Contact
+
+In case of problems or requests, please either submit a **[PSI Service
+Now](https://psi.service-now.com/psisp)** incident containing *"Merlin
+Jupyterhub"* as part of the subject, or contact us by mail through
+<merlin-admins@lists.psi.ch>.

docs/merlin6/jupyterhub/jupyterlab.md

@@ -0,0 +1,30 @@
+# Jupyterlab User interface
+
+## Testing out Jupyterlab
+
+**Jupyterlab** is a new interface to interact with your Jupyter notebooks.
+However, it is in very active development and undergoing constant changes. You
+can read about its features [on the official
+website](https://jupyterlab.readthedocs.io/en/stable/user/interface.html).
+
+You can test it out on our server by using the following kind of URL, where
+`$YOUR-USER` must be replaced by your PSI username. You must already have an
+active session on the jupyterhub.
+
+```text
+https://merlin-jupyter.psi.ch:8000/user/$YOUR-USER/lab
+```
+
+## Switching to the Classic Notebook user interface
+
+You can switch to the classical notebook UI by using the **"Launch Classic Notebook"** command from the left sidebar of JupyterLab.
+
+![Launch Classic Notebook](../../images/jupyter/jupyter-launch-classic.png)
+/// caption
+///
+
+## Jupyterlab does not support the older nbextensions
+
+These regrettably are not yet supported from within the JupyterLab UI,
+but you can activate them through the Classic Notebook interface (see
+above)

docs/merlin6/jupyterhub/jupytext.md

@@ -0,0 +1,52 @@
+# Jupytext - efficient editing
+
+[Jupytext](https://github.com/mwouts/jupytext) is a Jupyter serverextension
+that allows creating a text file from a notebook that can be kept in sync with
+it **with the aim of using more efficient editors or IDEs on it**. The file can
+be created in a number of formats, e.g. *markdown, *.py (light Script)*, and
+others. `Jupytext` will keep the both the notebook and this **paired** file in
+sync: If you save the paired file, changes will be carried over into the
+notebook, and vice versa. This pairing will persist also in new sessions of
+your notebook until you explicitely remove it again.
+
+The paired file contains only the cell contents and not the output. Therefore
+it also is **much better suited for revision control**, since the differences
+between versions are limited to the cells and these file formats yield more
+meaningful text differences than the default notebook storage format.
+
+## Creating a paired file in python format for efficient refactoring
+
+From your notebook, go to the `file` menu and navigate to the `jupytext`
+submenu. Select the **light script** pairing option. This will create a `*.py`
+file version with the same basename as your notebook file.
+
+![Jupytext menu](../../images/jupyter/jupytext_menu.png)
+/// caption
+///
+
+You can edit that file separately in your favourite python editor. The markdown
+text parts will be conserved in the file in the form of python comments.
+
+When you save the file and do a browser page reload of your jupyter notebook,
+you will see all the changes carried over into your jupyter notebook.
+
+## Creating a paired file in mardown format for efficient text authoring
+
+If you want to efficiently work on the descriptive text base of your notebook,
+just pair it using the `Pair notebook with Markdown` menu item and edit the
+generated `*.md` file with your favourite Markdown editor.
+
+## Disable autosaving when working  on the paired file
+
+Your notebooks usually auto save every 2 min (default). Turn this feature off
+when working with the paired file. Otherwise Jupyter will continue to save the
+state while you are editing the paired file, and the changes will be synced to
+the disk version of the paired file. You can disable the autosave by unchecking
+the `Autosave notebook` menu item in the Juytext menu (see above image).
+
+## Further information
+
+Please refer to:
+
+* [the Jupytext FAQ](https://jupytext.readthedocs.io/en/latest/faq.html)
+* [the Jupytext documentation](https://jupytext.readthedocs.io/en/latest/index.html)