3.2 KiB
title, last_updated, sidebar, permalink
| title | last_updated | sidebar | permalink |
|---|---|---|---|
| Jupyterhub Troubleshooting | 18 February 2020 | merlin6_sidebar | /merlin6/jupyterhub-trouble.html |
In case of problems or requests, please either submit a PSI Service Now incident containing "Merlin Jupyterhub" as part of the subject, or contact us by mail through merlin-admins@lists.psi.ch.
General steps for troubleshooting
Investigate the Slurm output file
Your jupyterhub session runs as a normal batch job on the cluster, and each launch will create a slurm output file in your HOME directory named like jupyterhub_batchspawner_{$JOBID}.log, where the $JOBID part is the slurm job ID of your job. After a failed launch, investigate the contents of that file. An error message will usually be found towards the end of the file, often including a python backtrace.
Investigate python environment interferences
Jupyterhub just runs a jupyter notebook executable as your user inside the batch job. A frequent source of errors consists of a user's local python environment definitions getting mixed up with the environment that jupyter needs to launch.
- setting PYTHONPATH inside of the ~/.bash_profile or any other startup script
- having installed packages to your local user area (e.g. using
pip install --user <some-package>). Such installation will interfere with the environment offered by themodulesystem on our cluster (based on anaconda). You can list such packages by executingpip list user. They are usually located in~/.local/lib/pythonX.Y/....
You can investigate the launching of a notebook interactively, by logging in to Merlin6 and running a jupyter command in the correct environment.
module use unstable
module load anaconda/2019.07
conda activate jupyterhub-1.0.0_py36
jupyter --paths
Known Problems and workarounds
Spawner times out
If the cluster is very full, it may be difficult to launch a session. We always reserve some slots for interactive Jupyterhub use, but it may be that these slots have been taken or that the resources you requested are currently not available.
Inside of a Merlin6 terminal shell, you can run the standard commands like sinfo and squeue to get an overview of how full the cluster is.
Your user environment is not among the kernels offered for choice
Refer to our documentation about using your own custom made environments with jupyterhub.
Cannot save notebook - xsrf argument missing
You cannot save your notebook anymore and you get this error:
**'_xsrf' argument missing from POST**
This issue occurs very seldomly. There exists the following workaround:
Go to the jupyterhub file browsing window and just open another notebook using the same kernel in another browser window. The issue should then go away. For more information refer to this github thread