diff --git a/docs/developer/editing_docs.md b/docs/developer/editing_docs.md index 6481740..7b10a2c 100644 --- a/docs/developer/editing_docs.md +++ b/docs/developer/editing_docs.md @@ -38,11 +38,11 @@ Every content page starts with a **cross-reference label** followed by a single top-level heading: ```md -(user.saxs.data_analysis)= -# Data analysis +(user.saxs.online_integration)= +# Setup online radial integration on Jungfraujoch ``` -- The label `(user.saxs.data_analysis)=` is what other pages, the section landing +- The label `(user.saxs.online_integration)=` is what other pages, the section landing pages, and the `{ref}` role link to. **Do not delete or rename it** without updating every reference, or links silently break. - The `# Title` is the page's H1. Sphinx needs exactly one; section pages do not @@ -81,7 +81,7 @@ Available types include `note`, `tip`, `important`, `warning`, `caution`, `dange - **To another page in this documentation**, link by its label, not by file path: ```md - See {ref}`the data analysis guide `. + See {ref}`the Jungfraujoch setup guide `. ``` - **External links** use ordinary Markdown: `[pyFAI](https://pyfai.readthedocs.io)`. @@ -114,7 +114,7 @@ Available types include `note`, `tip`, `important`, `warning`, `caution`, `dange hidden: true --- - data_analysis + online_integration ``` ```` diff --git a/docs/user/data_analysis/computing.md b/docs/user/data_analysis/computing.md new file mode 100644 index 0000000..54283d4 --- /dev/null +++ b/docs/user/data_analysis/computing.md @@ -0,0 +1,135 @@ +(user.data_analysis.computing)= +# Online data analysis + +We can access the online computing resources from the beamline consoles when logged in as the e-account. + +## Allocation of nodes + +Check available nodes and allocate one: + +```bash +sinfo +salloc -A csaxs -p csaxs -w [ra-c-110] +ssh -Y [ra-c-110] +``` + +Or for a GPU node: + +```bash +salloc -A csaxs -p gpu-csaxs --gpus=4 [-w ra-gpu-003] --mem=100G -n 8 +ssh -Y [ra-gpu-003] +``` + +(note that we need to ask them to explicitly use one of our GPU nodes there, and this needs a couple of days to set up) + +### Specifications of the computing nodes + +A few examples of the available nodes: + +| Node | CPU | Cores | Memory | GPUs | Partition | +|---|---|---|---|---|---| +| `ra-c-110` | Intel Xeon 6152 | 44 | 377 GB | — | `csaxs` | +| `ra-gpu-003` | AMD EPYC 7452 | 64 | 1007 GB | 4x NVIDIA A100 (40 GB) | `gpu-csaxs` | +| `ra-gpu-007` | Intel Xeon 6530 | 64 | 1007 GB | 4x NVIDIA L40S (46 GB)| `gpu-csaxs` | + + +**The home directory is the p group.** To go to the raw directory we need to navigate there: + +```bash +cd /sls/x12sa/data/[p22598]/raw +``` + +## Python tools + +### Python package installation + +Anaconda is no longer supported. Packages should now be installed directly using `pip`. + +For example, `pyFAI` can be installed as follows: + +```bash +module load Python/3.11.11 +pip install "pyFAI[gui]" +``` + +To launch the `pyFAI` applications: + +```bash +# Detector calibration +python -m pyFAI.app.calib2 + +# Batch integration +python -m pyFAI.app.integrate +``` + +### Recommended: using a Python virtual environment + +For long-term maintainability and reproducibility, it is recommended to use a dedicated Python virtual environment. + +#### Create a virtual environment + +```bash +cd /sls/x12sa/data/[p-group]/raw + +module load Python/3.11.11 +mkdir python-env +cd python-env + +# Create a virtual environment named ".venv" +python3 -m venv .venv + +# Activate the environment +source .venv/bin/activate + +# Install required packages +pip install "pyFAI[gui]" +``` + +#### Use an existing virtual environment + +```bash +cd /sls/x12sa/data/[p-group]/raw/python-env +source .venv/bin/activate +``` + +#### Troubleshooting + +If you encounter the following error when launching GUI applications (e.g. `pyFAI-calib2`): + +```text +X11 connection rejected +``` + +reconnect to the analysis node with X11 forwarding enabled: + +```bash +ssh -X [ra-c-110] +``` + +and then reactivate the virtual environment before launching the application. + +### Use Jupyter Notebook / Lab + +Create a virtual environment following the instructions above, then: + +```bash +pip install jupyter +``` + +To run Jupyter Lab: + +```bash +jupyter lab --port 6006 --ip $(hostname) --no-browser +``` +## cSAXS matlab scripts + +To use the cSAXS matlab scripts we can go to the `cxs_software` package in the raw folder. + +In base we can do: +```bash +source setup-environment.sh +``` +And then we can use Matlab interactively as usual. +```bash +matlab & +``` diff --git a/docs/user/data_analysis/data_analysis.md b/docs/user/data_analysis/data_analysis.md new file mode 100644 index 0000000..c145f37 --- /dev/null +++ b/docs/user/data_analysis/data_analysis.md @@ -0,0 +1,34 @@ +(user.data_analysis)= +# Data analysis +Welcome to the data analysis section of the cSAXS beamline. + +```{toctree} +--- +maxdepth: 2 +hidden: true +--- + +computing + +``` + + +*** + +````{grid} 2 +:gutter: 5 + +```{grid-item-card} +:link: user.data_analysis.computing +:link-type: ref +:img-top: /assets/index_user_guide.svg +:text-align: center +:class-item: index-card + +## Online data analysis + +Allocating compute nodes, programming using Python and Matlab during the beamtime. +``` + +```` +For offline analysis, more detailed documentation on use of slurm on Ra, description of NoMachine setup is available at https://dari.pages.psi.ch/ra/compute.html/. diff --git a/docs/user/saxs/online_integration.md b/docs/user/saxs/online_integration.md index d50caf3..6ff984f 100644 --- a/docs/user/saxs/online_integration.md +++ b/docs/user/saxs/online_integration.md @@ -1,117 +1,5 @@ -(user.saxs.data_analysis)= -# Data analysis - -We can access the online computing resources from the beamline consoles when logged in as the e-account. - -## Allocation of nodes - -Check available nodes and allocate one: - -```bash -sinfo -salloc -A csaxs -p csaxs -w [ra-c-110] -ssh -Y [ra-c-110] -``` - -Or for a GPU node: - -```bash -salloc -A csaxs -p gpu-csaxs --gpus=4 [-w ra-gpu-003] -ssh -Y [ra-gpu-003] -``` - -(note that we need to ask them to explicitly use one of our GPU nodes there, and this needs a couple of days to set up) - -**The home directory is the p group.** To go to the raw directory we need to navigate there: - -```bash -cd /sls/x12sa/data/[p22598]/raw -``` - -## Python tools - -### Python package installation - -Anaconda is no longer supported. Packages should now be installed directly using `pip`. - -For example, `pyFAI` can be installed as follows: - -```bash -module load Python/3.11.11 -pip install "pyFAI[gui]" -``` - -To launch the `pyFAI` applications: - -```bash -# Detector calibration -python -m pyFAI.app.calib2 - -# Batch integration -python -m pyFAI.app.integrate -``` - -### Recommended: using a Python virtual environment - -For long-term maintainability and reproducibility, it is recommended to use a dedicated Python virtual environment. - -#### Create a virtual environment - -```bash -cd /sls/x12sa/data/[p-group]/raw - -module load Python/3.11.11 -mkdir python-env -cd python-env - -# Create a virtual environment named ".venv" -python3 -m venv .venv - -# Activate the environment -source .venv/bin/activate - -# Install required packages -pip install "pyFAI[gui]" -``` - -#### Use an existing virtual environment - -```bash -cd /sls/x12sa/data/[p-group]/raw/python-env -source .venv/bin/activate -``` - -### Troubleshooting - -If you encounter the following error when launching GUI applications (e.g. `pyFAI-calib2`): - -```text -X11 connection rejected -``` - -reconnect to the analysis node with X11 forwarding enabled: - -```bash -ssh -X [ra-c-110] -``` - -and then reactivate the virtual environment before launching the application. - -## Use Jupyter Notebook / Lab - -Create a virtual environment following the instructions above, then: - -```bash -pip install jupyter -``` - -To run Jupyter Lab: - -```bash -jupyter lab --port 6006 --ip $(hostname) --no-browser -``` - -## Setup online radial integration on Jungfraujoch +(user.saxs.online_integration)= +# Setup online radial integration on Jungfraujoch 1. **Calibrate the detector using pyFAI.** diff --git a/docs/user/saxs/saxs.md b/docs/user/saxs/saxs.md index 5bef4d6..a314800 100644 --- a/docs/user/saxs/saxs.md +++ b/docs/user/saxs/saxs.md @@ -8,7 +8,7 @@ maxdepth: 2 hidden: true --- -data_analysis +online_integration ``` @@ -19,7 +19,7 @@ data_analysis :gutter: 5 ```{grid-item-card} -:link: user.saxs.data_analysis +:link: user.saxs.online_integration :link-type: ref :img-top: /assets/biotech.svg :text-align: center @@ -27,7 +27,7 @@ data_analysis ## Data analysis -Allocating compute nodes, pyFAI installation and usage, and online radial integration on Jungfraujoch. +Setup of online radial integration on Jungfraujoch. ``` ```` diff --git a/docs/user/user.md b/docs/user/user.md index 11081e7..4fbec36 100644 --- a/docs/user/user.md +++ b/docs/user/user.md @@ -11,6 +11,7 @@ hidden: true before_you_arrive/before_you_arrive.md ptychography/ptychography.md saxs/saxs.md +data_analysis/data_analysis.md ``` @@ -55,5 +56,17 @@ All about ptychography measurements and reconstructions. All about SAXS measurements and analysis. ``` +```{grid-item-card} +:link: user.data_analysis +:link-type: ref +:img-top: /assets/biotech.svg +:text-align: center +:class-item: index-card + +## Data analysis + +All about data analysis during the beamtime. +``` + ```` \ No newline at end of file