From eecc6cf37b9f47b8b1ab2706566df47524b82da5 Mon Sep 17 00:00:00 2001 From: Yuhe Zhang Date: Fri, 3 Jul 2026 10:10:16 +0200 Subject: [PATCH 1/4] create a seperate section for data analysis --- docs/user/saxs/{data_analysis.md => online_integration.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/user/saxs/{data_analysis.md => online_integration.md} (100%) diff --git a/docs/user/saxs/data_analysis.md b/docs/user/saxs/online_integration.md similarity index 100% rename from docs/user/saxs/data_analysis.md rename to docs/user/saxs/online_integration.md -- 2.54.0 From 8cfc82dfe643c9e53b2de963b3867b15f287ea5c Mon Sep 17 00:00:00 2001 From: Yuhe Zhang Date: Fri, 3 Jul 2026 10:12:21 +0200 Subject: [PATCH 2/4] Update data analysis section --- docs/developer/editing_docs.md | 10 +- docs/user/data_analysis/computing.md | 135 +++++++++++++++++++++++ docs/user/data_analysis/data_analysis.md | 34 ++++++ docs/user/saxs/online_integration.md | 116 +------------------ docs/user/saxs/saxs.md | 6 +- docs/user/user.md | 13 +++ 6 files changed, 192 insertions(+), 122 deletions(-) create mode 100644 docs/user/data_analysis/computing.md create mode 100644 docs/user/data_analysis/data_analysis.md diff --git a/docs/developer/editing_docs.md b/docs/developer/editing_docs.md index 6481740..7b10a2c 100644 --- a/docs/developer/editing_docs.md +++ b/docs/developer/editing_docs.md @@ -38,11 +38,11 @@ Every content page starts with a **cross-reference label** followed by a single top-level heading: ```md -(user.saxs.data_analysis)= -# Data analysis +(user.saxs.online_integration)= +# Setup online radial integration on Jungfraujoch ``` -- The label `(user.saxs.data_analysis)=` is what other pages, the section landing +- The label `(user.saxs.online_integration)=` is what other pages, the section landing pages, and the `{ref}` role link to. **Do not delete or rename it** without updating every reference, or links silently break. - The `# Title` is the page's H1. Sphinx needs exactly one; section pages do not @@ -81,7 +81,7 @@ Available types include `note`, `tip`, `important`, `warning`, `caution`, `dange - **To another page in this documentation**, link by its label, not by file path: ```md - See {ref}`the data analysis guide `. + See {ref}`the Jungfraujoch setup guide `. ``` - **External links** use ordinary Markdown: `[pyFAI](https://pyfai.readthedocs.io)`. @@ -114,7 +114,7 @@ Available types include `note`, `tip`, `important`, `warning`, `caution`, `dange hidden: true --- - data_analysis + online_integration ``` ```` diff --git a/docs/user/data_analysis/computing.md b/docs/user/data_analysis/computing.md new file mode 100644 index 0000000..54283d4 --- /dev/null +++ b/docs/user/data_analysis/computing.md @@ -0,0 +1,135 @@ +(user.data_analysis.computing)= +# Online data analysis + +We can access the online computing resources from the beamline consoles when logged in as the e-account. + +## Allocation of nodes + +Check available nodes and allocate one: + +```bash +sinfo +salloc -A csaxs -p csaxs -w [ra-c-110] +ssh -Y [ra-c-110] +``` + +Or for a GPU node: + +```bash +salloc -A csaxs -p gpu-csaxs --gpus=4 [-w ra-gpu-003] --mem=100G -n 8 +ssh -Y [ra-gpu-003] +``` + +(note that we need to ask them to explicitly use one of our GPU nodes there, and this needs a couple of days to set up) + +### Specifications of the computing nodes + +A few examples of the available nodes: + +| Node | CPU | Cores | Memory | GPUs | Partition | +|---|---|---|---|---|---| +| `ra-c-110` | Intel Xeon 6152 | 44 | 377 GB | — | `csaxs` | +| `ra-gpu-003` | AMD EPYC 7452 | 64 | 1007 GB | 4x NVIDIA A100 (40 GB) | `gpu-csaxs` | +| `ra-gpu-007` | Intel Xeon 6530 | 64 | 1007 GB | 4x NVIDIA L40S (46 GB)| `gpu-csaxs` | + + +**The home directory is the p group.** To go to the raw directory we need to navigate there: + +```bash +cd /sls/x12sa/data/[p22598]/raw +``` + +## Python tools + +### Python package installation + +Anaconda is no longer supported. Packages should now be installed directly using `pip`. + +For example, `pyFAI` can be installed as follows: + +```bash +module load Python/3.11.11 +pip install "pyFAI[gui]" +``` + +To launch the `pyFAI` applications: + +```bash +# Detector calibration +python -m pyFAI.app.calib2 + +# Batch integration +python -m pyFAI.app.integrate +``` + +### Recommended: using a Python virtual environment + +For long-term maintainability and reproducibility, it is recommended to use a dedicated Python virtual environment. + +#### Create a virtual environment + +```bash +cd /sls/x12sa/data/[p-group]/raw + +module load Python/3.11.11 +mkdir python-env +cd python-env + +# Create a virtual environment named ".venv" +python3 -m venv .venv + +# Activate the environment +source .venv/bin/activate + +# Install required packages +pip install "pyFAI[gui]" +``` + +#### Use an existing virtual environment + +```bash +cd /sls/x12sa/data/[p-group]/raw/python-env +source .venv/bin/activate +``` + +#### Troubleshooting + +If you encounter the following error when launching GUI applications (e.g. `pyFAI-calib2`): + +```text +X11 connection rejected +``` + +reconnect to the analysis node with X11 forwarding enabled: + +```bash +ssh -X [ra-c-110] +``` + +and then reactivate the virtual environment before launching the application. + +### Use Jupyter Notebook / Lab + +Create a virtual environment following the instructions above, then: + +```bash +pip install jupyter +``` + +To run Jupyter Lab: + +```bash +jupyter lab --port 6006 --ip $(hostname) --no-browser +``` +## cSAXS matlab scripts + +To use the cSAXS matlab scripts we can go to the `cxs_software` package in the raw folder. + +In base we can do: +```bash +source setup-environment.sh +``` +And then we can use Matlab interactively as usual. +```bash +matlab & +``` diff --git a/docs/user/data_analysis/data_analysis.md b/docs/user/data_analysis/data_analysis.md new file mode 100644 index 0000000..c145f37 --- /dev/null +++ b/docs/user/data_analysis/data_analysis.md @@ -0,0 +1,34 @@ +(user.data_analysis)= +# Data analysis +Welcome to the data analysis section of the cSAXS beamline. + +```{toctree} +--- +maxdepth: 2 +hidden: true +--- + +computing + +``` + + +*** + +````{grid} 2 +:gutter: 5 + +```{grid-item-card} +:link: user.data_analysis.computing +:link-type: ref +:img-top: /assets/index_user_guide.svg +:text-align: center +:class-item: index-card + +## Online data analysis + +Allocating compute nodes, programming using Python and Matlab during the beamtime. +``` + +```` +For offline analysis, more detailed documentation on use of slurm on Ra, description of NoMachine setup is available at https://dari.pages.psi.ch/ra/compute.html/. diff --git a/docs/user/saxs/online_integration.md b/docs/user/saxs/online_integration.md index d50caf3..6ff984f 100644 --- a/docs/user/saxs/online_integration.md +++ b/docs/user/saxs/online_integration.md @@ -1,117 +1,5 @@ -(user.saxs.data_analysis)= -# Data analysis - -We can access the online computing resources from the beamline consoles when logged in as the e-account. - -## Allocation of nodes - -Check available nodes and allocate one: - -```bash -sinfo -salloc -A csaxs -p csaxs -w [ra-c-110] -ssh -Y [ra-c-110] -``` - -Or for a GPU node: - -```bash -salloc -A csaxs -p gpu-csaxs --gpus=4 [-w ra-gpu-003] -ssh -Y [ra-gpu-003] -``` - -(note that we need to ask them to explicitly use one of our GPU nodes there, and this needs a couple of days to set up) - -**The home directory is the p group.** To go to the raw directory we need to navigate there: - -```bash -cd /sls/x12sa/data/[p22598]/raw -``` - -## Python tools - -### Python package installation - -Anaconda is no longer supported. Packages should now be installed directly using `pip`. - -For example, `pyFAI` can be installed as follows: - -```bash -module load Python/3.11.11 -pip install "pyFAI[gui]" -``` - -To launch the `pyFAI` applications: - -```bash -# Detector calibration -python -m pyFAI.app.calib2 - -# Batch integration -python -m pyFAI.app.integrate -``` - -### Recommended: using a Python virtual environment - -For long-term maintainability and reproducibility, it is recommended to use a dedicated Python virtual environment. - -#### Create a virtual environment - -```bash -cd /sls/x12sa/data/[p-group]/raw - -module load Python/3.11.11 -mkdir python-env -cd python-env - -# Create a virtual environment named ".venv" -python3 -m venv .venv - -# Activate the environment -source .venv/bin/activate - -# Install required packages -pip install "pyFAI[gui]" -``` - -#### Use an existing virtual environment - -```bash -cd /sls/x12sa/data/[p-group]/raw/python-env -source .venv/bin/activate -``` - -### Troubleshooting - -If you encounter the following error when launching GUI applications (e.g. `pyFAI-calib2`): - -```text -X11 connection rejected -``` - -reconnect to the analysis node with X11 forwarding enabled: - -```bash -ssh -X [ra-c-110] -``` - -and then reactivate the virtual environment before launching the application. - -## Use Jupyter Notebook / Lab - -Create a virtual environment following the instructions above, then: - -```bash -pip install jupyter -``` - -To run Jupyter Lab: - -```bash -jupyter lab --port 6006 --ip $(hostname) --no-browser -``` - -## Setup online radial integration on Jungfraujoch +(user.saxs.online_integration)= +# Setup online radial integration on Jungfraujoch 1. **Calibrate the detector using pyFAI.** diff --git a/docs/user/saxs/saxs.md b/docs/user/saxs/saxs.md index 5bef4d6..a314800 100644 --- a/docs/user/saxs/saxs.md +++ b/docs/user/saxs/saxs.md @@ -8,7 +8,7 @@ maxdepth: 2 hidden: true --- -data_analysis +online_integration ``` @@ -19,7 +19,7 @@ data_analysis :gutter: 5 ```{grid-item-card} -:link: user.saxs.data_analysis +:link: user.saxs.online_integration :link-type: ref :img-top: /assets/biotech.svg :text-align: center @@ -27,7 +27,7 @@ data_analysis ## Data analysis -Allocating compute nodes, pyFAI installation and usage, and online radial integration on Jungfraujoch. +Setup of online radial integration on Jungfraujoch. ``` ```` diff --git a/docs/user/user.md b/docs/user/user.md index 11081e7..4fbec36 100644 --- a/docs/user/user.md +++ b/docs/user/user.md @@ -11,6 +11,7 @@ hidden: true before_you_arrive/before_you_arrive.md ptychography/ptychography.md saxs/saxs.md +data_analysis/data_analysis.md ``` @@ -55,5 +56,17 @@ All about ptychography measurements and reconstructions. All about SAXS measurements and analysis. ``` +```{grid-item-card} +:link: user.data_analysis +:link-type: ref +:img-top: /assets/biotech.svg +:text-align: center +:class-item: index-card + +## Data analysis + +All about data analysis during the beamtime. +``` + ```` \ No newline at end of file -- 2.54.0 From b3800c88157ae82b94864bd7ee86e17ea32a09b2 Mon Sep 17 00:00:00 2001 From: Yuhe Zhang Date: Fri, 3 Jul 2026 16:09:00 +0200 Subject: [PATCH 3/4] Add an icon for data analysis section --- docs/assets/programming.svg | 1 + docs/user/user.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 docs/assets/programming.svg diff --git a/docs/assets/programming.svg b/docs/assets/programming.svg new file mode 100644 index 0000000..bc9c155 --- /dev/null +++ b/docs/assets/programming.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/user/user.md b/docs/user/user.md index 4fbec36..763d47d 100644 --- a/docs/user/user.md +++ b/docs/user/user.md @@ -59,7 +59,7 @@ All about SAXS measurements and analysis. ```{grid-item-card} :link: user.data_analysis :link-type: ref -:img-top: /assets/biotech.svg +:img-top: /assets/programming.svg :text-align: center :class-item: index-card -- 2.54.0 From cb509fbfc688e1686036fe6c6a43a4da4d4591e6 Mon Sep 17 00:00:00 2001 From: Yuhe Zhang Date: Fri, 3 Jul 2026 16:10:47 +0200 Subject: [PATCH 4/4] Change the heading to online integration --- docs/user/saxs/saxs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user/saxs/saxs.md b/docs/user/saxs/saxs.md index a314800..abaa235 100644 --- a/docs/user/saxs/saxs.md +++ b/docs/user/saxs/saxs.md @@ -25,7 +25,7 @@ online_integration :text-align: center :class-item: index-card -## Data analysis +## Online integration Setup of online radial integration on Jungfraujoch. ``` -- 2.54.0