Update user instruction data analysis #247

Merged
menzel merged 5 commits from zhang_y7/csaxs_bec:update_user_instruction_data_analysis into main 2026-07-03 16:30:56 +02:00
7 changed files with 120 additions and 49 deletions
+1
View File
@@ -0,0 +1 @@
<?xml version="1.0" encoding="utf-8"?><svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" viewBox="0 0 122.88 101.91" style="enable-background:new 0 0 122.88 101.91" xml:space="preserve"><style type="text/css">.st0{fill-rule:evenodd;clip-rule:evenodd;}</style><g><path class="st0" d="M3.34,0h116.2c1.84,0,3.34,1.5,3.34,3.34v76.98c0,1.84-1.5,3.34-3.34,3.34H3.34C1.5,83.66,0,82.16,0,80.32 V3.34C0,1.5,1.5,0,3.34,0L3.34,0L3.34,0z M12.57,11.68h9.82v2.56h-9.82V11.68L12.57,11.68z M27.07,38.8h-14.5v-2.57h14.5V38.8 L27.07,38.8z M46.8,38.81H30.02v-2.58H46.8V38.81L46.8,38.81z M75.66,38.81H49.64v-2.57h26.03V38.81L75.66,38.81z M37.25,28.07 H88.4v2.53H37.25V28.07L37.25,28.07z M25.22,28.05h9.26v2.56h-9.26V28.05L25.22,28.05z M12.57,28.05h9.82v2.56h-9.82V28.05 L12.57,28.05z M34.77,22.41H12.57v-2.54h22.21V22.41L34.77,22.41z M46.8,22.42h-9.26v-2.56h9.26V22.42L46.8,22.42z M75.66,22.43 H49.64v-2.57h26.03V22.43L75.66,22.43z M37.25,11.68h22.21v2.54H37.25V11.68L37.25,11.68z M25.22,11.68h9.26v2.56h-9.26V11.68 L25.22,11.68z M91.2,62.08l-9.1-5.1v-2.92l9.1-5.1v3.36l-6.34,3.16l6.34,3.26V62.08L91.2,62.08z M92.89,64.35l3.29-17.91h1.91 l-3.31,17.91H92.89L92.89,64.35z M99.48,62.1v-3.34l6.35-3.2l-6.35-3.24V49l9.1,5.14v2.88L99.48,62.1L99.48,62.1z M46.29,88.27 h30.3c0.08,5.24,2.24,9.93,8.09,13.63H38.2C42.88,98.51,46.31,94.39,46.29,88.27L46.29,88.27L46.29,88.27z M61.44,72.41 c2.37,0,4.29,1.92,4.29,4.29c0,2.37-1.92,4.29-4.29,4.29c-2.37,0-4.29-1.92-4.29-4.29C57.15,74.33,59.07,72.41,61.44,72.41 L61.44,72.41z M10.05,6.29h102.79c1.63,0,2.95,1.33,2.95,2.95v57.52c0,1.62-1.33,2.95-2.95,2.95H10.05c-1.62,0-2.95-1.33-2.95-2.95 V9.24C7.09,7.62,8.42,6.29,10.05,6.29L10.05,6.29L10.05,6.29z"/></g></svg>

After

Width:  |  Height:  |  Size: 1.7 KiB

+5 -5
View File
@@ -38,11 +38,11 @@ Every content page starts with a **cross-reference label** followed by a single
top-level heading:
```md
(user.saxs.data_analysis)=
# Data analysis
(user.saxs.online_integration)=
# Setup online radial integration on Jungfraujoch
```
- The label `(user.saxs.data_analysis)=` is what other pages, the section landing
- The label `(user.saxs.online_integration)=` is what other pages, the section landing
pages, and the `{ref}` role link to. **Do not delete or rename it** without
updating every reference, or links silently break.
- The `# Title` is the page's H1. Sphinx needs exactly one; section pages do not
@@ -81,7 +81,7 @@ Available types include `note`, `tip`, `important`, `warning`, `caution`, `dange
- **To another page in this documentation**, link by its label, not by file path:
```md
See {ref}`the data analysis guide <user.saxs.data_analysis>`.
See {ref}`the Jungfraujoch setup guide <user.saxs.online_integration>`.
```
- **External links** use ordinary Markdown: `[pyFAI](https://pyfai.readthedocs.io)`.
@@ -114,7 +114,7 @@ Available types include `note`, `tip`, `important`, `warning`, `caution`, `dange
hidden: true
---
data_analysis
online_integration
<name>
```
````
@@ -1,5 +1,5 @@
(user.saxs.data_analysis)=
# Data analysis
(user.data_analysis.computing)=
# Online data analysis
We can access the online computing resources from the beamline consoles when logged in as the e-account.
@@ -16,12 +16,23 @@ ssh -Y [ra-c-110]
Or for a GPU node:
```bash
salloc -A csaxs -p gpu-csaxs --gpus=4 [-w ra-gpu-003]
salloc -A csaxs -p gpu-csaxs --gpus=4 [-w ra-gpu-003] --mem=100G -n 8
ssh -Y [ra-gpu-003]
```
(note that we need to ask them to explicitly use one of our GPU nodes there, and this needs a couple of days to set up)
### Specifications of the computing nodes
A few examples of the available nodes:
| Node | CPU | Cores | Memory | GPUs | Partition |
|---|---|---|---|---|---|
| `ra-c-110` | Intel Xeon 6152 | 44 | 377 GB | — | `csaxs` |
| `ra-gpu-003` | AMD EPYC 7452 | 64 | 1007 GB | 4x NVIDIA A100 (40 GB) | `gpu-csaxs` |
| `ra-gpu-007` | Intel Xeon 6530 | 64 | 1007 GB | 4x NVIDIA L40S (46 GB)| `gpu-csaxs` |
**The home directory is the p group.** To go to the raw directory we need to navigate there:
```bash
@@ -81,7 +92,7 @@ cd /sls/x12sa/data/[p-group]/raw/python-env
source .venv/bin/activate
```
### Troubleshooting
#### Troubleshooting
If you encounter the following error when launching GUI applications (e.g. `pyFAI-calib2`):
@@ -97,7 +108,7 @@ ssh -X [ra-c-110]
and then reactivate the virtual environment before launching the application.
## Use Jupyter Notebook / Lab
### Use Jupyter Notebook / Lab
Create a virtual environment following the instructions above, then:
@@ -110,40 +121,15 @@ To run Jupyter Lab:
```bash
jupyter lab --port 6006 --ip $(hostname) --no-browser
```
## cSAXS matlab scripts
## Setup online radial integration on Jungfraujoch
To use the cSAXS matlab scripts we can go to the `cxs_software` package in the raw folder.
1. **Calibrate the detector using pyFAI.**
Analyze a calibration scan carefully with `pyFAI` and save the resulting **PONI** file and **pixel mask**. Also determine the beam center position.
:::{note}
As of 2026-06-18, Jungfraujoch only accepts `uint32` TIFF mask files. If the mask generated by `pyFAI` is stored as `uint8`, it needs to be converted to the required format before uploading.
To do this (from the home directory `/sls/x12sa/data/[p-group]/raw/`):
- Activate the python environment: `source mask/.mask/bin/activate`
- Run `python convert_mask_format.py -i <input .tif file path> -o <output .tiff file path>`
:::
2. **Update the BEC configuration.**
Update the detector distance and beam center coordinates in the BEC configuration files using the calibration results.
3. **Configure Jungfraujoch.**
Open the **Jungfraujoch Expert Configuration** panel in the frontend interface, for example:
```text
http://sls-jfjoch-001:8080/frontend
```
- Upload the pixel mask under **Pixel Mask**.
- Fill in the required parameters under **Azimuthal Integration Settings**.
- Make sure to enable **Force CPU calculation in FPGA workflow**.
- After uploading a new configuration, do not press "initialize", as this will remove all the previously uploaded configuration.
4. **Detector rotation (if required).**
:::{note}
Adding rotation of the detector still needs to be documented (likely configured in the BEC configuration).
:::
In base we can do:
```bash
source setup-environment.sh
```
And then we can use Matlab interactively as usual.
```bash
matlab &
```
+34
View File
@@ -0,0 +1,34 @@
(user.data_analysis)=
# Data analysis
Welcome to the data analysis section of the cSAXS beamline.
```{toctree}
---
maxdepth: 2
hidden: true
---
computing
```
***
````{grid} 2
:gutter: 5
```{grid-item-card}
:link: user.data_analysis.computing
:link-type: ref
:img-top: /assets/index_user_guide.svg
:text-align: center
:class-item: index-card
## Online data analysis
Allocating compute nodes, programming using Python and Matlab during the beamtime.
```
````
For offline analysis, more detailed documentation on use of slurm on Ra, description of NoMachine setup is available at https://dari.pages.psi.ch/ra/compute.html/.
+37
View File
@@ -0,0 +1,37 @@
(user.saxs.online_integration)=
# Setup online radial integration on Jungfraujoch
1. **Calibrate the detector using pyFAI.**
Analyze a calibration scan carefully with `pyFAI` and save the resulting **PONI** file and **pixel mask**. Also determine the beam center position.
:::{note}
As of 2026-06-18, Jungfraujoch only accepts `uint32` TIFF mask files. If the mask generated by `pyFAI` is stored as `uint8`, it needs to be converted to the required format before uploading.
To do this (from the home directory `/sls/x12sa/data/[p-group]/raw/`):
- Activate the python environment: `source mask/.mask/bin/activate`
- Run `python convert_mask_format.py -i <input .tif file path> -o <output .tiff file path>`
:::
2. **Update the BEC configuration.**
Update the detector distance and beam center coordinates in the BEC configuration files using the calibration results.
3. **Configure Jungfraujoch.**
Open the **Jungfraujoch Expert Configuration** panel in the frontend interface, for example:
```text
http://sls-jfjoch-001:8080/frontend
```
- Upload the pixel mask under **Pixel Mask**.
- Fill in the required parameters under **Azimuthal Integration Settings**.
- Make sure to enable **Force CPU calculation in FPGA workflow**.
- After uploading a new configuration, do not press "initialize", as this will remove all the previously uploaded configuration.
4. **Detector rotation (if required).**
:::{note}
Adding rotation of the detector still needs to be documented (likely configured in the BEC configuration).
:::
+4 -4
View File
@@ -8,7 +8,7 @@ maxdepth: 2
hidden: true
---
data_analysis
online_integration
```
@@ -19,15 +19,15 @@ data_analysis
:gutter: 5
```{grid-item-card}
:link: user.saxs.data_analysis
:link: user.saxs.online_integration
:link-type: ref
:img-top: /assets/biotech.svg
:text-align: center
:class-item: index-card
## Data analysis
## Online integration
Allocating compute nodes, pyFAI installation and usage, and online radial integration on Jungfraujoch.
Setup of online radial integration on Jungfraujoch.
```
````
+13
View File
@@ -11,6 +11,7 @@ hidden: true
before_you_arrive/before_you_arrive.md
ptychography/ptychography.md
saxs/saxs.md
data_analysis/data_analysis.md
```
@@ -55,5 +56,17 @@ All about ptychography measurements and reconstructions.
All about SAXS measurements and analysis.
```
```{grid-item-card}
:link: user.data_analysis
:link-type: ref
:img-top: /assets/programming.svg
:text-align: center
:class-item: index-card
## Data analysis
All about data analysis during the beamtime.
```
````