Update user instruction data analysis #243
@@ -38,11 +38,11 @@ Every content page starts with a **cross-reference label** followed by a single
|
||||
top-level heading:
|
||||
|
||||
```md
|
||||
(user.saxs.data_analysis)=
|
||||
# Data analysis
|
||||
(user.saxs.online_integration)=
|
||||
# Setup online radial integration on Jungfraujoch
|
||||
```
|
||||
|
||||
- The label `(user.saxs.data_analysis)=` is what other pages, the section landing
|
||||
- The label `(user.saxs.online_integration)=` is what other pages, the section landing
|
||||
pages, and the `{ref}` role link to. **Do not delete or rename it** without
|
||||
updating every reference, or links silently break.
|
||||
- The `# Title` is the page's H1. Sphinx needs exactly one; section pages do not
|
||||
@@ -81,7 +81,7 @@ Available types include `note`, `tip`, `important`, `warning`, `caution`, `dange
|
||||
- **To another page in this documentation**, link by its label, not by file path:
|
||||
|
||||
```md
|
||||
See {ref}`the data analysis guide <user.saxs.data_analysis>`.
|
||||
See {ref}`the Jungfraujoch setup guide <user.saxs.online_integration>`.
|
||||
```
|
||||
|
||||
- **External links** use ordinary Markdown: `[pyFAI](https://pyfai.readthedocs.io)`.
|
||||
@@ -114,7 +114,7 @@ Available types include `note`, `tip`, `important`, `warning`, `caution`, `dange
|
||||
hidden: true
|
||||
---
|
||||
|
||||
data_analysis
|
||||
online_integration
|
||||
<name>
|
||||
```
|
||||
````
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
(user.saxs.data_analysis)=
|
||||
# Data analysis
|
||||
(user.data_analysis.computing)=
|
||||
# Online data analysis
|
||||
|
||||
We can access the online computing resources from the beamline consoles when logged in as the e-account.
|
||||
|
||||
@@ -16,12 +16,23 @@ ssh -Y [ra-c-110]
|
||||
Or for a GPU node:
|
||||
|
||||
```bash
|
||||
salloc -A csaxs -p gpu-csaxs --gpus=4 [-w ra-gpu-003]
|
||||
salloc -A csaxs -p gpu-csaxs --gpus=4 [-w ra-gpu-003] --mem=100G -n 8
|
||||
ssh -Y [ra-gpu-003]
|
||||
```
|
||||
|
||||
(note that we need to ask them to explicitly use one of our GPU nodes there, and this needs a couple of days to set up)
|
||||
|
||||
### Specifications of the computing nodes
|
||||
|
||||
A few examples of the available nodes:
|
||||
|
||||
| Node | CPU | Cores | Memory | GPUs | Partition |
|
||||
|---|---|---|---|---|---|
|
||||
| `ra-c-110` | Intel Xeon 6152 | 44 | 377 GB | — | `csaxs` |
|
||||
| `ra-gpu-003` | AMD EPYC 7452 | 64 | 1007 GB | 4x NVIDIA A100 (40 GB) | `gpu-csaxs` |
|
||||
| `ra-gpu-007` | Intel Xeon 6530 | 64 | 1007 GB | 4x NVIDIA L40S (46 GB)| `gpu-csaxs` |
|
||||
|
||||
|
||||
**The home directory is the p group.** To go to the raw directory we need to navigate there:
|
||||
|
||||
```bash
|
||||
@@ -81,7 +92,7 @@ cd /sls/x12sa/data/[p-group]/raw/python-env
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
#### Troubleshooting
|
||||
|
||||
If you encounter the following error when launching GUI applications (e.g. `pyFAI-calib2`):
|
||||
|
||||
@@ -97,7 +108,7 @@ ssh -X [ra-c-110]
|
||||
|
||||
and then reactivate the virtual environment before launching the application.
|
||||
|
||||
## Use Jupyter Notebook / Lab
|
||||
### Use Jupyter Notebook / Lab
|
||||
|
||||
Create a virtual environment following the instructions above, then:
|
||||
|
||||
@@ -110,40 +121,15 @@ To run Jupyter Lab:
|
||||
```bash
|
||||
jupyter lab --port 6006 --ip $(hostname) --no-browser
|
||||
```
|
||||
## cSAXS matlab scripts
|
||||
|
||||
## Setup online radial integration on Jungfraujoch
|
||||
To use the cSAXS matlab scripts we can go to the `cxs_software` package in the raw folder.
|
||||
|
||||
1. **Calibrate the detector using pyFAI.**
|
||||
|
||||
Analyze a calibration scan carefully with `pyFAI` and save the resulting **PONI** file and **pixel mask**. Also determine the beam center position.
|
||||
|
||||
:::{note}
|
||||
As of 2026-06-18, Jungfraujoch only accepts `uint32` TIFF mask files. If the mask generated by `pyFAI` is stored as `uint8`, it needs to be converted to the required format before uploading.
|
||||
To do this (from the home directory `/sls/x12sa/data/[p-group]/raw/`):
|
||||
|
||||
- Activate the python environment: `source mask/.mask/bin/activate`
|
||||
- Run `python convert_mask_format.py -i <input .tif file path> -o <output .tiff file path>`
|
||||
:::
|
||||
|
||||
2. **Update the BEC configuration.**
|
||||
|
||||
Update the detector distance and beam center coordinates in the BEC configuration files using the calibration results.
|
||||
|
||||
3. **Configure Jungfraujoch.**
|
||||
|
||||
Open the **Jungfraujoch Expert Configuration** panel in the frontend interface, for example:
|
||||
|
||||
```text
|
||||
http://sls-jfjoch-001:8080/frontend
|
||||
```
|
||||
|
||||
- Upload the pixel mask under **Pixel Mask**.
|
||||
- Fill in the required parameters under **Azimuthal Integration Settings**.
|
||||
- Make sure to enable **Force CPU calculation in FPGA workflow**.
|
||||
- After uploading a new configuration, do not press "initialize", as this will remove all the previously uploaded configuration.
|
||||
|
||||
4. **Detector rotation (if required).**
|
||||
|
||||
:::{note}
|
||||
Adding rotation of the detector still needs to be documented (likely configured in the BEC configuration).
|
||||
:::
|
||||
In base we can do:
|
||||
```bash
|
||||
source setup-environment.sh
|
||||
```
|
||||
And then we can use Matlab interactively as usual.
|
||||
```bash
|
||||
matlab &
|
||||
```
|
||||
@@ -0,0 +1,34 @@
|
||||
(user.data_analysis)=
|
||||
# Data analysis
|
||||
Welcome to the data analysis section of the cSAXS beamline.
|
||||
|
||||
```{toctree}
|
||||
---
|
||||
maxdepth: 2
|
||||
hidden: true
|
||||
---
|
||||
|
||||
computing
|
||||
|
||||
```
|
||||
|
||||
|
||||
***
|
||||
|
||||
````{grid} 2
|
||||
:gutter: 5
|
||||
|
||||
```{grid-item-card}
|
||||
:link: user.data_analysis.computing
|
||||
:link-type: ref
|
||||
:img-top: /assets/index_user_guide.svg
|
||||
:text-align: center
|
||||
:class-item: index-card
|
||||
|
||||
## Online data analysis
|
||||
|
||||
Allocating compute nodes, programming using Python and Matlab during the beamtime.
|
||||
```
|
||||
|
||||
````
|
||||
For offline analysis, more detailed documentation on use of slurm on Ra, description of NoMachine setup is available at https://dari.pages.psi.ch/ra/compute.html/.
|
||||
@@ -0,0 +1,37 @@
|
||||
(user.saxs.online_integration)=
|
||||
# Setup online radial integration on Jungfraujoch
|
||||
|
||||
1. **Calibrate the detector using pyFAI.**
|
||||
|
||||
Analyze a calibration scan carefully with `pyFAI` and save the resulting **PONI** file and **pixel mask**. Also determine the beam center position.
|
||||
|
||||
:::{note}
|
||||
As of 2026-06-18, Jungfraujoch only accepts `uint32` TIFF mask files. If the mask generated by `pyFAI` is stored as `uint8`, it needs to be converted to the required format before uploading.
|
||||
To do this (from the home directory `/sls/x12sa/data/[p-group]/raw/`):
|
||||
|
||||
- Activate the python environment: `source mask/.mask/bin/activate`
|
||||
- Run `python convert_mask_format.py -i <input .tif file path> -o <output .tiff file path>`
|
||||
:::
|
||||
|
||||
2. **Update the BEC configuration.**
|
||||
|
||||
Update the detector distance and beam center coordinates in the BEC configuration files using the calibration results.
|
||||
|
||||
3. **Configure Jungfraujoch.**
|
||||
|
||||
Open the **Jungfraujoch Expert Configuration** panel in the frontend interface, for example:
|
||||
|
||||
```text
|
||||
http://sls-jfjoch-001:8080/frontend
|
||||
```
|
||||
|
||||
- Upload the pixel mask under **Pixel Mask**.
|
||||
- Fill in the required parameters under **Azimuthal Integration Settings**.
|
||||
- Make sure to enable **Force CPU calculation in FPGA workflow**.
|
||||
- After uploading a new configuration, do not press "initialize", as this will remove all the previously uploaded configuration.
|
||||
|
||||
4. **Detector rotation (if required).**
|
||||
|
||||
:::{note}
|
||||
Adding rotation of the detector still needs to be documented (likely configured in the BEC configuration).
|
||||
:::
|
||||
@@ -8,7 +8,7 @@ maxdepth: 2
|
||||
hidden: true
|
||||
---
|
||||
|
||||
data_analysis
|
||||
online_integration
|
||||
|
||||
```
|
||||
|
||||
@@ -19,7 +19,7 @@ data_analysis
|
||||
:gutter: 5
|
||||
|
||||
```{grid-item-card}
|
||||
:link: user.saxs.data_analysis
|
||||
:link: user.saxs.online_integration
|
||||
:link-type: ref
|
||||
:img-top: /assets/biotech.svg
|
||||
:text-align: center
|
||||
@@ -27,7 +27,7 @@ data_analysis
|
||||
|
||||
## Data analysis
|
||||
|
||||
Allocating compute nodes, pyFAI installation and usage, and online radial integration on Jungfraujoch.
|
||||
Setup of online radial integration on Jungfraujoch.
|
||||
```
|
||||
|
||||
````
|
||||
|
||||
@@ -11,6 +11,7 @@ hidden: true
|
||||
before_you_arrive/before_you_arrive.md
|
||||
ptychography/ptychography.md
|
||||
saxs/saxs.md
|
||||
data_analysis/data_analysis.md
|
||||
|
||||
```
|
||||
|
||||
@@ -55,5 +56,17 @@ All about ptychography measurements and reconstructions.
|
||||
|
||||
All about SAXS measurements and analysis.
|
||||
```
|
||||
```{grid-item-card}
|
||||
:link: user.data_analysis
|
||||
:link-type: ref
|
||||
:img-top: /assets/biotech.svg
|
||||
:text-align: center
|
||||
:class-item: index-card
|
||||
|
||||
## Data analysis
|
||||
|
||||
All about data analysis during the beamtime.
|
||||
```
|
||||
|
||||
|
||||
````
|
||||
Reference in New Issue
Block a user