aareproc (0.1.16)

Published 2025-12-21 11:57:32 +01:00 by leonarski_f

To install the package using pip, run the following command:

pip install --index-url https://gitea.psi.ch/api/packages/mx/pypi/simple/ --extra-index-url https://pypi.org/simple aareproc

For more information on the PyPI registry, see the documentation.

HPC data processing pipeline (AareProc) for Swiss Light Source (SLS) 2.0 PXI, PXII, PXIII

AareProc

Description

Data processing pipeline that interacts with AareDB and process data for MX beamlines at SLS 2.0.

Service Operation

AareProc software is running as aareproc.service by svcusr-mx_writer on mx-x06da-queue-01.

Login with your own account to mx-x06da-queue-01:
- To check status: systemctl status aareproc.service
Need sudo rights To start/restart/stop the service: sudo systemctl start aareproc.service (replace the "start" to "restart" or "stop" as desired)
Need sudo rights To check the journal: sudo journalctl -u aareproc.service -f

Deployment

Git version of AareProc is stored in /sls/MX/applications/git/AareProc, when I made changes to the code, it will be synced to this folder.
Deployment is done by bash /sls/MX/applications/AareProc/deployment.sh
- This bash script copies aareproc and pdb_id_cell_update in /sls/MX/applications/git/AareProc/src to /sls/MX/applications/AareProc/src [!!! ATTENSION !!! Not the git folder, they are alike but not the same] and it restarts the aareproc.service to apply the latest changes
- To do this, you'll need write access to /sls/MX/applications, might only be feasible through beamline consoles or virtual machines when you are in the right group (unx-sls_mx)
- For testing, go to /sls/MX/applications/AareProc/src and make the current stable version to another name, e.g. aareproc_stable_2025Sept17, then deploy the latest change. If it doesn't work, rename the stable version back to aareproc and restart the aareproc.service (See Service Operation 👆).

Current Online Processing Folder Structure

Raw data is written by AareGUI/AareDAQ in /sls/mx/data/<pgroup>/raw
The processed data can be found in /sls/mx/data/<pgroup>/raw/processing then following the identical structure as in
- The common file structure will be /sls/mx/data/<pgroup>/raw/processing/<yyyymmdd>/<puckname>/<pin_number>
  - [!!! ATTENSION !!!] it could change depending on what you put in the AareGUI during data collection.
- Data path will be changed and tested after the current beamtime (September/October 2025)
In the corresponding sample folder, you will see:
- aareproc
  - various aareproc output files
  - [Folder] dimple_aareproc
- autoproc
  - various autoproc output files
  - [Folder] dimple_autoproc
- man_proc
  - XDS.INP
  - [Maybe] downloaded pdb file if user provided a PDB code in AareDB for this sample

Current Features

CLOUD feature was tested on September and temporarily disabled for the rest of September and October

Installation

To install the package using pip, run the following command:

pip install --index-url https://gitea.psi.ch/api/packages/mx/pypi/simple/ aareproc

Usage

If you want to run the program with interactive python:

Activate environment with source /sls/MX/applications/venvs/aareproc_mxsvc/bin/activate
For the database communication to work, need to export the shared password in the current shell.

Contributing

Jiaxin (Dawn) Duan

Authors and acknowledgment

Filip Leonarski (Advisor, Jungfraujoch, AareDAQ)
Guillaume Gotthard (AareDB)
Martin Appleby (AareGUI)

License

To be determined

Project status

In initial development

FAQ and Details

XDS.INP template

!Settings for PILATUS4 2M @ X06DA SLS 2.0
!Job control
JOB=XYCORR INIT COLSPOT IDXREF DEFPIX INTEGRATE CORRECT

ORGX=781.19
ORGY=828.17
DETECTOR_DISTANCE=110.0
OSCILLATION_RANGE=0.2
X-RAY_WAVELENGTH=0.9531057810578105 !around 12keV
NAME_TEMPLATE_OF_DATA_FRAMES=/sls/mx/data/p11206/raw/20250916/B-12/14/hewl_14_19b301_??????.h5 GENERIC
DATA_RANGE= 1 450
SPOT_RANGE= 1 50
BACKGROUND_RANGE=  11 20
SPACE_GROUP_NUMBER=0
UNIT_CELL_CONSTANTS=0 0 0 0 0 0

!REFINE(IDXREF)=BEAM AXIS ORIENTATION CELL POSITION
!REFINE(INTEGRATE)=POSITION BEAM ORIENTATION !CELL AXIS
!REFINE(CORRECT)=POSITION BEAM ORIENTATION CELL AXIS

SEPMIN= 4.0
CLUSTER_RADIUS= 2
!MAXIMUM_ERROR_OF_SPOT_POSITION= 4.0
!MINIMUM_FRACTION_OF_INDEXED_SPOTS= 0.70

FRIEDEL'S_LAW=FALSE
STRICT_ABSORPTION_CORRECTION=FALSE ! but read Tips_and_Tricks in XDSwiki

! parameters with changes wrt default values:
CORRECTIONS=DECAY MODULATION ABSORP !default value
INCLUDE_RESOLUTION_RANGE=50 0

! parameters specifically for this detector and beamline:
DETECTOR=Dectris PILATUS4 Si 2M NX=1553 NY=1630 QX=0.15000000596046448 QY=0.15000000596046448
SENSOR_THICKNESS=0.45
OVERLOAD=1048500

! Defines path to the Dectris neggia library (to use H5ToXds comment this line out)
LIB=/opt/psi/MX/dectris-neggia/17.09/lib/durin-plugin.so
DIRECTION_OF_DETECTOR_X-AXIS=1 0 0
DIRECTION_OF_DETECTOR_Y-AXIS=0 1 0
INCIDENT_BEAM_DIRECTION=0 0 1
ROTATION_AXIS=1 0 0
FRACTION_OF_POLARIZATION=0.99
POLARIZATION_PLANE_NORMAL=0 1 0
MAXIMUM_NUMBER_OF_JOBS= 8
MAXIMUM_NUMBER_OF_PROCESSORS= 20

Example parameters received from the database

{'beamline': 'X06DA',
 'comments': 'Lysozyme PEG 1000 100µm',
 'created_at': '2025-09-16T14:30:50.851990',
 'data_collection_parameters': {'adpenabled': None,
                                'aperture': None,
                                'autoprocextraparams': None,
                                'autoprocfull': None,
                                'cellparameters': '79 79 38 90 90 90',
                                'chiphiangles': None,
                                'cloud': False,
                                'datacollectiontype': None,
                                'directory': '{sgPuck}/{sgPosition}',
                                'dose': None,
                                'exposure': None,
                                'ffcscampaign': None,
                                'noano': None,
                                'oscillation': None,
                                'pdbid': '',
                                'pdbmodel': None,
                                'processingpipeline': '',
                                'procfull': None,
                                'rescutkey': None,
                                'rescutvalue': None,
                                'spacegroupnumber': 96,
                                'targetresolution': None,
                                'totalrange': None,
                                'transmission': None,
                                'trustedhigh': None,
                                'userresolution': None},
 'experiment_parameters': {'beamSizeHeight': 3.0,
                           'beamSizeWidth': 3.0,
                           'beamline': 'X06DA',
                           'beamlineFluxAtSample_ph_s': 223700674459.08,
                           'characterization': None,
                           'cryojetTemperature_K': 100.0,
                           'detector': {'beamCenterX_px': 781.19,
                                        'beamCenterY_px': 828.17,
                                        'dataset': {'filepath': '20250916/B-12/14/hewl_14_19b301',
                                                    'status': 'written',
                                                    'written_at': '2025-09-16T14:30:28.203275'},
                                        'detectorDistance_mm': 110.0,
                                        'manufacturer': 'DECTRIS',
                                        'model': 'Dectris PILATUS4 Si 2M',
                                        'pixelSizeX_um': 150.00000596046448,
                                        'pixelSizeY_um': 150.00000596046448,
                                        'serialNumber': 'D029955',
                                        'type': 'photon-counting'},
                           'energy_keV': 13.008000000000001,
                           'focusingOptic': 'Kirkpatrick-Baez',
                           'gridScan': None,
                           'humidifierHumidity': None,
                           'humidifierTemperature_K': None,
                           'jet': None,
                           'monochromator': 'Si111',
                           'reprocessing_history': [],
                           'reprocessing_version': None,
                           'ringCurrent_mA': 295.8701108753681,
                           'ringMode': 'Beamline Development',
                           'rotation': {'chi': -5.551726083983046e-05,
                                        'exposureTime_s': 0.02,
                                        'numberOfImages': 900,
                                        'omegaStart_deg': 0.0,
                                        'omegaStep': 0.2,
                                        'phi': -8.960600000000001e-05},
                           'synchrotron': 'Swiss Light Source',
                           'transmission': 1.0,
                           'undulator': None,
                           'undulatorgap_mm': None,
                           'wavelength_Ang': 0.9531057810578105},
 'filepath': '/sls/mx/data/p11206/raw/20250916/B-12/14/hewl_14_19b301',
 'job_id': 1475,
 'pgroup': 'p11206',
 'process_path': '/sls/mx/data/p11206/raw/processing/20250916/B-12/14/hewl_14_19b301',
 'puck_name': 'B-12',
 'reprocessing_history': [],
 'reprocessing_version': None,
 'run_id': 3501,
 'run_number': 6,
 'sample_id': 970,
 'sample_name': 'hewl_14',
 'slurm_id': None,
 'status': 'to_do',
 'type': 'standard',
 'updated_at': None}

Requires Python: >=3.10

Details

PyPI

2025-12-21 11:57:32 +01:00

129 KiB

Assets (2)

aareproc-0.1.16-py3-none-any.whl 62 KiB

aareproc-0.1.16.tar.gz 68 KiB

Versions (22) View all

0.3.0

2026-05-20

0.2.2

2026-05-20

0.2.1

2026-04-30

0.2.0

2026-04-16

0.1.18

2025-12-22

aareproc (0.1.16)

Installation

About this package

AareProc

Description

Service Operation

Deployment

Current Online Processing Folder Structure

Current Features

CLOUD feature was tested on September and temporarily disabled for the rest of September and October

Installation

Usage

Contributing

Authors and acknowledgment

License

Project status

FAQ and Details

XDS.INP template

Example parameters received from the database

Requirements