cristallina_analysis_package/examples/SmallExample.ipynb

In [1]:

!pwd

/das/work/units/cristallina/p21147/analysis/analysis_maelo/Cristallina_Intro

In [1]:

%load_ext autoreload
%autoreload 2
%matplotlib widget

import re
from tqdm import tqdm
from collections import defaultdict
from typing import Optional
import os

import numpy as np
import pandas as pd

from matplotlib import pyplot as plt

from scipy import stats


from sfdata import SFDataFiles, sfdatafile, SFScanInfo, SFProcFile


from cristallina.utils import ROI
import cristallina.utils as cu
import cristallina as cr

Intro

Here is a small example of how to manipulate data acquired during a SwissFEL beamtime! This is of course not complete, but it is already a good start, since there is quite a lof of information!

Literature

A lot of what you will need can be found in the github of SFdatafile.

https://github.com/paulscherrerinstitute/sf_datafiles

It is not require that you read all of what is there, but if you want to know something about this library, it is often a good place to start with.

There is also some nice functions that the Cristallina team wrote in order to make the handling of data more convenient. This can be found, when logged in the Cristallina subnetwork, at

/sf/cristallina/applications/cristallina/src/cristallina

You can have a look at the file utils, analysis and plot.

Data structure

The data file are stored in

p21147/raw/

You can open a run file, let's say run0001. You will then find 3 folders:

data, meta and logs.

data is the actual data stored in h5 files. the SFdatafile library mentioned is a way to deal with those h5 files in a convenient way! there is multiple files, each of them contains data that was saved during acquisition. To be short, there are 4 mains files that you will see, with the example taken from

p21147/raw/run0001/data/

1. JF16T03V01, JF stands for Jungfrau. which is the top tier x-ray camera used to record x-ray photons. It has its own file just for that. This is what we want to look at most of the time when doing experiments.
2. BSDATA, BS stands for bull- for beam synchronous. It means that, for every x-ray pulses coming, you have information stored regarding that pulse. Example: pulse energy, photon energy, etc.
3. PVDATA, this record data that is not really synchronous. A good example of what is recording here is motor position of the sample stage, or so, which is usually constant for a run
4. CAMERA, this record all camera except for the Jungfrau. In particular, at Cristallina we have another camera called the x-ray eye.

everything contained in those files is called a channel. For example, the x-ray eye, the sample holder motor positioner and the single.shot photon spectrometer are all channels, even if there are not in the same file.

you can see that in a run folder, the filed name always contain "acqu0001". Now that we know roughly where the data is stored and how it is splitted, we need to understand the whole data structure.

when starting acquisition, we create something called a run (which is the folder I was mentioning before). In one run, there are then different steps, which are splitted by the files present in the run folder. In our case, in

p21147/raw/run0001/data/

we only have acqu0001, so only one step. a step is usually used if we scan a quantity. For example, if I want to do a run where I change the photon energy, I will have one step at 8000 eV, then the next step at 8010 eV, etc.

Finally, for a given step, there is a number of photon pulses that we record. This is decided during the measurement and I will show you later how to get access to this number.

In summary:

a run contains many steps which contains many shots.

Look at a run

Let's see how we can look at a given run using the library cristallina.utils (which can be seen at the path mentioned in the literature section), which is already imported as cu (see first cell). This library can be seen as a wrapper of the SFdatafile library.

Usually, the starting point is to use the function scan_info from this library. Note that this does not load all the data acquired (it can by 10, 100 of Gb, even some Tb due to the number of images recorded from the Jungfrau!). What it does is loading only the shape of the data

In [2]:

run_number = 1

scan = cu.scan_info(run_number)

Something else that we can do is look at the list of all the channels that were recorded during this run. For that, we also use something from the utils class:

In [3]:

cu.channel_names(run_number)

Out [3]:

/sf/cristallina/applications/conda/envs/analysis_forge/lib/python3.10/site-packages/sfdata-0.3.2-py3.11.egg/sfdata/sfdatafile.py:84: UserWarning: Skipping channel "SARFE10-PBPS053:MOTOR_X2" since it caused DatasetNotInGroupError: Cannot get dataset "data" from: <HDF5 group "/SARFE10-PBPS053:MOTOR_X2" (0 members)>
/sf/cristallina/applications/conda/envs/analysis_forge/lib/python3.10/site-packages/sfdata-0.3.2-py3.11.egg/sfdata/sfdatafile.py:84: UserWarning: Skipping channel "SAROP31-OPPI151:MOTOR_X.RBV" since it caused DatasetNotInGroupError: Cannot get dataset "data" from: <HDF5 group "/SAROP31-OPPI151:MOTOR_X.RBV" (0 members)>
/sf/cristallina/applications/conda/envs/analysis_forge/lib/python3.10/site-packages/sfdata-0.3.2-py3.11.egg/sfdata/sfdatafile.py:84: UserWarning: Skipping channel "SAROP31-OPPI151:MOTOR_Y.RBV" since it caused DatasetNotInGroupError: Cannot get dataset "data" from: <HDF5 group "/SAROP31-OPPI151:MOTOR_Y.RBV" (0 members)>

['JF',
 'JF_first_img',
 'JF_info',
 'JF_mean_img',
 'JF_means',
 'JF_std',
 'JF_sum_img',
 'S10CB01-RLLE-DSP:AMPLT-VS',
 'S10CB01-RLLE-DSP:PHASE-VS',
 'S10CB02-RLLE-DSP:AMPLT-VS',
 'S10CB02-RLLE-DSP:PHASE-VS',
 'S10CB03-RLLE-DSP:AMPLT-VS',
 'S10CB03-RLLE-DSP:PHASE-VS',
 'S10CB04-RLLE-DSP:AMPLT-VS',
 'S10CB04-RLLE-DSP:PHASE-VS',
 'S10CB05-RLLE-DSP:AMPLT-VS',
 'S10CB05-RLLE-DSP:PHASE-VS',
 'S10CB06-RLLE-DSP:AMPLT-VS',
 'S10CB06-RLLE-DSP:PHASE-VS',
 'S10CB07-RLLE-DSP:AMPLT-VS',
 'S10CB07-RLLE-DSP:PHASE-VS',
 'S10CB08-RLLE-DSP:AMPLT-VS',
 'S10CB08-RLLE-DSP:PHASE-VS',
 'S10CB09-RLLE-DSP:AMPLT-VS',
 'S10CB09-RLLE-DSP:PHASE-VS',
 'S20CB01-RLLE-DSP:AMPLT-VS',
 'S20CB01-RLLE-DSP:PHASE-VS',
 'S20CB02-RLLE-DSP:AMPLT-VS',
 'S20CB02-RLLE-DSP:PHASE-VS',
 'S20CB03-RLLE-DSP:AMPLT-VS',
 'S20CB03-RLLE-DSP:PHASE-VS',
 'S20CB04-RLLE-DSP:AMPLT-VS',
 'S20CB04-RLLE-DSP:PHASE-VS',
 'S30CB01-RLLE-DSP:AMPLT-VS',
 'S30CB01-RLLE-DSP:PHASE-VS',
 'S30CB02-RLLE-DSP:AMPLT-VS',
 'S30CB02-RLLE-DSP:PHASE-VS',
 'S30CB03-RLLE-DSP:AMPLT-VS',
 'S30CB03-RLLE-DSP:PHASE-VS',
 'S30CB04-RLLE-DSP:AMPLT-VS',
 'S30CB04-RLLE-DSP:PHASE-VS',
 'S30CB05-RLLE-DSP:AMPLT-VS',
 'S30CB05-RLLE-DSP:PHASE-VS',
 'S30CB06-RLLE-DSP:AMPLT-VS',
 'S30CB06-RLLE-DSP:PHASE-VS',
 'S30CB07-RLLE-DSP:AMPLT-VS',
 'S30CB07-RLLE-DSP:PHASE-VS',
 'S30CB08-RLLE-DSP:AMPLT-VS',
 'S30CB08-RLLE-DSP:PHASE-VS',
 'S30CB09-RLLE-DSP:AMPLT-VS',
 'S30CB09-RLLE-DSP:PHASE-VS',
 'S30CB10-RLLE-DSP:AMPLT-VS',
 'S30CB10-RLLE-DSP:PHASE-VS',
 'S30CB11-RLLE-DSP:AMPLT-VS',
 'S30CB11-RLLE-DSP:PHASE-VS',
 'S30CB12-RLLE-DSP:AMPLT-VS',
 'S30CB12-RLLE-DSP:PHASE-VS',
 'S30CB13-RLLE-DSP:AMPLT-VS',
 'S30CB13-RLLE-DSP:PHASE-VS',
 'S30CB14-RLLE-DSP:AMPLT-VS',
 'S30CB14-RLLE-DSP:PHASE-VS',
 'SARES30-CAMS156-XE:intensity',
 'SARFE10-PBIG050-EVR0:CALCI',
 'SARFE10-PBPG050:HAMP-INTENSITY-CAL',
 'SARFE10-PBPG050:PHOTON-ENERGY-PER-PULSE-AVG',
 'SARFE10-PBPS053:INTENSITY',
 'SARFE10-PBPS053:XPOS',
 'SARFE10-PBPS053:YPOS',
 'SARFE10-PSSS059:FIT-COM',
 'SARFE10-PSSS059:FIT-FWHM',
 'SARFE10-PSSS059:FIT-RES',
 'SARFE10-PSSS059:FIT-RMS',
 'SARFE10-PSSS059:SPECT-COM',
 'SARFE10-PSSS059:SPECT-RES',
 'SARFE10-PSSS059:SPECT-RMS',
 'SARFE10-PSSS059:SPECTRUM_X',
 'SARFE10-PSSS059:SPECTRUM_Y',
 'SARFE10-PSSS059:SPECTRUM_Y_SUM',
 'SARFE10-PSSS059:processing_parameters',
 'SINDI01-RLLE-DSP:AMPLT-VS',
 'SINDI01-RLLE-DSP:PHASE-VS',
 'SINEG01-RLLE-DSP:AMPLT-VS',
 'SINSB01-RLLE-DSP:AMPLT-VS',
 'SINSB01-RLLE-DSP:PHASE-VS',
 'SINSB02-RLLE-DSP:AMPLT-VS',
 'SINSB02-RLLE-DSP:PHASE-VS',
 'SINSB03-RLLE-DSP:AMPLT-VS',
 'SINSB03-RLLE-DSP:PHASE-VS',
 'SINSB04-RLLE-DSP:AMPLT-VS',
 'SINSB04-RLLE-DSP:PHASE-VS',
 'SINXB01-RLLE-DSP:AMPLT-VS',
 'SINXB01-RLLE-DSP:PHASE-VS',
 'SARES30-CAMS156-XE:FPICTURE',
 'JF16T03V01',
 'SARCL02-MBND100:P-READ',
 'SARES30-MCS2750:MOT_1.RBV',
 'SARES30-MCS2750:MOT_2.RBV',
 'SARES30-MCS2750:MOT_3.RBV',
 'SARFE10-OATT053:ENERGY',
 'SARFE10-OATT053:MOTOR_1.RBV',
 'SARFE10-OATT053:MOTOR_2.RBV',
 'SARFE10-OATT053:MOTOR_3.RBV',
 'SARFE10-OATT053:MOTOR_4.RBV',
 'SARFE10-OATT053:MOTOR_5.RBV',
 'SARFE10-OATT053:MOTOR_6.RBV',
 'SARFE10-OATT053:TRANS_RB',
 'SARFE10-OATT053:TRANS_SP',
 'SARFE10-PBPG050:PHOTON-ENERGY-PER-PULSE-DS',
 'SARFE10-PBPG050:PHOTON-ENERGY-PER-PULSE-US',
 'SARFE10-PBPS053:MOTOR_PROBE',
 'SARFE10-PBPS053:MOTOR_X1',
 'SAROP31-OAPU107:MOTOR_X.RBV',
 'SAROP31-OAPU107:MOTOR_Y.RBV',
 'SAROP31-OAPU149:MOTOR_X.RBV',
 'SAROP31-OAPU149:MOTOR_Y.RBV',
 'SAROP31-OATA150:MOTOR_1.RBV',
 'SAROP31-OATA150:MOTOR_2.RBV',
 'SAROP31-OATA150:MOTOR_3.RBV',
 'SAROP31-OATA150:MOTOR_4.RBV',
 'SAROP31-OATA150:MOTOR_5.RBV',
 'SAROP31-OATA150:MOTOR_6.RBV',
 'SAROP31-OATA150:TRANS_SP',
 'SAROP31-ODMV152:BD.RBV',
 'SAROP31-ODMV152:BU.RBV',
 'SAROP31-ODMV152:TX.RBV',
 'SAROP31-ODMV152:VS1.RBV',
 'SAROP31-ODMV152:VS2.RBV',
 'SAROP31-ODMV152:VS3.RBV',
 'SAROP31-ODMV152:W_RX.RBV',
 'SAROP31-ODMV152:W_RZ.RBV',
 'SAROP31-ODMV152:W_X.RBV',
 'SAROP31-ODMV152:W_Y.RBV',
 'SAROP31-OKBH154:BD.RBV',
 'SAROP31-OKBH154:BU.RBV',
 'SAROP31-OKBH154:TX2.RBV',
 'SAROP31-OKBH154:TY1.RBV',
 'SAROP31-OKBH154:TY2.RBV',
 'SAROP31-OKBH154:TY3.RBV',
 'SAROP31-OKBH154:W_RX.RBV',
 'SAROP31-OKBH154:W_RY.RBV',
 'SAROP31-OKBH154:W_RZ.RBV',
 'SAROP31-OKBH154:W_X.RBV',
 'SAROP31-OKBH154:W_Y.RBV',
 'SAROP31-OKBV153:BD.RBV',
 'SAROP31-OKBV153:BU.RBV',
 'SAROP31-OKBV153:TX1.RBV',
 'SAROP31-OKBV153:TX2.RBV',
 'SAROP31-OKBV153:TY1.RBV',
 'SAROP31-OKBV153:TY2.RBV',
 'SAROP31-OKBV153:TY3.RBV',
 'SAROP31-OKBV153:W_RX.RBV',
 'SAROP31-OKBV153:W_RY.RBV',
 'SAROP31-OKBV153:W_RZ.RBV',
 'SAROP31-OKBV153:W_X.RBV',
 'SAROP31-OKBV153:W_Y.RBV',
 'SAROP31-OLAS147:MOTOR_1.RBV',
 'SAROP31-OOMH067:BD.RBV',
 'SAROP31-OOMH067:BU.RBV',
 'SAROP31-OOMH067:RY.RBV',
 'SAROP31-OOMH067:TX.RBV',
 'SAROP31-OOMH067:VS1.RBV',
 'SAROP31-OOMH067:VS2.RBV',
 'SAROP31-OOMH067:VS3.RBV',
 'SAROP31-OOMH067:W_RX.RBV',
 'SAROP31-OOMH067:W_RY.RBV',
 'SAROP31-OOMH067:W_RZ.RBV',
 'SAROP31-OOMH067:W_X.RBV',
 'SAROP31-OOMH067:W_Y.RBV',
 'SAROP31-OOMH084:W_RX.RBV',
 'SAROP31-OOMH084:W_RY.RBV',
 'SAROP31-OOMH084:W_RZ.RBV',
 'SAROP31-OOMH084:W_X.RBV',
 'SAROP31-OOMH084:W_Y.RBV',
 'SAROP31-PBPS113:MOTOR_PROBE.RBV',
 'SAROP31-PBPS113:MOTOR_X1.RBV',
 'SAROP31-PBPS113:MOTOR_Y1.RBV',
 'SAROP31-PPRM085:MOTOR_PROBE.RBV',
 'SAROP31-PPRM113:MOTOR_PROBE.RBV',
 'SAROP31-PSCR068:MOTOR_PROBE.RBV',
 'SARUN03-UIND030:K_SET.VAL',
 'SARUN04-UIND030:K_SET.VAL',
 'SARUN05-UIND030:K_SET.VAL',
 'SARUN06-UIND030:K_SET.VAL',
 'SARUN07-UIND030:K_SET.VAL',
 'SARUN08-UIND030:K_SET.VAL',
 'SARUN09-UIND030:K_SET.VAL',
 'SARUN10-UIND030:K_SET.VAL',
 'SARUN11-UIND030:K_SET.VAL',
 'SARUN12-UIND030:K_SET.VAL',
 'SARUN13-UIND030:K_SET.VAL',
 'SARUN14-UIND030:K_SET.VAL',
 'SARUN15-UIND030:K_SET.VAL',
 'SARUN:FELPHOTENE.VAL']

A lot of channels isnt it! You do not need to learn of all them by heart, there is important ones that you need to know. If you are unsure what channel is what, ask the Cristallina team ehe they will be happy to help you!

But we want some data!! How do we access to the actual data?

Here, it is important to remember the datastructure I told you (one run, many steps, many shot per step). Now that we used scan_info() to pre-load run 1, we need to say which step we want to look at. For that, we first need to know how many steps were recorded for run 1:

In [23]:

scan.readbacks

Out [23]:

array([0])

It seems that there is only one readback, so only one step. The readbacks are what we are scanning. For example, if we were scanning the photon energy, the array would contain every energy that were scanned, and one given energy would correspond to one step. In our case, we were not scanning anything so the readback is just the step number (its the first step but python start counting from 0). We then dont have choice but to look at step one, and that so:

scan[0]

Now that we specified which run, which step, we need to look at the? shots? YESSSS. Each x-ray pulse a its own ID. The first and the last pulse IDs of the run can be seen there:

In [39]:

scan.info['pulseIds']

Out [39]:

[[18027050018, 18027051018]]

So the number of pulses is given by:

In [41]:

scan.info['pulseIds'][0][1] - scan.info['pulseIds'][0][0]

Out [41]:

1000

So there is 1000 pulses that were recorded for run 1, step 1. Finally, we want to look at actual data. To do so, we need to decide what channel we want to look at. I will make an example with 3 different channels:

JF for Jungfrau, a 2D image per pulse
PSSS for photon single-shot spectrometer, a 1D spectrum per shot
GASMONITOR, a single value per shot.

In [6]:

JF = 'JF16T03V01'
PSSS_Y = 'SARFE10-PSSS059:SPECTRUM_Y'
GASMONITOR =  'SARFE10-PBIG050-EVR0:CALCI'

To access the data, we can then do:

In [9]:

scan[0][JF].data[0]

Out [9]:

array([[ 1.9400723e-01,  3.8746792e-01, -3.2162413e-01, ...,
        -3.0771542e-02, -2.2246270e-01, -1.5886860e-01],
       [ 5.9779334e-01, -1.2918156e-01,  1.7123930e-02, ...,
         4.7929460e-01, -1.1291032e-01, -1.5057071e-02],
       [ 2.4136053e-01,  5.2310520e-01,  3.5945097e-01, ...,
         4.5555338e-01,  4.4841704e-01,  1.1548842e-01],
       ...,
       [ 2.9501557e-01,  1.4901014e-01,  7.4720460e-01, ...,
         0.0000000e+00,  0.0000000e+00,  7.7488422e+02],
       [ 7.0479155e-01,  5.7560408e-01,  5.9870780e-01, ...,
         0.0000000e+00,  0.0000000e+00,  0.0000000e+00],
       [ 9.4347262e-01, -8.1681706e-02,  5.5027962e-01, ...,
         0.0000000e+00,  0.0000000e+00,  0.0000000e+00]], dtype=float32)

This actually shows the image corresponding to the first pulse, since I added [0] at the end of it (because I did not want to look at all the images and make the notebook crash). To see how many JF images were recorded in run 1, step 1, we can do

In [10]:

scan[0][JF].data.shape

Out [10]:

(1001, 1554, 1030)

The first number is always the number of images recorded (1001, so there is somehow one more image than the number of pulses we wanted to record. Better that than the contrary ehehe)

1554x1030 is the pixel grid of the detector. Let's look at any pulse we want:

In [14]:

pulse_number = 100

fig, ax = plt.subplots(1,1, constrained_layout=True)

ax.imshow(scan[0][JF].data[pulse_number],vmin=0,vmax=50)

Out [14]:

<matplotlib.image.AxesImage at 0x2ac0f311fa30>

Look like there is some kind of diffraction pattern, and somehting was put in front of the detector (this is actually the beamstop, to make sure only the diffracted beam hit the detector, and the direct beam is block to avoid any damages)

Let's do the same for the two other channels and look at all of them together to compare:

In [22]:

pulse_number = 100

fig, ax = plt.subplots(1,3, figsize=(10,5),constrained_layout=True)

ax[0].imshow(scan[0][JF].data[pulse_number],vmin=0,vmax=50)

ax[1].plot(scan[0][PSSS_Y].data[pulse_number])
ax[1].set_xlabel('Energy [eV]')
ax[1].set_ylabel('Amplitude [a.u.]')

ax[2].plot(scan[0][GASMONITOR].data[pulse_number],marker='x',markersize=10, color='red')

Out [22]:

[<matplotlib.lines.Line2D at 0x2ac0faebed40>]

So the image on the left is the JF, which is looking at the pulse after interacting with the sample

The image in the middle is how the x-ray pulse looks like in the energy domain (and so it gives the photon energy)

The last one should be the pulse energy I think.

In [ ]: