sf-daq/dap
0
1
Fork 2
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Streak Finder algorithm for CBD experiment #2

Merged
augustin_s merged 46 commits from ext-dorofe_e/dap:chapman into main 2025-07-14 11:18:07 +02:00
Conversation 0 Commits 46 Files Changed 6 +82 -82
2 changed files with 82 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 1d88f33cf3 - Show all commits

90
README.md
View File

@@ -111,33 +111,33 @@ options:
Input parameters:
* `'do_snr': 1/0` - Specifies whether to do whitefield and std correction, if selected - changes image and mask in place.
* `'sf_std_data_file': str` - hdf5 data file containing pre-calculated std.
* `'sf_std_dataset': str` - [optional] dataset containing pre-calculated std, defaults to `'entry/crystallography/std'`.
* `'sf_whitefield_data_file': str` - hdf5 data file containing pre-calculated white field.
* `'sf_whitefield_dataset': str` - [optional] dataset containing pre-calculated white field, defaults to `'entry/crystallography/whitefield'`.
* `'sf_mask_data_file': str` - [optional] hdf5 data file containing pre-calculated mask.
* `'sf_mask_dataset': str` - [optional] dataset containing pre-calculated mask, defaults to `'entry/instrument/detector/mask'`.
* `'sf_scale_whitefield': 1/0` - Specifies whether to scale whitefield to signal, useful if intensity jumps.
* `'cbd_std_data_file': str` - hdf5 data file containing pre-calculated std.
* `'cbd_std_dataset': str` - [optional] dataset containing pre-calculated std, defaults to `'entry/crystallography/std'`.
* `'cbd_whitefield_data_file': str` - hdf5 data file containing pre-calculated white field.
* `'cbd_whitefield_dataset': str` - [optional] dataset containing pre-calculated white field, defaults to `'entry/crystallography/whitefield'`.
* `'cbd_mask_data_file': str` - [optional] hdf5 data file containing pre-calculated mask.
* `'cbd_mask_dataset': str` - [optional] dataset containing pre-calculated mask, defaults to `'entry/instrument/detector/mask'`.
* `'cbd_scale_whitefield': 1/0` - Specifies whether to scale whitefield to signal, useful if intensity jumps.
* `'do_streakfinder_analysis': 1/0` - Specifies whether to execute the streak-finder algorithm.
* `'sf_peak_structure_radius': int` - Connectivity structure radius for *peaks* detection.
* `'sf_peak_structure_rank': int` - Connectivity structure rank for *peaks* detection.
* `'sf_peak_vmin': float` - Peak threshold. All peaks with values lower than ``sf_peak_vmin`` are discarded.
* `'sf_npts': int` - Support size threshold. The support structure is a connected set of pixels which
value is above the threshold ``sf_peak_vmin``. A peak is discarded is the size of support
set is lower than ``sf_npts``.
* `'sf_streak_structure_radius': int` - Connectivity structure radius for *streaks* detection.
* `'sf_streak_structure_rank': int` - Connectivity structure rank for *streaks* detection.
* `'sf_streak_vmin': float` - Streak threshold. All streaks with values lower than ``sf_vmin`` are discarded.
* `'sf_min_size': float` - Minimum number of linelets required in a detected streak.
* `'sf_xtol': float` - Distance threshold. A new linelet is added to a streak if it's distance to the
streak is no more than ``sf_xtol``.
* `'sf_nfa': 1` - Number of false alarms, allowed number of unaligned points in a streak.
* `'sf_num_threads': int` - Number of threads to use for peak finder algorithm.
* `'sf_negative_handler': 'mask'/'zero'/'shift'/''` - [optional] Method to handle negative values in converted frames, defaults to `''` (do not handle).
* `'sf_min_hit_streaks': int` - [optional] Minimum number of discovered streaks to categorize frame as a hit, defaults to 5.
* `'sf_mask_rois': list[(int, int, int, int)]` - [optional] list of `(y_min, y_max, x_min, x_max)` coordinates of ROIs to mask out during peak finding.
* `'sf_crop_roi': [int, int, int, int]` - [optional] run streak finder on a cropped image, e.g. one quadrant, for purpose of spedup.
* `'cbd_peak_structure_radius': int` - Connectivity structure radius for *peaks* detection.
* `'cbd_peak_structure_rank': int` - Connectivity structure rank for *peaks* detection.
* `'cbd_peak_vmin': float` - Peak threshold. All peaks with values lower than ``cbd_peak_vmin`` are discarded.
* `'cbd_npts': int` - Support size threshold. The support structure is a connected set of pixels which
value is above the threshold ``cbd_peak_vmin``. A peak is discarded is the size of support
set is lower than ``cbd_npts``.
* `'cbd_streak_structure_radius': int` - Connectivity structure radius for *streaks* detection.
* `'cbd_streak_structure_rank': int` - Connectivity structure rank for *streaks* detection.
* `'cbd_streak_vmin': float` - Streak threshold. All streaks with values lower than ``cbd_vmin`` are discarded.
* `'cbd_min_size': float` - Minimum number of linelets required in a detected streak.
* `'cbd_xtol': float` - Distance threshold. A new linelet is added to a streak if it's distance to the
streak is no more than ``cbd_xtol``.
* `'cbd_nfa': 1` - Number of false alarms, allowed number of unaligned points in a streak.
* `'cbd_num_threads': int` - Number of threads to use for peak finder algorithm.
* `'cbd_negative_handler': 'mask'/'zero'/'shift'/''` - [optional] Method to handle negative values in converted frames, defaults to `''` (do not handle).
* `'cbd_min_hit_streaks': int` - [optional] Minimum number of discovered streaks to categorize frame as a hit, defaults to 5.
* `'cbd_mask_rois': list[(int, int, int, int)]` - [optional] list of `(y_min, y_max, x_min, x_max)` coordinates of ROIs to mask out during peak finding.
* `'cbd_crop_roi': [int, int, int, int]` - [optional] run streak finder on a cropped image, e.g. one quadrant, for purpose of spedup.
Algorithm Output:
* `'number_of_streaks': int` - Indicates the count of identified streaks.
@@ -318,27 +318,27 @@ Example JSON for Convergent-Beam Diffraction Streak-Finder:
"mask_file": "/sf/jungfrau/config/additional_mask/JF07T32V02.h5",
"mask_dataset": "mask_data",
"do_snr": 0,
"sf_std_data_file": "/sf/instrument/exp/00m_mustermann/res/aux_data/streakfinder_metadata.h5",
"sf_std_dataset": "entry/crystallography/std",
"sf_whitefield_data_file": "/sf/bernina/exp/00m_mustermann/res/aux_data/streakfinder_metadata.h5",
"sf_whitefield_dataset": "entry/crystallography/whitefield",
"sf_mask_data_file": "/sf/bernina/exp/00m_mustermann/res/aux_data/JF_mask.h5",
"sf_mask_dataset": "mask_data",
"sf_scale_whitefield": 0,
"cbd_std_data_file": "/sf/instrument/exp/00m_mustermann/res/aux_data/streakfinder_metadata.h5",
"cbd_std_dataset": "entry/crystallography/std",
"cbd_whitefield_data_file": "/sf/bernina/exp/00m_mustermann/res/aux_data/streakfinder_metadata.h5",
"cbd_whitefield_dataset": "entry/crystallography/whitefield",
"cbd_mask_data_file": "/sf/bernina/exp/00m_mustermann/res/aux_data/JF_mask.h5",
"cbd_mask_dataset": "mask_data",
"cbd_scale_whitefield": 0,
"do_streakfinder_analysis": 1,
"sf_negative_handler": "zero",
"sf_peak_structure_radius": 2,
"sf_peak_structure_rank": 2,
"sf_peak_vmin": 50,
"sf_npts": 10,
"sf_streak_structure_radius": 6,
"sf_streak_structure_rank": 4,
"sf_xtol": 2.0,
"sf_streak_vmin": 30,
"sf_min_size": 25,
"sf_nfa": 1,
"sf_num_threads": 32,
"sf_crop_roi": [0, 2107, 0, 2216]
"cbd_negative_handler": "zero",
"cbd_peak_structure_radius": 2,
"cbd_peak_structure_rank": 2,
"cbd_peak_vmin": 50,
"cbd_npts": 10,
"cbd_streak_structure_radius": 6,
"cbd_streak_structure_rank": 4,
"cbd_xtol": 2.0,
"cbd_streak_vmin": 30,
"cbd_min_size": 25,
"cbd_nfa": 1,
"cbd_num_threads": 32,
"cbd_crop_roi": [0, 2107, 0, 2216]
}
```
# Acknowledgment

74
dap/algos/streakfind.py
View File

@@ -33,7 +33,7 @@ def calc_streakfinder_analysis(results, data, pf_pixel_mask):
if not do_snr and not do_streakfinder_analysis:
return data
negative_val_handler = results.get("sf_negative_handler", "")
negative_val_handler = results.get("cbd_negative_handler", "")
_handle_negative_values(data, pf_pixel_mask, negative_val_handler)
try:
@@ -55,24 +55,24 @@ def calc_streakfinder_analysis(results, data, pf_pixel_mask):
def _generate_cryst_data(results, data, pf_pixel_mask) -> CrystData:
params_required = [
"sf_whitefield_data_file",
"sf_std_data_file",
"sf_scale_whitefield", # Bool
"cbd_whitefield_data_file",
"cbd_std_data_file",
"cbd_scale_whitefield", # Bool
]
if not all(param in results.keys() for param in params_required):
raise ValueError(f"ERROR: Not enough parameters for CBD correction. Skipping\n"
f"{params_required=}")
whitefield_data_file = results["sf_whitefield_data_file"]
std_data_file = results["sf_std_data_file"]
scale_whitefield = results["sf_scale_whitefield"]
whitefield_data_file = results["cbd_whitefield_data_file"]
std_data_file = results["cbd_std_data_file"]
scale_whitefield = results["cbd_scale_whitefield"]
# Using CXI Store specification as default
whitefield_dataset = results.get("sf_whitefield_dataset", "entry/crystallography/whitefield")
std_dataset = results.get("sf_std_dataset", "entry/crystallography/std")
whitefield_dataset = results.get("cbd_whitefield_dataset", "entry/crystallography/whitefield")
std_dataset = results.get("cbd_std_dataset", "entry/crystallography/std")
num_threads = results.get("sf_num_threads", DEFAULT_NUM_THREADS)
num_threads = results.get("cbd_num_threads", DEFAULT_NUM_THREADS)
with h5py.File(whitefield_data_file, "r") as hf:
whitefield = hf[whitefield_dataset][:]
@@ -80,11 +80,11 @@ def _generate_cryst_data(results, data, pf_pixel_mask) -> CrystData:
with h5py.File(std_data_file, "r") as hf:
std = hf[std_dataset][:]
mask_data_file = results.get("sf_mask_data_file", None)
mask_data_file = results.get("cbd_mask_data_file", None)
if mask_data_file is None:
mask = pf_pixel_mask
else:
mask_dataset = results.get("sf_mask_dataset", "entry/instrument/detector/mask")
mask_dataset = results.get("cbd_mask_dataset", "entry/instrument/detector/mask")
with h5py.File(mask_data_file, "r") as hf:
mask = hf[mask_dataset][:].astype(np.bool)
mask *= pf_pixel_mask
@@ -107,17 +107,17 @@ def _calc_streakfinder_analysis(results, cryst_data: CrystData):
return
params_required = [
"sf_peak_structure_radius",
"sf_peak_structure_rank",
"sf_streak_structure_radius",
"sf_streak_structure_rank",
"sf_peak_vmin",
"sf_streak_vmin",
"sf_min_size",
"sf_npts",
"sf_xtol",
"sf_nfa",
"sf_num_threads",
"cbd_peak_structure_radius",
"cbd_peak_structure_rank",
"cbd_streak_structure_radius",
"cbd_streak_structure_rank",
"cbd_peak_vmin",
"cbd_streak_vmin",
"cbd_min_size",
"cbd_npts",
"cbd_xtol",
"cbd_nfa",
"cbd_num_threads",
]
if not all(param in results.keys() for param in params_required):
@@ -125,29 +125,29 @@ def _calc_streakfinder_analysis(results, cryst_data: CrystData):
f"{params_required=}")
return
peak_structure_radius = results["sf_peak_structure_radius"] # peak
peak_structure_rank = results["sf_peak_structure_rank"]
streak_structure_radius = results["sf_streak_structure_radius"] # streak
streak_structure_rank = results["sf_streak_structure_rank"]
peak_vmin = results["sf_peak_vmin"] # peak
streak_vmin = results["sf_streak_vmin"] # streak
min_size = results["sf_min_size"]
npts = results["sf_npts"]
xtol = results["sf_xtol"]
nfa = results["sf_nfa"]
num_threads = results["sf_num_threads"]
peak_structure_radius = results["cbd_peak_structure_radius"] # peak
peak_structure_rank = results["cbd_peak_structure_rank"]
streak_structure_radius = results["cbd_streak_structure_radius"] # streak
streak_structure_rank = results["cbd_streak_structure_rank"]
peak_vmin = results["cbd_peak_vmin"] # peak
streak_vmin = results["cbd_streak_vmin"] # streak
min_size = results["cbd_min_size"]
npts = results["cbd_npts"]
xtol = results["cbd_xtol"]
nfa = results["cbd_nfa"]
num_threads = results["cbd_num_threads"]
min_hit_streaks = results.get("sf_min_hit_streaks", DEFAULT_MIN_HIT_STREAKS)
min_hit_streaks = results.get("cbd_min_hit_streaks", DEFAULT_MIN_HIT_STREAKS)
x_center = results.get("beam_center_x", None)
y_center = results.get("beam_center_y", None)
mask_rois = results.get("sf_mask_rois", []) # list of [y_min, y_max, x_min, x_max]
mask_rois = results.get("cbd_mask_rois", []) # list of [y_min, y_max, x_min, x_max]
for mask_roi in mask_rois:
cryst_data = cryst_data.mask_region(mask_roi)
crop_roi = results.get("sf_crop_roi", None) # [y_min, y_max, x_min, x_max]
crop_roi = results.get("cbd_crop_roi", None) # [y_min, y_max, x_min, x_max]
if crop_roi is not None:
crop_roi_t = [crop_roi[2], crop_roi[3], crop_roi[0], crop_roi[1]]# y0, y1, x0, x1