Files
Jungfraujoch/common/DatasetSettings.cpp
leonarski_fandjungfrau 4dc2534dbf
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
v1.0.0.rc-162 (#72)
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-25 08:21:39 +02:00

519 lines
15 KiB
C++

// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <cmath>
#include "DatasetSettings.h"
#include "Definitions.h"
#include "JFJochException.h"
#include "CheckPath.h"
#define check_max(param, val, max) if ((val) > (max)) throw JFJochException(JFJochExceptionCategory::InputParameterAboveMax, param)
#define check_min(param, val, min) if ((val) < (min)) throw JFJochException(JFJochExceptionCategory::InputParameterBelowMin, param)
#define check_finite(param, val) if (!std::isfinite(val)) throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, param)
DatasetSettings::DatasetSettings() {
photon_energy_keV = WVL_1A_IN_KEV;
detector_distance_mm = 100;
beam_x_pxl = 0.0;
beam_y_pxl = 0.0;
file_prefix = "test";
ntrigger = 1;
images_per_trigger = 1;
compression = CompressionAlgorithm::BSHUF_LZ4;
data_reduction_factor_serialmx = 1.0;
write_nxmx_hdf5_master = true;
spot_finding_enable = true;
poni_rot_1_rad = 0.0f;
poni_rot_2_rad = 0.0f;
poni_rot_3_rad = 0.0f;
max_spot_count = std::min(MAX_SPOT_COUNT, 250);
detect_ice_rings = false;
}
DatasetSettings &DatasetSettings::ImagesPerTrigger(int64_t input) {
check_max("Total number of images", input, 10*1000*1000);
check_min("Total number of images", input, 0);
images_per_trigger = input;
return *this;
}
DatasetSettings &DatasetSettings::NumTriggers(int64_t input) {
check_max("Total number of triggers", input, 10*1000*1000);
check_min("Total number of triggers", input, 1);
ntrigger = input;
return *this;
}
DatasetSettings &DatasetSettings::PhotonEnergy_keV(float input) {
check_finite("Energy (keV)", input);
check_min("Energy (keV)", input, MIN_ENERGY_KEV);
check_max("Energy (keV)", input, MAX_ENERGY_KEV);
photon_energy_keV = input;
return *this;
}
DatasetSettings &DatasetSettings::BeamX_pxl(float input) {
check_finite("Beam center x", input);
beam_x_pxl = input;
return *this;
}
DatasetSettings &DatasetSettings::BeamY_pxl(float input) {
check_finite("Beam center y", input);
beam_y_pxl = input;
return *this;
}
DatasetSettings &DatasetSettings::DetectorDistance_mm(float input) {
check_finite("Detector distance (mm)", input);
check_min("Detector distance (mm)", input, 1);
detector_distance_mm = input;
return *this;
}
DatasetSettings &DatasetSettings::FilePrefix(std::string input) {
CheckPath(input); // multi-user guard: no absolute path, no '..' traversal
return FilePrefixTrusted(std::move(input));
}
DatasetSettings &DatasetSettings::FilePrefixTrusted(std::string input) {
// Offline/local callers (rugnux, the viewer's processing) supply their own output path, so the
// CheckPath guard applied by FilePrefix() is deliberately skipped here - an absolute path is
// allowed. Not reachable from the broker/writer remote-input path.
if ((input.find("_master.h5") == input.length() - 10) && (input.length() > 10))
file_prefix = input.substr(0, input.length() - 10);
else
file_prefix = input;
return *this;
}
DatasetSettings &DatasetSettings::Compression(CompressionAlgorithm input) {
switch (input) {
case CompressionAlgorithm::NO_COMPRESSION:
case CompressionAlgorithm::BSHUF_LZ4:
case CompressionAlgorithm::BSHUF_ZSTD:
case CompressionAlgorithm::BSHUF_ZSTD_RLE:
case CompressionAlgorithm::BSHUF_ZSTD_RLE_HUFF:
compression = input;
break;
default:
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"Invalid value for compression enum parameter");
}
return *this;
}
DatasetSettings &DatasetSettings::SetUnitCell(const std::optional<UnitCell> &cell) {
// If one or more cell parameters are zero - assume cell is not provided
// But if all parameters non-zero - throw exception for non-sense values
if (cell && (cell->a != 0) && (cell->b != 0) && (cell->c != 0)
&& (cell->alpha != 0) && (cell->beta != 0) && (cell->gamma != 0)) {
check_min("Unit cell a", cell->a, 0.1);
check_min("Unit cell b", cell->b, 0.1);
check_min("Unit cell c", cell->c, 0.1);
check_min("Angle alpha", cell->alpha, 5.0);
check_min("Angle beta", cell->beta, 5.0);
check_min("Angle gamma", cell->gamma, 5.0);
check_max("Angle alpha", cell->alpha, 355.0);
check_max("Angle beta", cell->beta, 355.0);
check_max("Angle gamma", cell->gamma, 360);
unit_cell = cell;
} else
unit_cell.reset();
return *this;
}
DatasetSettings &DatasetSettings::SpaceGroupNumber(std::optional<int64_t> input) {
if (input) {
check_min("Space group number", input, 1);
check_max("Space group number", input, 230);
}
space_group_number = input;
return *this;
}
DatasetSettings &DatasetSettings::SampleName(std::string input) {
sample_name = input;
return *this;
}
DatasetSettings &DatasetSettings::AttenuatorTransmission(const std::optional<float> &input) {
if (input) {
check_finite("Attenuator transmission", input.value());
check_max("Attenuator transmission", input.value(), 1.0);
check_min("Attenuator transmission", input.value(), 0.0);
}
attenuator_transmission = input;
return *this;
}
DatasetSettings &DatasetSettings::TotalFlux(const std::optional<float> &input) {
if (input) {
check_finite("Total flux", input.value());
check_min("Total flux", input.value(), 0.0);
}
total_flux = input;
return *this;
}
DatasetSettings &DatasetSettings::Goniometer(const std::optional<GoniometerAxis> &input) {
goniometer = input;
return *this;
}
DatasetSettings &DatasetSettings::HeaderAppendix(const nlohmann::json &input) {
header_appendix = input;
return *this;
}
DatasetSettings &DatasetSettings::ImageAppendix(const nlohmann::json &input) {
image_appendix = input;
return *this;
}
std::optional<float> DatasetSettings::GetAttenuatorTransmission() const {
return attenuator_transmission;
}
std::optional<float> DatasetSettings::GetTotalFlux() const {
return total_flux;
}
const std::optional<GoniometerAxis> &DatasetSettings::GetGoniometer() const {
return goniometer;
}
std::optional<GoniometerAxis> &DatasetSettings::Goniometer() {
return goniometer;
}
const nlohmann::json& DatasetSettings::GetHeaderAppendix() const {
return header_appendix;
}
const nlohmann::json& DatasetSettings::GetImageAppendix() const {
return image_appendix;
}
std::optional<UnitCell> DatasetSettings::GetUnitCell() const {
return unit_cell;
}
std::optional<int64_t> DatasetSettings::GetSpaceGroupNumber() const {
return space_group_number;
}
std::string DatasetSettings::GetSampleName() const {
return sample_name;
}
float DatasetSettings::GetPhotonEnergy_keV() const {
return photon_energy_keV;
}
float DatasetSettings::GetBeamX_pxl() const {
return beam_x_pxl;
}
float DatasetSettings::GetBeamY_pxl() const {
return beam_y_pxl;
}
float DatasetSettings::GetDetectorDistance_mm() const {
return detector_distance_mm;
}
Coord DatasetSettings::GetScatteringVector() const {
return {0, 0, photon_energy_keV / WVL_1A_IN_KEV};
}
std::string DatasetSettings::GetFilePrefix() const {
return file_prefix;
}
CompressionAlgorithm DatasetSettings::GetCompressionAlgorithm() const {
return compression;
}
int64_t DatasetSettings::GetNumTriggers() const {
return ntrigger;
}
int64_t DatasetSettings::GetImageNumPerTrigger() const {
return images_per_trigger;
}
DatasetSettings &DatasetSettings::ImagesPerFile(const std::optional<int64_t> &input) {
if (input.has_value())
check_min("Images per file", input.value(), 1);
images_per_file = input;
return *this;
}
std::optional<int64_t> DatasetSettings::GetImagesPerFile() const {
return images_per_file;
}
DatasetSettings &DatasetSettings::LossyCompressionSerialMX(float input) {
check_min("Data reduction factor for serial MX", input, 0.0);
check_max("Data reduction factor for serial MX", input, 1.0);
data_reduction_factor_serialmx = input;
return *this;
}
float DatasetSettings::GetLossyCompressionSerialMX() const {
return data_reduction_factor_serialmx;
}
DatasetSettings &DatasetSettings::RunNumber(const std::optional<uint64_t> &input) {
if (input) {
check_min("Run number", input, 0);
check_max("Run number", input, INT64_MAX);
}
run_number = input;
return *this;
}
DatasetSettings & DatasetSettings::RunName(const std::optional<std::string> &input) {
if (input && input.value().empty())
run_name = {};
else
run_name = input;
return *this;
}
DatasetSettings &DatasetSettings::ExperimentGroup(const std::string &input) {
group = input;
return *this;
}
std::optional<uint64_t> DatasetSettings::GetRunNumber() const {
return run_number;
}
std::optional<std::string> DatasetSettings::GetRunName() const {
return run_name;
}
std::string DatasetSettings::GetExperimentGroup() const {
return group;
}
std::optional<std::chrono::nanoseconds> DatasetSettings::GetImageTime() const {
return image_time;
}
DatasetSettings &DatasetSettings::ImageTime(const std::optional<std::chrono::nanoseconds> &input) {
if (input && (input.value().count() == 0))
image_time = {};
else
image_time = input;
return *this;
}
DatasetSettings &DatasetSettings::LossyCompressionPoisson(std::optional<int64_t> input) {
if (!input || (input == 0))
compression_poisson_factor = {};
else {
check_min("Poisson compression factor", input.value(), 1);
check_max("Poisson compression factor", input.value(), 16);
compression_poisson_factor = input;
}
return *this;
}
std::optional<int64_t> DatasetSettings::GetLossyCompressionPoisson() const {
return compression_poisson_factor;
}
DatasetSettings &DatasetSettings::PixelValueLowThreshold(const std::optional<int64_t> &input) {
if (!input || (input == 0))
pixel_value_low_threshold = {};
else {
check_min("Pixel value low threshold", input.value(), 0);
check_max("Pixel value low threshold", input.value(), INT24_MAX - 1);
pixel_value_low_threshold = input;
}
return *this;
}
DatasetSettings &DatasetSettings::PixelValueHighThreshold(const std::optional<int64_t> &input) {
if (!input || (input == 0))
pixel_value_high_threshold = {};
else {
check_min("Pixel value high threshold", input.value(), 1);
check_max("Pixel value high threshold", input.value(), INT32_MAX);
pixel_value_high_threshold = input;
}
return *this;
}
std::optional<int64_t> DatasetSettings::GetPixelValueLowThreshold() const {
return pixel_value_low_threshold;
}
std::optional<int64_t> DatasetSettings::GetPixelValueHighThreshold() const {
return pixel_value_high_threshold;
}
bool DatasetSettings::IsWriteNXmxHDF5Master() const {
return write_nxmx_hdf5_master;
}
DatasetSettings &DatasetSettings::WriteNXmxHDF5Master(bool input) {
write_nxmx_hdf5_master = input;
return *this;
}
std::optional<bool> DatasetSettings::IsSaveCalibration() const {
return save_calibration;
}
DatasetSettings &DatasetSettings::SaveCalibration(std::optional<bool> input) {
save_calibration = input;
return *this;
}
DatasetSettings &DatasetSettings::GridScan(const std::optional<GridScanSettings> &input) {
grid_scan = input;
return *this;
}
std::optional<GridScanSettings> &DatasetSettings::GridScan() {
return grid_scan;
}
const std::optional<GridScanSettings> &DatasetSettings::GetGridScan() const {
return grid_scan;
}
std::optional<float> DatasetSettings::GetPolarizationFactor() const {
return polarization_factor;
}
DatasetSettings &DatasetSettings::BandwidthFWHM(const std::optional<float> &input) {
bandwidth_fwhm = input;
return *this;
}
std::optional<float> DatasetSettings::GetBandwidthFWHM() const {
return bandwidth_fwhm;
}
float DatasetSettings::GetPoniRot3_rad() const {
return poni_rot_3_rad;
}
float DatasetSettings::GetPoniRot2_rad() const {
return poni_rot_2_rad;
}
float DatasetSettings::GetPoniRot1_rad() const {
return poni_rot_1_rad;
}
DatasetSettings &DatasetSettings::PoniRot1_rad(float input) {
check_finite("PONI rotation 1 (radians)", input);
poni_rot_1_rad = input;
return *this;
}
DatasetSettings &DatasetSettings::PoniRot2_rad(float input) {
check_finite("PONI rotation 2 (radians)", input);
poni_rot_2_rad = input;
return *this;
}
DatasetSettings &DatasetSettings::PoniRot3_rad(float input) {
check_finite("PONI rotation 3 (radians)", input);
poni_rot_3_rad = input;
return *this;
}
DatasetSettings &DatasetSettings::PolarizationFactor(const std::optional<float> &input) {
if (input.has_value()) {
check_finite("Polarization factor", input.value());
check_min("Polarization factor", input.value(), -1.0);
check_max("Polarization factor", input.value(), 1.0);
}
polarization_factor = input;
return *this;
}
std::optional<float> DatasetSettings::GetRingCurrent_mA() const {
return ring_current_mA;
}
DatasetSettings &DatasetSettings::RingCurrent_mA(const std::optional<float> &input) {
if (input.has_value()) {
check_min("Ring current (mA)", input, 0.0);
}
ring_current_mA = input;
return *this;
}
std::optional<float> DatasetSettings::GetSampleTemperature_K() const {
return sample_temperature_K;
}
DatasetSettings &DatasetSettings::SampleTemperature_K(const std::optional<float> &input) {
if (input.has_value()) {
check_min("Sample temperature (K)", input, 0.0);
check_max("Sample temperature (K)", input, 1000.0);
}
sample_temperature_K = input;
return *this;
}
DatasetSettings &DatasetSettings::SpotFindingEnable(bool input) {
spot_finding_enable = input;
return *this;
}
bool DatasetSettings::IsSpotFindingEnabled() const {
return spot_finding_enable;
}
DatasetSettings &DatasetSettings::MaxSpotCount(int64_t input) {
check_min("Max spot count", input, 10);
check_max("Max spot count", input, MAX_SPOT_COUNT);
max_spot_count = input;
return *this;
}
DatasetSettings & DatasetSettings::DetectIceRings(bool input) {
detect_ice_rings = input;
return *this;
}
bool DatasetSettings::IsDetectIceRings() const {
return detect_ice_rings;
}
DatasetSettings &DatasetSettings::FluorescenceSpectrum(const XrayFluorescenceSpectrum &input) {
fluorescence_spectrum = input;
return *this;
}
const XrayFluorescenceSpectrum & DatasetSettings::GetFluorescenceSpectrum() const {
return fluorescence_spectrum;
}
int64_t DatasetSettings::GetMaxSpotCount() const {
return max_spot_count;
}
std::optional<SmargonPosition> DatasetSettings::GetSmargonPosition() const {
return smargon_position;
}
DatasetSettings &DatasetSettings::Smargon(const std::optional<SmargonPosition> &input) {
smargon_position = input;
return *this;
}