Files
Jungfraujoch/common/AzimuthalIntegrationMapping.cpp
leonarski_fandjungfrau 4dc2534dbf
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
v1.0.0.rc-162 (#72)
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-25 08:21:39 +02:00

270 lines
10 KiB
C++

// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include "JFJochMath.h"
#include <cmath>
#include <thread>
#include <future>
#include "AzimuthalIntegrationMapping.h"
#include "JFJochException.h"
#include "DiffractionGeometry.h"
#include "RawToConvertedGeometry.h"
#include "TableChecksum.h"
AzimuthalIntegrationMapping::AzimuthalIntegrationMapping(const DiffractionExperiment &experiment,
const PixelMask& mask,
size_t in_nthreads)
: settings(experiment.GetAzimuthalIntegrationSettings()),
wavelength(experiment.GetWavelength_A()),
// The dimensions of the image this mapping is built for: converted when the geometry is
// transformed, raw when it is not. They have to match pixel_to_bin, which is sized per mode
// below - the adaptive spot finders walk the image with these and index pixel_to_bin with it.
width(experiment.GetXPixelsNum()),
height(experiment.GetYPixelsNum()) {
if (width <= 0)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"Detector width must be above 0");
if (height <= 0)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"Detector height must be above 0");
if (settings.GetBinCount() >= UINT16_MAX)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"Cannot handle more than 65534 az. int. bins");
polarization_factor = experiment.GetPolarizationFactor();
if (in_nthreads == 0)
nthreads = std::thread::hardware_concurrency();
else
nthreads = in_nthreads;
nthreads = std::clamp<size_t>(nthreads, 1, 64);
if (!experiment.IsGeometryTransformed())
SetupRawGeom(experiment, mask.GetMaskRaw());
else
SetupConvGeom(experiment.GetDiffractionGeometry(),mask.GetMask());
UpdateMaxBinNumber();
pixel_to_bin_checksum = TableChecksum(pixel_to_bin.data(), pixel_to_bin.size() * sizeof(uint16_t));
corrections_checksum = TableChecksum(corrections.data(), corrections.size() * sizeof(float));
}
void AzimuthalIntegrationMapping::SetupConvGeomRows(const DiffractionGeometry &geom, const std::vector<uint32_t> &mask,
size_t row0, size_t row_end) {
for (size_t row = row0; row < row_end && row < height; row++) {
for (size_t col = 0; col < width; col++)
SetupPixel(geom, mask, row * width + col, col, row);
}
}
void AzimuthalIntegrationMapping::SetupConvGeom(const DiffractionGeometry &geom, const std::vector<uint32_t> &mask) {
pixel_to_bin.resize(width * height, UINT16_MAX);
pixel_resolution.resize(width * height, 0);
corrections.resize(width * height, 0);
if (mask.size() != width * height)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Mask size invalid");
if (nthreads <= 1) {
SetupConvGeomRows(geom, mask, 0, height);
} else {
auto local_nthreads = std::min(nthreads, height);
std::vector<std::future<void>> futures;
futures.reserve(local_nthreads);
for (size_t t = 0; t < local_nthreads; ++t)
futures.emplace_back(std::async(std::launch::async,
&AzimuthalIntegrationMapping::SetupConvGeomRows,
this, std::cref(geom), std::cref(mask),
t * height / local_nthreads,
(t + 1) * height / local_nthreads));
for (auto &f: futures)
f.get();
}
}
void AzimuthalIntegrationMapping::SetupRawGeom(const DiffractionExperiment &experiment,
const std::vector<uint32_t> &mask) {
if (mask.size() != RAW_MODULE_SIZE * experiment.GetModulesNum())
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Mask size invalid");
pixel_to_bin.resize(RAW_MODULE_SIZE * experiment.GetModulesNum(), UINT16_MAX);
pixel_resolution.resize(RAW_MODULE_SIZE * experiment.GetModulesNum(), 0);
corrections.resize(RAW_MODULE_SIZE * experiment.GetModulesNum(), 0);
auto geom = experiment.GetDiffractionGeometry();
if (nthreads <= 1) {
for (int m = 0; m < experiment.GetModulesNum(); m++) {
for (int pxl = 0; pxl < RAW_MODULE_SIZE; pxl++) {
auto [x,y] = RawToConvertedCoordinate(experiment, m, pxl);
SetupPixel(geom, mask, m * RAW_MODULE_SIZE + pxl, x, y);
}
}
} else {
auto local_nthreads = std::min<size_t>(nthreads, experiment.GetModulesNum());
std::vector<std::future<void>> futures;
futures.reserve(local_nthreads);
for (size_t t = 0; t < local_nthreads; ++t) {
const size_t module_begin = t * experiment.GetModulesNum() / local_nthreads;
const size_t module_end = (t + 1) * experiment.GetModulesNum() / local_nthreads;
futures.emplace_back(std::async(std::launch::async, [&, module_begin, module_end] {
for (size_t m = module_begin; m < module_end; ++m) {
for (int pxl = 0; pxl < RAW_MODULE_SIZE; ++pxl) {
auto [x, y] = RawToConvertedCoordinate(experiment, m, pxl);
SetupPixel(geom, mask, m * RAW_MODULE_SIZE + pxl, x, y);
}
}
}));
}
for (auto &f: futures)
f.get();
}
}
void AzimuthalIntegrationMapping::SetupPixel(const DiffractionGeometry &geom,
const std::vector<uint32_t> &mask,
uint32_t pxl, uint32_t col, uint32_t row) {
if (mask[pxl] != 0)
return;
auto x = static_cast<float>(col);
auto y = static_cast<float>(row);
float d = geom.PxlToRes(x, y);
float phi_rad = geom.Phi_rad(x, y);
pixel_resolution[pxl] = d;
float corr = 1.0;
if (settings.IsSolidAngleCorrection())
corr /= geom.CalcAzIntSolidAngleCorr(x, y);
if (settings.IsPolarizationCorrection() && polarization_factor)
corr /= geom.CalcAzIntPolarizationCorr(x, y, polarization_factor.value());
corrections[pxl] = corr;
if (d > 0) {
float q = 2.0f * static_cast<float>(PI) / d;
pixel_to_bin[pxl] = settings.GetBin(q, phi_rad * 180.0 / PI);
}
}
uint16_t AzimuthalIntegrationMapping::GetBinNumber() const {
return settings.GetBinCount();
}
const std::vector<uint16_t> &AzimuthalIntegrationMapping::GetPixelToBin() const {
return pixel_to_bin;
}
const std::vector<float> &AzimuthalIntegrationMapping::GetBinToQ() const {
return bin_to_q;
}
const std::vector<float> &AzimuthalIntegrationMapping::GetBinToD() const {
return bin_to_d;
}
const std::vector<float> &AzimuthalIntegrationMapping::GetBinToTwoTheta() const {
return bin_to_2theta;
}
const std::vector<float> &AzimuthalIntegrationMapping::GetBinToPhi() const {
return bin_to_phi;
}
uint16_t AzimuthalIntegrationMapping::QToBin(float q) const {
return settings.QToBin(q);
}
void AzimuthalIntegrationMapping::UpdateMaxBinNumber() {
bin_to_q.resize(settings.GetBinCount());
bin_to_d.resize(settings.GetBinCount());
bin_to_2theta.resize(settings.GetBinCount());
bin_to_phi.resize(settings.GetBinCount());
for (int j = 0; j < settings.GetAzimuthalBinCount(); j++) {
for (int i = 0; i < settings.GetQBinCount(); i++) {
bin_to_q[j * settings.GetQBinCount() + i] = static_cast<float>(settings.GetQSpacing_recipA() * (i + 0.5) + settings.GetLowQ_recipA());
bin_to_d[j * settings.GetQBinCount() + i] = 2.0f * static_cast<float>(PI) / bin_to_q[j * settings.GetQBinCount() + i];
bin_to_2theta[j * settings.GetQBinCount() + i] = 2.0f * asinf(bin_to_q[i] * wavelength / (4.0f * static_cast<float>(PI))) * 180.0f /
static_cast<float>(PI);
bin_to_phi[j * settings.GetQBinCount() + i] = static_cast<float>(j) * 360.0f / static_cast<float>(settings.GetAzimuthalBinCount());
}
}
}
const std::vector<float> &AzimuthalIntegrationMapping::Corrections() const {
return corrections;
}
const std::vector<float> &AzimuthalIntegrationMapping::Resolution() const {
return pixel_resolution;
}
uint64_t AzimuthalIntegrationMapping::GetPixelToBinChecksum() const {
return pixel_to_bin_checksum;
}
uint64_t AzimuthalIntegrationMapping::GetCorrectionsChecksum() const {
return corrections_checksum;
}
std::shared_ptr<const std::vector<uint32_t>>
AzimuthalIntegrationMapping::ResolutionMaskBits(std::optional<float> high_res,
std::optional<float> low_res) const {
const std::lock_guard lock(res_mask_mutex);
if (res_mask_bits && res_mask_high == high_res && res_mask_low == low_res)
return res_mask_bits;
// An unset limit masks nothing at that end. At the high-resolution end 0 does that on its own - no
// pixel has d < 0, and the detector's own edge is where the pixels stop anyway; at the
// low-resolution end every pixel lies above any finite stand-in, so it takes an infinite one.
const float high = high_res.value_or(0.0f);
const float low = low_res.value_or(INFINITY);
const size_t npixel = pixel_resolution.size();
auto bits = std::make_shared<std::vector<uint32_t>>(npixel / 32 + (npixel % 32 != 0 ? 1 : 0), 0);
for (size_t i = 0; i < npixel; i++)
if (pixel_resolution[i] > low || pixel_resolution[i] < high)
(*bits)[i / 32] |= 1u << (i % 32);
res_mask_high = high_res;
res_mask_low = low_res;
res_mask_bits = bits;
return bits;
}
const AzimuthalIntegrationSettings &AzimuthalIntegrationMapping::Settings() const {
return settings;
}
size_t AzimuthalIntegrationMapping::GetWidth() const {
return width;
}
size_t AzimuthalIntegrationMapping::GetHeight() const {
return height;
}
int32_t AzimuthalIntegrationMapping::GetAzimuthalBinCount() const {
return settings.GetAzimuthalBinCount();
}
int32_t AzimuthalIntegrationMapping::GetQBinCount() const {
return settings.GetQBinCount();
}
size_t AzimuthalIntegrationMapping::GetNThreads() const {
return nthreads;
}