Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
270 lines
10 KiB
C++
270 lines
10 KiB
C++
// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include "JFJochMath.h"
|
|
#include <cmath>
|
|
#include <thread>
|
|
#include <future>
|
|
|
|
#include "AzimuthalIntegrationMapping.h"
|
|
#include "JFJochException.h"
|
|
#include "DiffractionGeometry.h"
|
|
#include "RawToConvertedGeometry.h"
|
|
#include "TableChecksum.h"
|
|
|
|
AzimuthalIntegrationMapping::AzimuthalIntegrationMapping(const DiffractionExperiment &experiment,
|
|
const PixelMask& mask,
|
|
size_t in_nthreads)
|
|
: settings(experiment.GetAzimuthalIntegrationSettings()),
|
|
wavelength(experiment.GetWavelength_A()),
|
|
// The dimensions of the image this mapping is built for: converted when the geometry is
|
|
// transformed, raw when it is not. They have to match pixel_to_bin, which is sized per mode
|
|
// below - the adaptive spot finders walk the image with these and index pixel_to_bin with it.
|
|
width(experiment.GetXPixelsNum()),
|
|
height(experiment.GetYPixelsNum()) {
|
|
|
|
if (width <= 0)
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
|
|
"Detector width must be above 0");
|
|
|
|
if (height <= 0)
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
|
|
"Detector height must be above 0");
|
|
|
|
if (settings.GetBinCount() >= UINT16_MAX)
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
|
|
"Cannot handle more than 65534 az. int. bins");
|
|
|
|
polarization_factor = experiment.GetPolarizationFactor();
|
|
|
|
if (in_nthreads == 0)
|
|
nthreads = std::thread::hardware_concurrency();
|
|
else
|
|
nthreads = in_nthreads;
|
|
|
|
nthreads = std::clamp<size_t>(nthreads, 1, 64);
|
|
|
|
if (!experiment.IsGeometryTransformed())
|
|
SetupRawGeom(experiment, mask.GetMaskRaw());
|
|
else
|
|
SetupConvGeom(experiment.GetDiffractionGeometry(),mask.GetMask());
|
|
|
|
UpdateMaxBinNumber();
|
|
|
|
pixel_to_bin_checksum = TableChecksum(pixel_to_bin.data(), pixel_to_bin.size() * sizeof(uint16_t));
|
|
corrections_checksum = TableChecksum(corrections.data(), corrections.size() * sizeof(float));
|
|
}
|
|
|
|
void AzimuthalIntegrationMapping::SetupConvGeomRows(const DiffractionGeometry &geom, const std::vector<uint32_t> &mask,
|
|
size_t row0, size_t row_end) {
|
|
for (size_t row = row0; row < row_end && row < height; row++) {
|
|
for (size_t col = 0; col < width; col++)
|
|
SetupPixel(geom, mask, row * width + col, col, row);
|
|
}
|
|
}
|
|
|
|
void AzimuthalIntegrationMapping::SetupConvGeom(const DiffractionGeometry &geom, const std::vector<uint32_t> &mask) {
|
|
pixel_to_bin.resize(width * height, UINT16_MAX);
|
|
pixel_resolution.resize(width * height, 0);
|
|
corrections.resize(width * height, 0);
|
|
|
|
if (mask.size() != width * height)
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Mask size invalid");
|
|
|
|
if (nthreads <= 1) {
|
|
SetupConvGeomRows(geom, mask, 0, height);
|
|
} else {
|
|
auto local_nthreads = std::min(nthreads, height);
|
|
std::vector<std::future<void>> futures;
|
|
futures.reserve(local_nthreads);
|
|
|
|
for (size_t t = 0; t < local_nthreads; ++t)
|
|
futures.emplace_back(std::async(std::launch::async,
|
|
&AzimuthalIntegrationMapping::SetupConvGeomRows,
|
|
this, std::cref(geom), std::cref(mask),
|
|
t * height / local_nthreads,
|
|
(t + 1) * height / local_nthreads));
|
|
|
|
for (auto &f: futures)
|
|
f.get();
|
|
}
|
|
}
|
|
|
|
void AzimuthalIntegrationMapping::SetupRawGeom(const DiffractionExperiment &experiment,
|
|
const std::vector<uint32_t> &mask) {
|
|
if (mask.size() != RAW_MODULE_SIZE * experiment.GetModulesNum())
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Mask size invalid");
|
|
|
|
pixel_to_bin.resize(RAW_MODULE_SIZE * experiment.GetModulesNum(), UINT16_MAX);
|
|
pixel_resolution.resize(RAW_MODULE_SIZE * experiment.GetModulesNum(), 0);
|
|
corrections.resize(RAW_MODULE_SIZE * experiment.GetModulesNum(), 0);
|
|
|
|
auto geom = experiment.GetDiffractionGeometry();
|
|
|
|
if (nthreads <= 1) {
|
|
for (int m = 0; m < experiment.GetModulesNum(); m++) {
|
|
for (int pxl = 0; pxl < RAW_MODULE_SIZE; pxl++) {
|
|
auto [x,y] = RawToConvertedCoordinate(experiment, m, pxl);
|
|
SetupPixel(geom, mask, m * RAW_MODULE_SIZE + pxl, x, y);
|
|
}
|
|
}
|
|
} else {
|
|
auto local_nthreads = std::min<size_t>(nthreads, experiment.GetModulesNum());
|
|
std::vector<std::future<void>> futures;
|
|
futures.reserve(local_nthreads);
|
|
|
|
for (size_t t = 0; t < local_nthreads; ++t) {
|
|
const size_t module_begin = t * experiment.GetModulesNum() / local_nthreads;
|
|
const size_t module_end = (t + 1) * experiment.GetModulesNum() / local_nthreads;
|
|
|
|
futures.emplace_back(std::async(std::launch::async, [&, module_begin, module_end] {
|
|
for (size_t m = module_begin; m < module_end; ++m) {
|
|
for (int pxl = 0; pxl < RAW_MODULE_SIZE; ++pxl) {
|
|
auto [x, y] = RawToConvertedCoordinate(experiment, m, pxl);
|
|
SetupPixel(geom, mask, m * RAW_MODULE_SIZE + pxl, x, y);
|
|
}
|
|
}
|
|
}));
|
|
}
|
|
|
|
for (auto &f: futures)
|
|
f.get();
|
|
}
|
|
}
|
|
|
|
void AzimuthalIntegrationMapping::SetupPixel(const DiffractionGeometry &geom,
|
|
const std::vector<uint32_t> &mask,
|
|
uint32_t pxl, uint32_t col, uint32_t row) {
|
|
if (mask[pxl] != 0)
|
|
return;
|
|
|
|
auto x = static_cast<float>(col);
|
|
auto y = static_cast<float>(row);
|
|
float d = geom.PxlToRes(x, y);
|
|
float phi_rad = geom.Phi_rad(x, y);
|
|
|
|
pixel_resolution[pxl] = d;
|
|
|
|
float corr = 1.0;
|
|
if (settings.IsSolidAngleCorrection())
|
|
corr /= geom.CalcAzIntSolidAngleCorr(x, y);
|
|
if (settings.IsPolarizationCorrection() && polarization_factor)
|
|
corr /= geom.CalcAzIntPolarizationCorr(x, y, polarization_factor.value());
|
|
corrections[pxl] = corr;
|
|
|
|
if (d > 0) {
|
|
float q = 2.0f * static_cast<float>(PI) / d;
|
|
pixel_to_bin[pxl] = settings.GetBin(q, phi_rad * 180.0 / PI);
|
|
}
|
|
}
|
|
|
|
uint16_t AzimuthalIntegrationMapping::GetBinNumber() const {
|
|
return settings.GetBinCount();
|
|
}
|
|
|
|
const std::vector<uint16_t> &AzimuthalIntegrationMapping::GetPixelToBin() const {
|
|
return pixel_to_bin;
|
|
}
|
|
|
|
const std::vector<float> &AzimuthalIntegrationMapping::GetBinToQ() const {
|
|
return bin_to_q;
|
|
}
|
|
|
|
const std::vector<float> &AzimuthalIntegrationMapping::GetBinToD() const {
|
|
return bin_to_d;
|
|
}
|
|
|
|
const std::vector<float> &AzimuthalIntegrationMapping::GetBinToTwoTheta() const {
|
|
return bin_to_2theta;
|
|
}
|
|
|
|
const std::vector<float> &AzimuthalIntegrationMapping::GetBinToPhi() const {
|
|
return bin_to_phi;
|
|
}
|
|
|
|
uint16_t AzimuthalIntegrationMapping::QToBin(float q) const {
|
|
return settings.QToBin(q);
|
|
}
|
|
|
|
void AzimuthalIntegrationMapping::UpdateMaxBinNumber() {
|
|
bin_to_q.resize(settings.GetBinCount());
|
|
bin_to_d.resize(settings.GetBinCount());
|
|
bin_to_2theta.resize(settings.GetBinCount());
|
|
bin_to_phi.resize(settings.GetBinCount());
|
|
|
|
for (int j = 0; j < settings.GetAzimuthalBinCount(); j++) {
|
|
for (int i = 0; i < settings.GetQBinCount(); i++) {
|
|
bin_to_q[j * settings.GetQBinCount() + i] = static_cast<float>(settings.GetQSpacing_recipA() * (i + 0.5) + settings.GetLowQ_recipA());
|
|
bin_to_d[j * settings.GetQBinCount() + i] = 2.0f * static_cast<float>(PI) / bin_to_q[j * settings.GetQBinCount() + i];
|
|
bin_to_2theta[j * settings.GetQBinCount() + i] = 2.0f * asinf(bin_to_q[i] * wavelength / (4.0f * static_cast<float>(PI))) * 180.0f /
|
|
static_cast<float>(PI);
|
|
bin_to_phi[j * settings.GetQBinCount() + i] = static_cast<float>(j) * 360.0f / static_cast<float>(settings.GetAzimuthalBinCount());
|
|
}
|
|
}
|
|
}
|
|
|
|
const std::vector<float> &AzimuthalIntegrationMapping::Corrections() const {
|
|
return corrections;
|
|
}
|
|
|
|
const std::vector<float> &AzimuthalIntegrationMapping::Resolution() const {
|
|
return pixel_resolution;
|
|
}
|
|
|
|
uint64_t AzimuthalIntegrationMapping::GetPixelToBinChecksum() const {
|
|
return pixel_to_bin_checksum;
|
|
}
|
|
|
|
uint64_t AzimuthalIntegrationMapping::GetCorrectionsChecksum() const {
|
|
return corrections_checksum;
|
|
}
|
|
|
|
std::shared_ptr<const std::vector<uint32_t>>
|
|
AzimuthalIntegrationMapping::ResolutionMaskBits(std::optional<float> high_res,
|
|
std::optional<float> low_res) const {
|
|
const std::lock_guard lock(res_mask_mutex);
|
|
if (res_mask_bits && res_mask_high == high_res && res_mask_low == low_res)
|
|
return res_mask_bits;
|
|
|
|
// An unset limit masks nothing at that end. At the high-resolution end 0 does that on its own - no
|
|
// pixel has d < 0, and the detector's own edge is where the pixels stop anyway; at the
|
|
// low-resolution end every pixel lies above any finite stand-in, so it takes an infinite one.
|
|
const float high = high_res.value_or(0.0f);
|
|
const float low = low_res.value_or(INFINITY);
|
|
|
|
const size_t npixel = pixel_resolution.size();
|
|
auto bits = std::make_shared<std::vector<uint32_t>>(npixel / 32 + (npixel % 32 != 0 ? 1 : 0), 0);
|
|
for (size_t i = 0; i < npixel; i++)
|
|
if (pixel_resolution[i] > low || pixel_resolution[i] < high)
|
|
(*bits)[i / 32] |= 1u << (i % 32);
|
|
|
|
res_mask_high = high_res;
|
|
res_mask_low = low_res;
|
|
res_mask_bits = bits;
|
|
return bits;
|
|
}
|
|
|
|
const AzimuthalIntegrationSettings &AzimuthalIntegrationMapping::Settings() const {
|
|
return settings;
|
|
}
|
|
|
|
size_t AzimuthalIntegrationMapping::GetWidth() const {
|
|
return width;
|
|
}
|
|
|
|
size_t AzimuthalIntegrationMapping::GetHeight() const {
|
|
return height;
|
|
}
|
|
|
|
int32_t AzimuthalIntegrationMapping::GetAzimuthalBinCount() const {
|
|
return settings.GetAzimuthalBinCount();
|
|
}
|
|
|
|
int32_t AzimuthalIntegrationMapping::GetQBinCount() const {
|
|
return settings.GetQBinCount();
|
|
}
|
|
|
|
size_t AzimuthalIntegrationMapping::GetNThreads() const {
|
|
return nthreads;
|
|
}
|