Files
Jungfraujoch/rugnux/RugnuxCalibration.cpp
T
leonarski_fandjungfrau 4dc2534dbf
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
v1.0.0.rc-162 (#72)
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-25 08:21:39 +02:00

141 lines
8.0 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <cmath>
#include <fstream>
#include <spdlog/fmt/fmt.h>
#include "RugnuxCalibration.h"
#include "../common/GitInfo.h"
#include "../common/JFJochMath.h"
#include "../image_analysis/geom_refinement/AssignSpotsToRings.h"
#include "../image_analysis/geom_refinement/RingOptimizer.h"
#include "../image_analysis/geom_refinement/RingsFromProfile.h"
namespace {
// How well the ring points sit on the fitted rings, in a unit a user can judge: the radial distance in
// pixels between where a point is and where the fitted geometry puts its ring. The fit's own residual
// is in q, so it is divided by the local dq/dr - measured by stepping one pixel outward along the radius
// rather than assumed, since dq/dr varies with two-theta and with the tilt.
//
// The beam centre enters a ring's radius as r(phi) = R + dx cos(phi) + dy sin(phi), so fitting it to n
// points of scatter s leaves the textbook var = 2 s^2 / n on each of dx and dy. That is the number that
// separates a beam centre that was measured from one that was merely reported.
CalibrationResult Summarize(const DiffractionGeometry &fitted,
const std::vector<RingOptimizerInput> &points) {
CalibrationResult result;
result.geometry = fitted;
const float cx = fitted.GetBeamX_pxl();
const float cy = fitted.GetBeamY_pxl();
double sum_sq = 0.0;
for (const auto &p : points) {
const float r = std::hypot(p.x - cx, p.y - cy);
if (!(r > 1.0f))
continue;
const float q = fitted.PxlToQ(p.x, p.y);
const float dq_dr = fitted.PxlToQ(p.x + (p.x - cx) / r, p.y + (p.y - cy) / r) - q;
if (!(std::abs(dq_dr) > 0.0f))
continue;
const double dr = (q - p.q_expected) / dq_dr;
sum_sq += dr * dr;
++result.ring_points;
}
if (result.ring_points > 0) {
result.rms_radial_pxl = std::sqrt(sum_sq / static_cast<double>(result.ring_points));
result.beam_sigma_pxl = result.rms_radial_pxl
* std::sqrt(2.0 / static_cast<double>(result.ring_points));
}
return result;
}
} // namespace
CalibrationResult CalibrateFromProfile(const std::vector<float> &profile,
const AzimuthalIntegrationMapping &mapping,
const DiffractionGeometry &geom,
const std::vector<float> &calibrant_ring_q) {
const auto points = RingsFromAzimuthalProfile(profile, mapping, geom, calibrant_ring_q);
if (points.empty())
throw JFJochException(JFJochExceptionCategory::CalibrationError,
"No powder ring found in the summed azimuthal profile");
return Summarize(RingOptimizer(geom).Run(points), points);
}
CalibrationResult CalibrateFromSpots(const std::vector<SpotToSave> &spots,
const DiffractionGeometry &geom,
const std::vector<float> &calibrant_ring_q) {
DiffractionGeometry fitted = geom;
// From scratch (Hough circle centre + ring clustering), then refined: the guess pins the centre to a
// whole pixel and only sees the spots its clustering kept, so the refine re-matches every spot at
// that geometry.
GuessGeometry(fitted, spots, calibrant_ring_q);
OptimizeGeometry(fitted, spots, calibrant_ring_q);
return Summarize(fitted, AssignSpotsToRings(fitted, spots, calibrant_ring_q));
}
void WritePoniFile(const std::string &path, const DiffractionExperiment &experiment,
const DiffractionGeometry &geom) {
std::ofstream f(path);
if (!f)
throw JFJochException(JFJochExceptionCategory::FileWriteError, "Cannot write " + path);
const double pixel_m = geom.GetPixelSize_mm() * 1e-3;
// pyFAI's axis convention is the trap: Poni1 (and pixel1) is the SLOW axis - rows, our y - and
// Poni2 the FAST axis - columns, our x - both in metres from the detector origin. A transposed PONI
// file is silently wrong, so the mapping is spelled out here rather than left to the reader.
//
// DiffractionGeometry's beam_x/beam_y IS the PONI: LabCoord rotates the vector measured FROM that
// pixel, i.e. it is the point of normal incidence, so it maps straight across with no correction.
// GetDirectBeam_pxl() is a different quantity - where the direct beam lands - and parts from the
// PONI as soon as rot1/rot2 are non-zero.
//
// The half pixel is the origin convention (see docs/DETECTOR_GEOMETRY.md): our coordinates are
// pixel-centred, so beam_x = 948 means the CENTRE of pixel 948, while pyFAI measures from the edge
// of the sensor and puts the centre of pixel i at (i + 0.5) * pixel size. Without it the pattern
// pyFAI integrates sits half a pixel off ours.
const double half_pixel_m = 0.5 * pixel_m;
f << fmt::format("# Calibration done by Jungfraujoch rugnux {}\n", jfjoch_version());
// poni_version 2.1 is what pyFAI introduced "orientation" with (pyFAI 2024.01).
f << "poni_version: 2.1\n";
f << "Detector: Detector\n";
// orientation 2 is pyFAI's "origin at the top left of the image when looking FROM the sample",
// which is the MX convention Jungfraujoch assembles to. Without it pyFAI assumes its own default,
// orientation 3 (bottom left), and quietly believes increasing row means physically upwards. The
// radial integration is identical either way - a mirror preserves 2theta - but the azimuth comes
// out with the opposite sense, which matters for anything that uses chi (cake or sector
// integration, texture).
f << fmt::format("Detector_config: {{\"pixel1\": {:g}, \"pixel2\": {:g}, \"max_shape\": [{}, {}], "
"\"orientation\": 2}}\n",
pixel_m, pixel_m, experiment.GetYPixelsNumConv(), experiment.GetXPixelsNumConv());
f << fmt::format("Distance: {:.9g}\n", geom.GetDetectorDistance_mm() * 1e-3);
// Poni1 is measured from pyFAI's own origin, so declaring orientation 2 re-anchors it to the top
// edge: the same physical point is now (height - 1 - beam_y) rows down from there.
f << fmt::format("Poni1: {:.9g}\n",
(experiment.GetYPixelsNumConv() - 1 - geom.GetBeamY_pxl()) * pixel_m + half_pixel_m);
f << fmt::format("Poni2: {:.9g}\n", geom.GetBeamX_pxl() * pixel_m + half_pixel_m);
// With orientation declared, rot2 and rot3 change sign and rot1 does not, and Rot3 carries a
// further half turn:
// (Rot1, Rot2, Rot3) = (+rot1, +rot2, -rot3 + pi)
// A row flip is an improper transformation, so it reverses the sense of rotations about x and
// about the beam while leaving the one about the vertical alone. The half turn is the azimuthal
// reference: pyFAI's in-plane axes are the negatives of ours, so without it every chi comes out
// 180 degrees away. It is a rotation about the beam, so it leaves 2theta untouched - which is
// why radial integration was right all along and only the azimuth was wrong.
// Pinned empirically against pyFAI 2026.5.0 on a tilted detector (4/-6.5/13 deg, off-centre
// beam), against the lab positions of the NXmx chain: 2theta to 3.6e-15 deg and chi to 2.8e-14
// deg over the whole detector. The half turn is needed for the orientation-3 form written before
// this too, so it is not an artefact of declaring the orientation.
f << fmt::format("Rot1: {:.9g}\n", geom.GetPoniRot1_rad());
// negate() rather than a bare minus so an unrefined angle prints as 0 and not -0.
const auto negate = [](float v) { return v == 0.0f ? 0.0f : -v; };
f << fmt::format("Rot2: {:.9g}\n", geom.GetPoniRot2_rad());
f << fmt::format("Rot3: {:.9g}\n", negate(geom.GetPoniRot3_rad()) + PI);
f << fmt::format("Wavelength: {:.9g}\n", geom.GetWavelength_A() * 1e-10);
f.flush();
if (!f)
throw JFJochException(JFJochExceptionCategory::FileWriteError, "Error writing " + path);
}