Files
Jungfraujoch/image_analysis/lattice_search/LatticeSearch.cpp
T
leonarski_fandjungfrau 4dc2534dbf
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
v1.0.0.rc-162 (#72)
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-25 08:21:39 +02:00

419 lines
15 KiB
C++

// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include "../../common/JFJochMath.h"
#include "LatticeSearch.h"
#include <gemmi/cellred.hpp>
#include <cmath>
#include <optional>
// How close the reduced beta has to be to 90 degrees for the two Niggli types to be genuinely
// interchangeable (see the retry at the end of LatticeSearch). Not the angle tolerance: that is how
// far a metric may sit from an ideal one and still be called it, which is far too generous here - a
// cell 2 degrees off the boundary is a real type-1 cell, and presenting it in the obtuse setting
// promotes a general triclinic lattice to C-centred monoclinic on residuals of ~2 degrees. Measured:
// the crystal this was found on sits 0.07 degrees from the boundary and matches on 0.006 to 0.135;
// the triclinic cell that must not be promoted sits 2.0 degrees from it.
constexpr double NIGGLI_TYPE_BOUNDARY_DEG = 0.5;
struct NiggliClass {
int number;
int type;
bool cond_AB;
bool cond_BC;
double cond_D;
double cond_E;
double cond_F;
bool cond_DEF;
bool cond_2DF;
gemmi::Mat33 reindex;
gemmi::CrystalSystem system;
char centering;
};
namespace {
// The body of both entry points. only_class, when given, keeps just the characters of that Bravais
// class - see LatticeSearchForClass. With no filter this is the original walk unchanged, and the
// triclinic character fits every metric, so it always returns a result.
std::optional<LatticeSearchResult> SearchCharacters(const CrystalLattice &L, double dist_tolerance,
double angle_tolerance,
const std::pair<gemmi::CrystalSystem, char> *only_class) {
UnitCell uc = L.GetUnitCell();
gemmi::UnitCell g_uc(uc.a, uc.b, uc.c, uc.alpha, uc.beta, uc.gamma);
gemmi::GruberVector g_vec(g_uc, 'P', true);
g_vec.niggli_reduce();
CrystalLattice L_niggli = L;
if (g_vec.change_of_basis)
L_niggli = L.Multiply(gemmi::rot_as_mat33(g_vec.change_of_basis->rot).transpose());
double A = g_vec.A;
double B = g_vec.B;
double C = g_vec.C;
double D = g_vec.xi / 2;
double E = g_vec.eta / 2;
double F = g_vec.zeta / 2;
// D, E, F are parameters so the table can also be built for the type-flipped setting below.
auto make_classes = [&](double D, double E, double F) {
return std::vector<NiggliClass>{
{
1, 1,
true, true, A / 2, A / 2, A / 2, false, false,
gemmi::Mat33{1, -1, 1, 1, 1, -1, -1, 1, 1},
gemmi::CrystalSystem::Cubic, 'F'
},
{
2, 1,
true, true, D, D, D, false, false,
{1, -1, 0, -1, 0, 1, -1, -1, -1},
gemmi::CrystalSystem::Trigonal, 'R'
},
{
3, 2,
true, true, 0, 0, 0, false, false,
gemmi::Mat33{1, 0, 0, 0, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Cubic, 'P'
},
{
5, 2,
true, true, -A / 3, -A / 3, -A / 3, false, false,
gemmi::Mat33{1, 0, 1, 1, 1, 0, 0, 1, 1},
gemmi::CrystalSystem::Cubic, 'I'
},
{
4, 2,
true, true, D, D, D, false, false,
{1, -1, 0, -1, 0, 1, -1, -1, -1},
gemmi::CrystalSystem::Trigonal, 'R'
},
{
6, 2,
true, true, D, D, F, true, false,
{0, 1, 1, 1, 0, 1, 1, 1, 0},
gemmi::CrystalSystem::Tetragonal, 'I'
},
{
7, 2,
true, true, D, E, E, true, false,
{1, 0, 1, 1, 1, 0, 0, 1, 1},
gemmi::CrystalSystem::Tetragonal, 'I'
},
{
8, 2,
true, true, D, E, F, true, false,
{-1, -1, 0, -1, 0, -1, 0, -1, -1},
gemmi::CrystalSystem::Orthorhombic, 'I'
},
{
9, 1,
true, false, A / 2, A / 2, A / 2, false, false,
{1, 0, 0, -1, 1, 0, -1, -1, 3},
gemmi::CrystalSystem::Trigonal, 'R'
},
{
10, 1,
true, false, D, D, F, false, false,
{1, 1, 0, 1, -1, 0, 0, 0, -1},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
11, 2,
true, false, 0, 0, 0, false, false,
{1, 0, 0, 0, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Tetragonal, 'P'
},
{
12, 2,
true, false, 0, 0, -A / 2, false, false,
{1, 0, 0, 0, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Hexagonal, 'P'
},
{
13, 2,
true, false, 0, 0, F, false, false,
{1, 1, 0, -1, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Orthorhombic, 'C'
},
{
15, 2,
true, false, -A / 2, -A / 2, 0, false, false,
{1, 0, 0, 0, 1, 0, 1, 1, 2},
gemmi::CrystalSystem::Tetragonal, 'I'
},
{
16, 2,
true, false, D, D, F, true, false,
{-1, -1, 0, 1, -1, 0, 1, 1, 2},
gemmi::CrystalSystem::Orthorhombic, 'F'
},
{
14, 2,
true, false, D, D, F, false, false,
{1, 1, 0, -1, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
17, 2,
true, false, D, E, F, true, false,
{1, -1, 0, 1, 1, 0, -1, 0, -1},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
18, 1,
false, true, A / 4, A / 2, A / 2, false, false,
{0, -1, 1, 1, -1, -1, 1, 0, 0},
gemmi::CrystalSystem::Tetragonal, 'I'
},
{
19, 1,
false, true, D, A / 2, A / 2, false, false,
{-1, 0, 0, 0, -1, 1, -1, 1, 1},
gemmi::CrystalSystem::Orthorhombic, 'I'
},
{
20, 1,
false, true, D, E, E, false, false,
{0, 1, 1, 0, 1, -1, -1, 0, 0},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
21, 2,
false, true, 0, 0, 0, false, false,
{0, 1, 0, 0, 0, 1, 1, 0, 0},
gemmi::CrystalSystem::Tetragonal, 'P'
},
{
22, 2,
false, true, -B / 2, 0, 0, false, false,
{0, 1, 0, 0, 0, 1, 1, 0, 0},
gemmi::CrystalSystem::Hexagonal, 'P'
},
{
23, 2,
false, true, D, 0, 0, false, false,
{0, 1, 1, 0, -1, 1, 1, 0, 0},
gemmi::CrystalSystem::Orthorhombic, 'C'
},
{
24, 2,
false, true, D, -A / 3, -A / 3, true, false,
{1, 2, 1, 0, -1, 1, 1, 0, 0},
gemmi::CrystalSystem::Trigonal, 'R'
},
{
25, 2,
false, true, D, E, E, false, false,
{0, 1, 1, 0, -1, 1, 1, 0, 0},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
26, 1,
false, false, A / 4, A / 2, A / 2, false, false,
{1, 0, 0, -1, 2, 0, -1, 0, 2},
gemmi::CrystalSystem::Orthorhombic, 'F'
},
{
27, 1,
false, false, D, A / 2, A / 2, false, false,
{-1, 2, 0, -1, 0, 0, 0, -1, 1},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
28, 1,
false, false, D, A / 2, 2 * D, false, false,
{-1, 0, 0, -1, 0, 2, 0, 1, 0},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
29, 1,
false, false, D, 2 * D, A / 2, false, false,
{1, 0, 0, 1, -2, 0, 0, 0, -1},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
30, 1,
false, false, B / 2, E, 2 * E, false, false,
{0, 1, 0, 0, 1, -2, -1, 0, 0},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
31, 1,
false, false, D, E, F, false, false,
{1, 0, 0, 0, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Triclinic, 'P'
},
{
32, 2,
false, false, 0, 0, 0, false, false,
{1, 0, 0, 0, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Orthorhombic, 'P'
},
{
40, 2,
false, false, -B / 2, 0, 0, false, false,
{0, -1, 0, 0, 1, 2, -1, 0, 0},
gemmi::CrystalSystem::Orthorhombic, 'C'
},
{
35, 2,
false, false, D, 0, 0, false, false,
{0, -1, 0, -1, 0, 0, 0, 0, -1},
gemmi::CrystalSystem::Monoclinic, 'P'
},
{
36, 2,
false, false, 0, -A / 2, 0, false, false,
{1, 0, 0, -1, 0, -2, 0, 1, 0},
gemmi::CrystalSystem::Orthorhombic, 'C'
},
{
33, 2,
false, false, 0, E, 0, false, false,
{1, 0, 0, 0, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Monoclinic, 'P'
},
{
38, 2,
false, false, 0, 0, -A / 2, false, false,
{-1, 0, 0, 1, 2, 0, 0, 0, -1},
gemmi::CrystalSystem::Orthorhombic, 'C'
},
{
34, 2,
false, false, 0, 0, F, false, false,
{-1, 0, 0, 0, 0, -1, 0, -1, 0},
gemmi::CrystalSystem::Monoclinic, 'P'
},
{
42, 2,
false, false, -B / 2, -A / 2, 0, false, false,
{-1, 0, 0, 0, -1, 0, 1, 1, 2},
gemmi::CrystalSystem::Orthorhombic, 'I'
},
{
41, 2,
false, false, -B / 2, E, 0, false, false,
{0, -1, -2, 0, -1, 0, -1, 0, 0},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
37, 2,
false, false, D, -A / 2, 0, false, false,
{1, 0, 2, 1, 0, 0, 0, 1, 0},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
39, 2,
false, false, D, 0, -A / 2, false, false,
{-1, -2, 0, -1, 0, 0, 0, 0, -1},
gemmi::CrystalSystem::Monoclinic, 'C'
},
{
44, 2,
false, false, D, E, F, false, false,
{1, 0, 0, 0, 1, 0, 0, 0, 1},
gemmi::CrystalSystem::Triclinic, 'P'
}
};
};
auto match = [&](const CrystalLattice &latt, double D, double E, double F)
-> std::optional<LatticeSearchResult> {
const auto uc_reduced = latt.GetUnitCell();
for (const auto &c: make_classes(D, E, F)) {
if (only_class && (c.system != only_class->first || c.centering != only_class->second))
continue;
if (c.type == 1 && uc_reduced.beta >= 90 - angle_tolerance )
continue;
bool ok = true;
if (c.cond_AB && fabs((uc_reduced.a - uc_reduced.b) / (0.5 * (uc_reduced.a + uc_reduced.b))) > dist_tolerance)
ok = false;
if (c.cond_BC && fabs((uc_reduced.b - uc_reduced.c) / (0.5 * (uc_reduced.b + uc_reduced.c))) > dist_tolerance)
ok = false;
double expected_alpha = acos(c.cond_D / sqrt(B*C)) * 180 / PI;
double expected_beta = acos(c.cond_E / sqrt(A*C)) * 180 / PI;
double expected_gamma = acos(c.cond_F / sqrt(A*B)) * 180 / PI;
if (fabs(expected_alpha - uc_reduced.alpha) > angle_tolerance)
ok = false;
if (fabs(expected_beta - uc_reduced.beta) > angle_tolerance)
ok = false;
if (fabs(expected_gamma - uc_reduced.gamma) > angle_tolerance)
ok = false;
double tmp1 = 2.0 * fabs(D + E + F);
double tmp2 = A + B;
if (c.cond_DEF && fabs((tmp1 - tmp2) / (0.5 * (tmp1 + tmp2))) > dist_tolerance)
ok = false;
if (ok) {
return LatticeSearchResult{
.niggli_class = c.number,
.primitive_reduced = latt,
.conventional = latt.Multiply(c.reindex),
.system = c.system,
.centering = c.centering,
.reindex = c.reindex,
};
}
}
return std::nullopt;
};
// Character 44 fits any cell, so a match is always found - "nothing fits" is reported as triclinic.
auto found = match(L_niggli, D, E, F);
if (found && found->system != gemmi::CrystalSystem::Triclinic)
return *found;
// A reduced cell whose beta is 90 to within NIGGLI_TYPE_BOUNDARY_DEG sits ON the boundary between the two
// Niggli types: the same lattice reduces to an all-acute cell or an all-obtuse one according to the
// last digits of whatever refinement produced it. The type-1 characters are skipped for such a cell
// (just above) and the type-2 ones are stated for the obtuse setting, so an acute cell can match none
// of them and comes back triclinic. Present it in the obtuse setting and try once more - negating a
// and c keeps the lattice and beta and turns alpha and gamma into their supplements. Measured on a
// C-centred monoclinic crystal whose reduced beta sits 0.07 deg from 90: its centring was read or
// missed according to the sign of that 0.07 deg, and with it the space group of the whole run.
if (L_niggli.GetUnitCell().beta >= 90 - NIGGLI_TYPE_BOUNDARY_DEG && D > 0 && E > 0 && F > 0) {
const gemmi::Mat33 obtuse(-1, 0, 0, 0, 1, 0, 0, 0, -1);
const auto flipped = match(L_niggli.Multiply(obtuse), -D, E, -F);
if (flipped && flipped->system != gemmi::CrystalSystem::Triclinic)
return *flipped;
}
if (found)
return *found;
if (only_class)
return std::nullopt; // no character of the requested class fits this metric
return LatticeSearchResult{
.niggli_class = 44,
.primitive_reduced = L_niggli,
.conventional = L_niggli,
.system = gemmi::CrystalSystem::Triclinic,
.centering = 'P',
.reindex = gemmi::Mat33(1, 0, 0, 0, 1, 0, 0, 0, 1),
};
}
} // namespace
LatticeSearchResult LatticeSearch(const CrystalLattice &L, double dist_tolerance, double angle_tolerance) {
return *SearchCharacters(L, dist_tolerance, angle_tolerance, nullptr);
}
std::optional<LatticeSearchResult> LatticeSearchForClass(const CrystalLattice &L,
gemmi::CrystalSystem system, char centering,
double dist_tolerance, double angle_tolerance) {
const std::pair<gemmi::CrystalSystem, char> only{system, centering};
return SearchCharacters(L, dist_tolerance, angle_tolerance, &only);
}