Files
Jungfraujoch/tests/ShadowFinderTest.cpp
T
leonarski_fandjungfrau 4dc2534dbf
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
v1.0.0.rc-162 (#72)
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-25 08:21:39 +02:00

178 lines
7.7 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <catch2/catch_all.hpp>
#include <algorithm>
#include <cmath>
#include <cstring>
#include <vector>
#include "../common/DetectorSetup.h"
#include "../common/DiffractionExperiment.h"
#include "../common/JFJochMessages.h"
#include "../common/PixelMask.h"
#include "../image_analysis/beam_stop/ShadowFinder.h"
namespace {
// Odd and square, so the beam sits on a pixel and a cross-shaped scene is exactly 4-fold
// symmetric; deliberately not a multiple of 64, so the column-blocked passes meet a short
// final block.
constexpr int W = 257, H = 257, C = 128;
constexpr int NFRAMES = 12;
constexpr int32_t BACKGROUND = 2; // integer and noise-free, so every mean is exact
constexpr int STOP_R = 22, ARM_HALF = 5;
constexpr size_t I(int x, int y) { return static_cast<size_t>(y) * W + x; }
DiffractionExperiment TestExperiment() {
DiffractionExperiment x(DetDECTRIS(W, H, "Test detector", ""));
x.IncidentEnergy_keV(WVL_1A_IN_KEV).DetectorDistance_mm(150.0f);
x.BeamX_pxl(static_cast<float>(C)).BeamY_pxl(static_cast<float>(C));
return x;
}
// Flat background, an opaque disk on the beam, and an arm running off it to the edge - a beam
// stop. `cross` gives it four arms instead of one, making the scene invariant under a quarter
// turn. `reflection` puts a cluster bright enough to count as a reflection inside the disk.
std::vector<int32_t> Scene(bool cross, bool reflection) {
std::vector<int32_t> f(static_cast<size_t>(W) * H, BACKGROUND);
for (int y = 0; y < H; y++) {
for (int x = 0; x < W; x++) {
const int dx = x - C, dy = y - C;
bool blocked = dx * dx + dy * dy <= STOP_R * STOP_R;
blocked = blocked || (cross ? (std::abs(dy) <= ARM_HALF || std::abs(dx) <= ARM_HALF)
: (std::abs(dy) <= ARM_HALF && dx >= 0));
if (blocked)
f[I(x, y)] = 0;
}
}
if (reflection) {
for (int y = C - 4; y <= C; y++)
for (int x = C - 16; x <= C - 12; x++)
f[I(x, y)] = 100;
}
return f;
}
// CompressedImage does not own its pixels, so the frames have to outlive the calls.
void Feed(ShadowFinder &finder, std::vector<std::vector<int32_t>> &frames, bool cross,
bool reflection_on_first) {
std::vector<uint8_t> buffer;
for (int f = 0; f < NFRAMES; f++) {
frames.push_back(Scene(cross, reflection_on_first && (f == 0)));
DataMessage msg{};
msg.image = CompressedImage(frames.back(), W, H);
finder.AddImage(msg, buffer);
}
}
}
// The scene is a beam stop: an opaque disk on the beam with an arm running off it. What comes back
// has to be the stop and nothing else - the corners of a detector are not shadowed - and a
// reflection recorded through the penumbra is given back rather than masked.
TEST_CASE("ShadowFinder_FindsAnInjectedBeamStop", "[ShadowFinder]") {
const DiffractionExperiment x = TestExperiment();
const PixelMask pixel_mask(x);
ShadowFinder finder(x, pixel_mask);
std::vector<std::vector<int32_t>> frames;
Feed(finder, frames, /*cross=*/false, /*reflection_on_first=*/true);
REQUIRE(finder.GetFrameCount() == NFRAMES);
const auto mask = finder.GetMask();
REQUIRE(mask.size() == static_cast<size_t>(W) * H);
CHECK(mask[I(C, C)] == 1); // the stop itself
CHECK(mask[I(C + STOP_R - 3, C)] == 1);
CHECK(mask[I(W - 3, C)] == 1); // the arm, followed to the edge
CHECK(mask[I(W - 3, C + 4 * ARM_HALF)] == 0); // and nothing beside it
CHECK(mask[I(0, 0)] == 0);
CHECK(mask[I(W - 1, 0)] == 0);
CHECK(mask[I(0, H - 1)] == 0);
CHECK(mask[I(W - 1, H - 1)] == 0);
CHECK(mask[I(C - 14, C - 2)] == 0); // a recorded reflection is given back
// The mean projection is what the mask is computed from: exact here, because the scene is
// integer and noise-free.
const auto projection = finder.GetMeanProjection();
REQUIRE(projection.size() == mask.size());
CHECK(projection[I(0, 0)] == Catch::Approx(BACKGROUND));
CHECK(projection[I(C, C)] == Catch::Approx(0.0));
// Pinned from the serial implementation. A rewrite of the dilation, the hole fill or the ring
// median that moves the mask by one pixel fails here, rather than in a merging statistic
// several stages downstream.
CHECK(std::count(mask.begin(), mask.end(), 1u) == 2669);
}
// The per-pixel passes are split across threads, so where the split falls must not be visible in the
// answer. The x pass and the y pass of the dilation and of the pooled sum are written differently -
// one a plain scan, the other blocked by column - so an x/y asymmetry is the plausible regression.
TEST_CASE("ShadowFinder_MaskDoesNotDependOnTheThreadCount", "[ShadowFinder]") {
const DiffractionExperiment x = TestExperiment();
const PixelMask pixel_mask(x);
ShadowFinder finder(x, pixel_mask);
std::vector<std::vector<int32_t>> frames;
Feed(finder, frames, /*cross=*/false, /*reflection_on_first=*/true);
const auto one = finder.GetMask(1);
CHECK(finder.GetMask(3) == one);
CHECK(finder.GetMask(8) == one);
}
// Workers accumulate into shards of their own and the shards are summed when the projection is read,
// so which worker saw which frame must not reach the answer - including the maximum, which only one
// shard holds when the reflection is on a single frame.
TEST_CASE("ShadowFinder_ShardingDoesNotChangeTheProjection", "[ShadowFinder]") {
const DiffractionExperiment x = TestExperiment();
const PixelMask pixel_mask(x);
ShadowFinder serial(x, pixel_mask);
ShadowFinder sharded(x, pixel_mask);
sharded.SetShardCount(4);
std::vector<std::vector<int32_t>> frames;
std::vector<uint8_t> buffer;
for (int f = 0; f < NFRAMES; f++) {
frames.push_back(Scene(/*cross=*/false, /*reflection=*/f == 0));
DataMessage msg{};
msg.image = CompressedImage(frames.back(), W, H);
serial.AddImage(msg, buffer, 0);
sharded.AddImage(msg, buffer, static_cast<size_t>(f) % 4);
}
CHECK(serial.GetFrameCount() == sharded.GetFrameCount());
const auto a = serial.GetMeanProjection();
const auto b = sharded.GetMeanProjection();
REQUIRE(a.size() == b.size());
// NAN marks a pixel nothing counted, and NAN != NAN, so compare the bits rather than the values.
CHECK(memcmp(a.data(), b.data(), a.size() * sizeof(float)) == 0);
// The reflection is on one frame, so its maximum lives in a single shard. If the fold lost it,
// the mask would swallow the reflection instead of giving it back.
CHECK(serial.GetMask(1) == sharded.GetMask(1));
CHECK(sharded.GetMask(1)[I(C - 14, C - 2)] == 0);
}
// Four opaque arms and a centred disk: the scene is invariant under a quarter turn, so the mask must
// be too, whatever the thread count.
TEST_CASE("ShadowFinder_ASymmetricSceneGivesASymmetricMask", "[ShadowFinder]") {
const DiffractionExperiment x = TestExperiment();
const PixelMask pixel_mask(x);
ShadowFinder finder(x, pixel_mask);
std::vector<std::vector<int32_t>> frames;
Feed(finder, frames, /*cross=*/true, /*reflection_on_first=*/false);
const auto mask = finder.GetMask(8);
for (int y = 0; y < H; y++) {
for (int xi = 0; xi < W; xi++) {
REQUIRE(mask[I(xi, y)] == mask[I(y, xi)]); // transpose
REQUIRE(mask[I(xi, y)] == mask[I(W - 1 - y, xi)]); // quarter turn
}
}
}