1.0.0-rc.174 (#84)
Build Packages / Create release (push) Successful in 17s
Build Packages / build:viewer:macos-arm64:nocuda (push) Successful in 3m22s
Build Packages / build:rugnux:macos-arm64:nocuda (push) Successful in 2m37s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m33s
Build Packages / build:rugnux:linux-x86_64:cuda (push) Successful in 10m39s
Build Packages / build:viewer:linux-x86_64:nocuda (push) Successful in 11m4s
Build Packages / build:viewer:linux-x86_64:cuda (push) Successful in 13m19s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 17m37s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 18m49s
Build Packages / build:viewer:windows-x86_64:nocuda (push) Successful in 19m10s
Build Packages / build:viewer:windows-x86_64:cuda (push) Successful in 24m26s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 25m31s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 18m54s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 20m45s
Build Packages / Generate python client (push) Successful in 37s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 20m20s
Build Packages / Build documentation (push) Successful in 1m32s
Build Packages / build:rugnux:windows-x86_64:cuda (push) Successful in 14m37s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 21m6s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 19m49s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 20m29s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 17m2s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 14m27s
Build Packages / Unit tests (push) Successful in 1h18m12s

* Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results.
* Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results.
* Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior.
* Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals.
* Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4).
* Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery.
* jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies.

Reviewed-on: #84
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This commit was merged in pull request #84.
This commit is contained in:
leonarski_f authored and leonarski_f committed 2026-10-06 14:03:18 +02:00
1 parent 84228bf8be
commit a395f358ef
292 files changed
+11586 -2438

No files matched your search

+112 -14
View File
@@ -8,6 +8,7 @@
#include <cmath>
#include <cstring>
#include <optional>
#include <thread>
#include <vector>
#include "../common/DetectorSetup.h"
@@ -168,16 +169,15 @@ TEST_CASE("ShadowFinder_MaskDoesNotDependOnTheThreadCount", "[ShadowFinder]") {
CHECK(finder.GetMask(8) == one);
}
// Workers accumulate into shards of their own and the shards are summed when the projection is read,
// so which worker saw which frame must not reach the answer - including the maximum, which only one
// shard holds when the reflection is on a single frame.
TEST_CASE("ShadowFinder_ShardingDoesNotChangeTheProjection", "[ShadowFinder]") {
// Workers add their frames concurrently, each starting at a different band of the projection, so
// which worker added which frame, and in what order, must not reach the answer - including the
// maximum, which one frame alone holds when the reflection is on a single frame.
TEST_CASE("ShadowFinder_ConcurrentWorkersDoNotChangeTheProjection", "[ShadowFinder]") {
const DiffractionExperiment x = TestExperiment();
const PixelMask pixel_mask(x);
ShadowFinder serial(x, pixel_mask);
ShadowFinder sharded(x, pixel_mask);
sharded.SetShardCount(4);
ShadowFinder concurrent(x, pixel_mask);
std::vector<std::vector<int32_t>> frames;
std::vector<uint8_t> buffer;
@@ -185,22 +185,32 @@ TEST_CASE("ShadowFinder_ShardingDoesNotChangeTheProjection", "[ShadowFinder]") {
frames.push_back(Scene(/*cross=*/false, /*reflection=*/f == 0));
DataMessage msg{};
msg.image = CompressedImage(frames.back(), W, H);
serial.AddImage(msg, buffer, 0);
sharded.AddImage(msg, buffer, static_cast<size_t>(f) % 4);
serial.AddImage(msg, buffer);
}
std::vector<std::thread> workers;
for (int t = 0; t < 4; t++)
workers.emplace_back([&, t] {
std::vector<uint8_t> worker_buffer;
for (int f = NFRAMES - 1 - t; f >= 0; f -= 4) {
DataMessage msg{};
msg.image = CompressedImage(frames[f], W, H);
concurrent.AddImage(msg, worker_buffer);
}
});
for (auto &w : workers) w.join();
CHECK(serial.GetFrameCount() == sharded.GetFrameCount());
CHECK(serial.GetFrameCount() == concurrent.GetFrameCount());
const auto a = serial.GetMeanProjection();
const auto b = sharded.GetMeanProjection();
const auto b = concurrent.GetMeanProjection();
REQUIRE(a.size() == b.size());
// NAN marks a pixel nothing counted, and NAN != NAN, so compare the bits rather than the values.
CHECK(memcmp(a.data(), b.data(), a.size() * sizeof(float)) == 0);
// The reflection is on one frame, so its maximum lives in a single shard. If the fold lost it,
// the mask would swallow the reflection instead of giving it back.
CHECK(serial.GetMask(1) == sharded.GetMask(1));
CHECK(sharded.GetMask(1)[I(C - 14, C - 2)] == 0);
// The reflection is on one frame, so only that frame's maximum sees it. If it were lost, the mask
// would swallow the reflection instead of giving it back.
CHECK(serial.GetMask(1) == concurrent.GetMask(1));
CHECK(concurrent.GetMask(1)[I(C - 14, C - 2)] == 0);
}
// Four opaque arms and a centred disk: the scene is invariant under a quarter turn, so the mask must
@@ -461,3 +471,91 @@ TEST_CASE("ShadowFinder_ABrightRingIsNotABeamStop", "[ShadowFinder]") {
// Anything much beyond the stop, its arm and their penumbra means the walk ran away.
CHECK(std::count(mask.begin(), mask.end(), 1u) < 6000);
}
#ifdef JFJOCH_USE_CUDA
#include "../common/CUDAWrapper.h"
#include "../compression/JFJochCompressor.h"
namespace {
// The mask and the mean projection of the same frames, once from the host projection (the frames
// handed over uncompressed) and once from the device's (handed over as bitshuffle+LZ4, which is
// what sends them to the GPU).
struct HostAndDevice {
std::vector<uint32_t> host_mask, device_mask;
std::vector<float> host_mean, device_mean;
};
HostAndDevice MaskBothWays(const DiffractionExperiment &x, const std::vector<std::vector<int32_t>> &frames) {
const PixelMask pixel_mask(x);
HostAndDevice out;
std::vector<uint8_t> buffer;
{
ShadowFinder finder(x, pixel_mask);
for (const auto &frame : frames) {
DataMessage msg{};
msg.image = CompressedImage(frame, W, H);
finder.AddImage(msg, buffer);
}
out.host_mask = finder.GetMask();
out.host_mean = finder.GetMeanProjection();
}
{
ShadowFinder finder(x, pixel_mask);
JFJochBitShuffleCompressor compressor(CompressionAlgorithm::BSHUF_LZ4);
std::vector<std::vector<uint8_t>> compressed;
for (const auto &frame : frames) {
compressed.push_back(compressor.Compress(frame));
DataMessage msg{};
msg.image = CompressedImage(compressed.back().data(), compressed.back().size(), W, H,
CompressedImageMode::Int32, CompressionAlgorithm::BSHUF_LZ4);
finder.AddImage(msg, buffer);
}
out.device_mask = finder.GetMask();
out.device_mean = finder.GetMeanProjection();
}
return out;
}
}
// The mask is made on the GPU wherever the projection is there. On scenes where no pixel sits within a
// rounding of a threshold the two must agree to the pixel: every step but the polarization factor,
// the Poisson test's logarithm and the arm search's azimuth is exact on both, and these scenes are
// built so that none of the three decides anything at an edge.
TEST_CASE("ShadowFinder_DeviceMaskMatchesHost", "[ShadowFinder]") {
if (get_gpu_count() == 0)
SKIP("no GPU");
SECTION("a beam stop with a reflection behind it") {
std::vector<std::vector<int32_t>> frames;
for (int f = 0; f < NFRAMES; f++)
frames.push_back(Scene(false, f == 0));
const auto r = MaskBothWays(TestExperiment(), frames);
CHECK(std::count(r.host_mask.begin(), r.host_mask.end(), 1u) == 2612);
CHECK(r.device_mask == r.host_mask);
CHECK(std::memcmp(r.device_mean.data(), r.host_mean.data(), r.host_mean.size() * sizeof(float)) == 0);
}
SECTION("an arm that lets part of the beam through, across a module gap") {
constexpr int32_t BRIGHT = 50;
constexpr int ARM_HALF_WIDE = 15, GAP_X0 = 200, GAP_X1 = 216, OPAQUE_FROM_X = 232;
std::vector<std::vector<int32_t>> frames;
for (int f = 0; f < NFRAMES; f++) {
frames.emplace_back(static_cast<size_t>(W) * H, BRIGHT);
auto &frame = frames.back();
for (int y = 0; y < H; y++)
for (int xi = 0; xi < W; xi++) {
const int dx = xi - C, dy = y - C;
if (dx * dx + dy * dy <= STOP_R * STOP_R)
frame[I(xi, y)] = 0;
else if (dx >= 0 && std::abs(dy) <= ARM_HALF_WIDE)
frame[I(xi, y)] = xi >= OPAQUE_FROM_X ? 0 : BRIGHT * 6 / 10;
if (xi >= GAP_X0 && xi <= GAP_X1)
frame[I(xi, y)] = INT32_MIN;
}
}
const auto r = MaskBothWays(TestExperiment(), frames);
CHECK(std::count(r.host_mask.begin(), r.host_mask.end(), ShadowFinder::TRANSMITTING) > 0);
CHECK(r.device_mask == r.host_mask);
}
}
#endif