1.0.0-rc.174 (#84)
Build Packages / Create release (push) Successful in 17s
Build Packages / build:viewer:macos-arm64:nocuda (push) Successful in 3m22s
Build Packages / build:rugnux:macos-arm64:nocuda (push) Successful in 2m37s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m33s
Build Packages / build:rugnux:linux-x86_64:cuda (push) Successful in 10m39s
Build Packages / build:viewer:linux-x86_64:nocuda (push) Successful in 11m4s
Build Packages / build:viewer:linux-x86_64:cuda (push) Successful in 13m19s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 17m37s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 18m49s
Build Packages / build:viewer:windows-x86_64:nocuda (push) Successful in 19m10s
Build Packages / build:viewer:windows-x86_64:cuda (push) Successful in 24m26s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 25m31s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 18m54s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 20m45s
Build Packages / Generate python client (push) Successful in 37s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 20m20s
Build Packages / Build documentation (push) Successful in 1m32s
Build Packages / build:rugnux:windows-x86_64:cuda (push) Successful in 14m37s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 21m6s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 19m49s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 20m29s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 17m2s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 14m27s
Build Packages / Unit tests (push) Successful in 1h18m12s
Build Packages / Create release (push) Successful in 17s
Build Packages / build:viewer:macos-arm64:nocuda (push) Successful in 3m22s
Build Packages / build:rugnux:macos-arm64:nocuda (push) Successful in 2m37s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m33s
Build Packages / build:rugnux:linux-x86_64:cuda (push) Successful in 10m39s
Build Packages / build:viewer:linux-x86_64:nocuda (push) Successful in 11m4s
Build Packages / build:viewer:linux-x86_64:cuda (push) Successful in 13m19s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 17m37s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 18m49s
Build Packages / build:viewer:windows-x86_64:nocuda (push) Successful in 19m10s
Build Packages / build:viewer:windows-x86_64:cuda (push) Successful in 24m26s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 25m31s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 18m54s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 20m45s
Build Packages / Generate python client (push) Successful in 37s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 20m20s
Build Packages / Build documentation (push) Successful in 1m32s
Build Packages / build:rugnux:windows-x86_64:cuda (push) Successful in 14m37s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 21m6s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 19m49s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 20m29s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 17m2s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 14m27s
Build Packages / Unit tests (push) Successful in 1h18m12s
* Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This commit was merged in pull request #84.
This commit is contained in:
1 parent
84228bf8be
commit
a395f358ef
292 files changed
+11586
-2438
No files matched your search
+112
-14
@@ -8,6 +8,7 @@
|
||||
#include <cmath>
|
||||
#include <cstring>
|
||||
#include <optional>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "../common/DetectorSetup.h"
|
||||
@@ -168,16 +169,15 @@ TEST_CASE("ShadowFinder_MaskDoesNotDependOnTheThreadCount", "[ShadowFinder]") {
|
||||
CHECK(finder.GetMask(8) == one);
|
||||
}
|
||||
|
||||
// Workers accumulate into shards of their own and the shards are summed when the projection is read,
|
||||
// so which worker saw which frame must not reach the answer - including the maximum, which only one
|
||||
// shard holds when the reflection is on a single frame.
|
||||
TEST_CASE("ShadowFinder_ShardingDoesNotChangeTheProjection", "[ShadowFinder]") {
|
||||
// Workers add their frames concurrently, each starting at a different band of the projection, so
|
||||
// which worker added which frame, and in what order, must not reach the answer - including the
|
||||
// maximum, which one frame alone holds when the reflection is on a single frame.
|
||||
TEST_CASE("ShadowFinder_ConcurrentWorkersDoNotChangeTheProjection", "[ShadowFinder]") {
|
||||
const DiffractionExperiment x = TestExperiment();
|
||||
const PixelMask pixel_mask(x);
|
||||
|
||||
ShadowFinder serial(x, pixel_mask);
|
||||
ShadowFinder sharded(x, pixel_mask);
|
||||
sharded.SetShardCount(4);
|
||||
ShadowFinder concurrent(x, pixel_mask);
|
||||
|
||||
std::vector<std::vector<int32_t>> frames;
|
||||
std::vector<uint8_t> buffer;
|
||||
@@ -185,22 +185,32 @@ TEST_CASE("ShadowFinder_ShardingDoesNotChangeTheProjection", "[ShadowFinder]") {
|
||||
frames.push_back(Scene(/*cross=*/false, /*reflection=*/f == 0));
|
||||
DataMessage msg{};
|
||||
msg.image = CompressedImage(frames.back(), W, H);
|
||||
serial.AddImage(msg, buffer, 0);
|
||||
sharded.AddImage(msg, buffer, static_cast<size_t>(f) % 4);
|
||||
serial.AddImage(msg, buffer);
|
||||
}
|
||||
std::vector<std::thread> workers;
|
||||
for (int t = 0; t < 4; t++)
|
||||
workers.emplace_back([&, t] {
|
||||
std::vector<uint8_t> worker_buffer;
|
||||
for (int f = NFRAMES - 1 - t; f >= 0; f -= 4) {
|
||||
DataMessage msg{};
|
||||
msg.image = CompressedImage(frames[f], W, H);
|
||||
concurrent.AddImage(msg, worker_buffer);
|
||||
}
|
||||
});
|
||||
for (auto &w : workers) w.join();
|
||||
|
||||
CHECK(serial.GetFrameCount() == sharded.GetFrameCount());
|
||||
CHECK(serial.GetFrameCount() == concurrent.GetFrameCount());
|
||||
|
||||
const auto a = serial.GetMeanProjection();
|
||||
const auto b = sharded.GetMeanProjection();
|
||||
const auto b = concurrent.GetMeanProjection();
|
||||
REQUIRE(a.size() == b.size());
|
||||
// NAN marks a pixel nothing counted, and NAN != NAN, so compare the bits rather than the values.
|
||||
CHECK(memcmp(a.data(), b.data(), a.size() * sizeof(float)) == 0);
|
||||
|
||||
// The reflection is on one frame, so its maximum lives in a single shard. If the fold lost it,
|
||||
// the mask would swallow the reflection instead of giving it back.
|
||||
CHECK(serial.GetMask(1) == sharded.GetMask(1));
|
||||
CHECK(sharded.GetMask(1)[I(C - 14, C - 2)] == 0);
|
||||
// The reflection is on one frame, so only that frame's maximum sees it. If it were lost, the mask
|
||||
// would swallow the reflection instead of giving it back.
|
||||
CHECK(serial.GetMask(1) == concurrent.GetMask(1));
|
||||
CHECK(concurrent.GetMask(1)[I(C - 14, C - 2)] == 0);
|
||||
}
|
||||
|
||||
// Four opaque arms and a centred disk: the scene is invariant under a quarter turn, so the mask must
|
||||
@@ -461,3 +471,91 @@ TEST_CASE("ShadowFinder_ABrightRingIsNotABeamStop", "[ShadowFinder]") {
|
||||
// Anything much beyond the stop, its arm and their penumbra means the walk ran away.
|
||||
CHECK(std::count(mask.begin(), mask.end(), 1u) < 6000);
|
||||
}
|
||||
|
||||
#ifdef JFJOCH_USE_CUDA
|
||||
#include "../common/CUDAWrapper.h"
|
||||
#include "../compression/JFJochCompressor.h"
|
||||
|
||||
namespace {
|
||||
// The mask and the mean projection of the same frames, once from the host projection (the frames
|
||||
// handed over uncompressed) and once from the device's (handed over as bitshuffle+LZ4, which is
|
||||
// what sends them to the GPU).
|
||||
struct HostAndDevice {
|
||||
std::vector<uint32_t> host_mask, device_mask;
|
||||
std::vector<float> host_mean, device_mean;
|
||||
};
|
||||
|
||||
HostAndDevice MaskBothWays(const DiffractionExperiment &x, const std::vector<std::vector<int32_t>> &frames) {
|
||||
const PixelMask pixel_mask(x);
|
||||
HostAndDevice out;
|
||||
std::vector<uint8_t> buffer;
|
||||
{
|
||||
ShadowFinder finder(x, pixel_mask);
|
||||
for (const auto &frame : frames) {
|
||||
DataMessage msg{};
|
||||
msg.image = CompressedImage(frame, W, H);
|
||||
finder.AddImage(msg, buffer);
|
||||
}
|
||||
out.host_mask = finder.GetMask();
|
||||
out.host_mean = finder.GetMeanProjection();
|
||||
}
|
||||
{
|
||||
ShadowFinder finder(x, pixel_mask);
|
||||
JFJochBitShuffleCompressor compressor(CompressionAlgorithm::BSHUF_LZ4);
|
||||
std::vector<std::vector<uint8_t>> compressed;
|
||||
for (const auto &frame : frames) {
|
||||
compressed.push_back(compressor.Compress(frame));
|
||||
DataMessage msg{};
|
||||
msg.image = CompressedImage(compressed.back().data(), compressed.back().size(), W, H,
|
||||
CompressedImageMode::Int32, CompressionAlgorithm::BSHUF_LZ4);
|
||||
finder.AddImage(msg, buffer);
|
||||
}
|
||||
out.device_mask = finder.GetMask();
|
||||
out.device_mean = finder.GetMeanProjection();
|
||||
}
|
||||
return out;
|
||||
}
|
||||
}
|
||||
|
||||
// The mask is made on the GPU wherever the projection is there. On scenes where no pixel sits within a
|
||||
// rounding of a threshold the two must agree to the pixel: every step but the polarization factor,
|
||||
// the Poisson test's logarithm and the arm search's azimuth is exact on both, and these scenes are
|
||||
// built so that none of the three decides anything at an edge.
|
||||
TEST_CASE("ShadowFinder_DeviceMaskMatchesHost", "[ShadowFinder]") {
|
||||
if (get_gpu_count() == 0)
|
||||
SKIP("no GPU");
|
||||
|
||||
SECTION("a beam stop with a reflection behind it") {
|
||||
std::vector<std::vector<int32_t>> frames;
|
||||
for (int f = 0; f < NFRAMES; f++)
|
||||
frames.push_back(Scene(false, f == 0));
|
||||
const auto r = MaskBothWays(TestExperiment(), frames);
|
||||
CHECK(std::count(r.host_mask.begin(), r.host_mask.end(), 1u) == 2612);
|
||||
CHECK(r.device_mask == r.host_mask);
|
||||
CHECK(std::memcmp(r.device_mean.data(), r.host_mean.data(), r.host_mean.size() * sizeof(float)) == 0);
|
||||
}
|
||||
|
||||
SECTION("an arm that lets part of the beam through, across a module gap") {
|
||||
constexpr int32_t BRIGHT = 50;
|
||||
constexpr int ARM_HALF_WIDE = 15, GAP_X0 = 200, GAP_X1 = 216, OPAQUE_FROM_X = 232;
|
||||
std::vector<std::vector<int32_t>> frames;
|
||||
for (int f = 0; f < NFRAMES; f++) {
|
||||
frames.emplace_back(static_cast<size_t>(W) * H, BRIGHT);
|
||||
auto &frame = frames.back();
|
||||
for (int y = 0; y < H; y++)
|
||||
for (int xi = 0; xi < W; xi++) {
|
||||
const int dx = xi - C, dy = y - C;
|
||||
if (dx * dx + dy * dy <= STOP_R * STOP_R)
|
||||
frame[I(xi, y)] = 0;
|
||||
else if (dx >= 0 && std::abs(dy) <= ARM_HALF_WIDE)
|
||||
frame[I(xi, y)] = xi >= OPAQUE_FROM_X ? 0 : BRIGHT * 6 / 10;
|
||||
if (xi >= GAP_X0 && xi <= GAP_X1)
|
||||
frame[I(xi, y)] = INT32_MIN;
|
||||
}
|
||||
}
|
||||
const auto r = MaskBothWays(TestExperiment(), frames);
|
||||
CHECK(std::count(r.host_mask.begin(), r.host_mask.end(), ShadowFinder::TRANSMITTING) > 0);
|
||||
CHECK(r.device_mask == r.host_mask);
|
||||
}
|
||||
}
|
||||
#endif
|
||||
Reference in new issue
Block a user