Files
Jungfraujoch/tests/StreamWriterTest.cpp
leonarski_f 4809c1622c
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 16m43s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 28m46s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 32m5s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 39m44s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m7s
Build Packages / Create release (push) Successful in 8s
Build Packages / Build documentation (push) Successful in 49s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m30s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m54s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 11m29s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m51s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m7s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 29m38s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m6s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 29m44s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 33m22s
Build Packages / Generate python client (push) Successful in 47s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 34m6s
Build Packages / Unit tests (push) Successful in 1h2m47s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 35m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 34m8s
Build Packages / Validation Battery (push) Successful in 38m31s
v1.0.0-rc.187 (#98)
* Rugnux: Refinement several times faster, one protocol on the CPU and the GPU, parts of it on the Apple GPU; its random draws are the same on every platform.
* Rugnux: Refinement follows common practice: the free set decides only the restraint weights and the displacement model (isotropic, TLS or anisotropic, by Hamilton's test), everything else is decided on the working set; lower R-free for placed and predicted models.
* Rugnux: Restraint dictionaries via `--restraints` (or among the inputs) and more built-in ligands; a model whose space group is a subgroup of the data's is refined in its own group; reflections plus a model select `--mode refine` on their own.
* Rugnux: `-C` fixes the crystal system on the axes as typed; `-S` alone fixes the space group only, the cell is still found and reduced; a space group or cell stored in the input file is ignored, with a line in the log.
* Rugnux: More reliable multi-sweep merging; `--mode scale` merges several stills runs as one; `_plot.txt` and the report name each input; a second crystal form is reported for stills given a cell.
* jfjoch_broker (experimental): Live Rugnux on FPGA detectors (JUNGFRAU) takes the same path as a DECTRIS stream, rotation and serial alike, model step included, with analysis threads per GPU; frames waiting for the analysis can be viewed; no host stage drops a frame - a source faster than the analysis is held at the card.
* jfjoch_broker: Summation of more than 256 frames per image is split between the card and the host (`summation_mode` is then `hybrid`), and live Rugnux runs on such collections; pedestal runs take a receiver of their own.
* jfjoch_broker (experimental): `stills_downsample` analyses 1 of every N serial images (chosen by a hash; the first 2000 all; the others written without analysis; reset to 1 by `/initialize`); `max_spot_count` left out means 250 for per-image indexing and up to 1000 for live stills; the live status gives the merge before the model work, the fitted resolution and the anomalous peak height of each unexplained blob.
* jfjoch_broker: Without a `pcie` entry the broker runs on a mock of the card instead of the HLS simulation; its spot, ROI and azimuthal results are zero then.
* Web frontend: Redesigned Rugnux live page; Vite 8, TypeScript 6, Mol* 5.13 and dependency security updates.
* Tools: `jfjoch_fpga_stills_test` measures live stills on the cards with a stored dataset; `jfjoch_fpga_test` loses the HLS-simulation option.
* Other: FFTW no longer fetched or linked (the CPU FFT indexer keeps the name `FFTW`); PCIe driver now GPL-2.0-or-later and third-party notices complete; several crash fixes.

Reviewed-on: #98
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-11-19 20:49:28 +01:00

300 lines
13 KiB
C++

// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <catch2/catch_all.hpp>
#include <filesystem>
#include <future>
#include <thread>
#include <sys/stat.h>
#include "../writer/StreamWriter.h"
#include "../image_pusher/ZMQStream2Pusher.h"
#include "../receiver/JFJochReceiverService.h"
#include "../image_pusher/ZMQWriterNotificationPuller.h"
#include "../image_puller/TestImagePuller.h"
#include "../acquisition_device/MockAcquisitionDevice.h"
TEST_CASE("StreamWriterTest_ZMQ", "[StreamWriter]") {
RegisterHDF5Filter();
Logger logger("StreamWriterTest_ZMQ");
DiffractionExperiment x(DetJF(2));
x.FilePrefix("subdir/StreamWriterTest").NumTriggers(1).ImagesPerTrigger(5)
.UseInternalPacketGenerator(true).Raw().PedestalG0Frames(0).OverwriteExistingFiles(true);
PixelMask pixel_mask(x);
JFModuleGainCalibration gain;
AcquisitionDeviceGroup aq_devices;
for (int i = 0; i < x.GetDataStreamsNum(); i++)
aq_devices.Add(std::make_unique<MockAcquisitionDevice>(i, 64));
ZMQStream2Pusher pusher({"ipc://*"});
JFJochReceiverService fpga_receiver_service(aq_devices, logger, pusher);
std::unique_ptr<StreamWriter> writer;
REQUIRE(x.GetImageNum() == 5);
auto pusher_addr = pusher.GetAddress();
REQUIRE(pusher_addr.size() == 1);
ZMQImagePuller puller(pusher_addr[0]);
REQUIRE_NOTHROW(writer = std::make_unique<StreamWriter>(logger, puller));
CHECK(writer->GetStatistics().state == StreamWriterState::Idle);
REQUIRE_NOTHROW(fpga_receiver_service.Start(x, pixel_mask, nullptr));
REQUIRE_NOTHROW(writer->Run());
REQUIRE_NOTHROW(fpga_receiver_service.Stop());
REQUIRE(fpga_receiver_service.GetStatus()->images_collected == 5);
REQUIRE(fpga_receiver_service.GetStatus()->images_sent == 5);
CHECK(writer->GetStatistics().state == StreamWriterState::Finalized);
CHECK(writer->GetStatistics().processed_images == 5);
CHECK(writer->GetStatistics().file_prefix == x.GetFilePrefix());
// HDF5 file can be opened
std::unique_ptr<HDF5ReadOnlyFile> file;
REQUIRE_NOTHROW(file = std::make_unique<HDF5ReadOnlyFile>("subdir/StreamWriterTest_data_000001.h5"));
std::unique_ptr<HDF5DataSet> dataset;
REQUIRE_NOTHROW(dataset = std::make_unique<HDF5DataSet>(*file, "/entry/data/data"));
std::unique_ptr<HDF5DataSpace> dataspace;
REQUIRE_NOTHROW(dataspace = std::make_unique<HDF5DataSpace>(*dataset));
REQUIRE(dataspace->GetNumOfDimensions() == 3);
REQUIRE(dataspace->GetDimensions()[0] == 5);
REQUIRE(dataspace->GetDimensions()[1] == RAW_MODULE_COLS);
REQUIRE(dataspace->GetDimensions()[2] == 2*RAW_MODULE_LINES);
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest_master.h5"));
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest_data_000001.h5"));
REQUIRE(std::filesystem::remove("subdir"));
}
TEST_CASE("StreamWriterTest_ZMQ_Update", "[StreamWriter]") {
RegisterHDF5Filter();
Logger logger("StreamWriterTest_ZMQ_Update");
DatasetSettings d;
d.FilePrefix("subdir/StreamWriterTest2").NumTriggers(1).ImagesPerTrigger(5).RunName("run1").RunNumber(256);
DiffractionExperiment x(DetJF(2));
x.UseInternalPacketGenerator(true).Raw().PedestalG0Frames(0)
.ImportDatasetSettings(d).OverwriteExistingFiles(true);
PixelMask pixel_mask(x);
JFModuleGainCalibration gain;
AcquisitionDeviceGroup aq_devices;
for (int i = 0; i < x.GetDataStreamsNum(); i++)
aq_devices.Add(std::make_unique<MockAcquisitionDevice>(i, 64));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
JFJochReceiverService fpga_receiver_service(aq_devices, logger, pusher);
std::unique_ptr<StreamWriter> writer;
REQUIRE(x.GetImageNum() == 5);
auto pusher_addr = pusher.GetAddress();
REQUIRE(pusher_addr.size() == 1);
ZMQImagePuller puller(pusher_addr[0]);
REQUIRE_NOTHROW(writer = std::make_unique<StreamWriter>(logger, puller));
CHECK(writer->GetStatistics().state == StreamWriterState::Idle);
REQUIRE_NOTHROW(fpga_receiver_service.Start(x, pixel_mask, nullptr));
REQUIRE_NOTHROW(writer->Run());
REQUIRE_NOTHROW(fpga_receiver_service.Stop());
REQUIRE(fpga_receiver_service.GetStatus()->images_collected == 5);
REQUIRE(fpga_receiver_service.GetStatus()->images_sent == 5);
CHECK(writer->GetStatistics().state == StreamWriterState::Finalized);
CHECK(writer->GetStatistics().processed_images == 5);
CHECK(writer->GetStatistics().file_prefix == x.GetFilePrefix());
// HDF5 file can be opened
std::unique_ptr<HDF5ReadOnlyFile> file;
REQUIRE_NOTHROW(file = std::make_unique<HDF5ReadOnlyFile>("subdir/StreamWriterTest2_data_000001.h5"));
std::unique_ptr<HDF5DataSet> dataset;
REQUIRE_NOTHROW(dataset = std::make_unique<HDF5DataSet>(*file, "/entry/data/data"));
std::unique_ptr<HDF5DataSpace> dataspace;
REQUIRE_NOTHROW(dataspace = std::make_unique<HDF5DataSpace>(*dataset));
REQUIRE(dataspace->GetNumOfDimensions() == 3);
REQUIRE(dataspace->GetDimensions()[0] == 5);
REQUIRE(dataspace->GetDimensions()[1] == RAW_MODULE_COLS);
REQUIRE(dataspace->GetDimensions()[2] == 2*RAW_MODULE_LINES);
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest2_master.h5"));
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest2_data_000001.h5"));
REQUIRE(std::filesystem::remove("subdir"));
}
TEST_CASE("StreamWriterTest_ZMQ_Update_NoNotification", "[StreamWriter]") {
// This tests simulates what happens if writer notification about writing end is missing
// Expected end result: receiver ends with an exception
RegisterHDF5Filter();
Logger logger("StreamWriterTest_ZMQ_Update_NoNotification");
DatasetSettings d;
d.FilePrefix("subdir/StreamWriterTest3").NumTriggers(1).ImagesPerTrigger(5).RunName("run1").RunNumber(256);
DiffractionExperiment x(DetJF(2));
x.UseInternalPacketGenerator(true).Raw().PedestalG0Frames(0)
.ImportDatasetSettings(d).OverwriteExistingFiles(true);
PixelMask pixel_mask(x);
JFModuleGainCalibration gain;
AcquisitionDeviceGroup aq_devices;
for (int i = 0; i < x.GetDataStreamsNum(); i++)
aq_devices.Add(std::make_unique<MockAcquisitionDevice>(i, 64));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
JFJochReceiverService fpga_receiver_service(aq_devices, logger, pusher);
std::unique_ptr<StreamWriter> writer;
REQUIRE(x.GetImageNum() == 5);
auto pusher_addr = pusher.GetAddress();
REQUIRE(pusher_addr.size() == 1);
ZMQImagePuller puller(pusher_addr[0]);
REQUIRE_NOTHROW(writer = std::make_unique<StreamWriter>(logger, puller));
writer->DebugSkipWriteNotification(true);
CHECK(writer->GetStatistics().state == StreamWriterState::Idle);
REQUIRE_NOTHROW(fpga_receiver_service.Start(x, pixel_mask, nullptr));
REQUIRE_NOTHROW(writer->Run());
JFJochReceiverOutput r;
REQUIRE_NOTHROW(r = fpga_receiver_service.Stop());
REQUIRE(!r.writer_err.empty());
REQUIRE(fpga_receiver_service.GetStatus()->images_collected == 5);
REQUIRE(fpga_receiver_service.GetStatus()->images_sent == 5);
CHECK(writer->GetStatistics().state == StreamWriterState::Finalized);
CHECK(writer->GetStatistics().processed_images == 5);
CHECK(writer->GetStatistics().file_prefix == x.GetFilePrefix());
// HDF5 file can be opened
std::unique_ptr<HDF5ReadOnlyFile> file;
REQUIRE_NOTHROW(file = std::make_unique<HDF5ReadOnlyFile>("subdir/StreamWriterTest3_data_000001.h5"));
std::unique_ptr<HDF5DataSet> dataset;
REQUIRE_NOTHROW(dataset = std::make_unique<HDF5DataSet>(*file, "/entry/data/data"));
std::unique_ptr<HDF5DataSpace> dataspace;
REQUIRE_NOTHROW(dataspace = std::make_unique<HDF5DataSpace>(*dataset));
REQUIRE(dataspace->GetNumOfDimensions() == 3);
REQUIRE(dataspace->GetDimensions()[0] == 5);
REQUIRE(dataspace->GetDimensions()[1] == RAW_MODULE_COLS);
REQUIRE(dataspace->GetDimensions()[2] == 2*RAW_MODULE_LINES);
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest3_master.h5"));
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest3_data_000001.h5"));
REQUIRE(std::filesystem::remove("subdir"));
}
// Two writers, the second one writing every other data file: the master, written by the first, must
// appear only after the second has closed its data files - here the second one starts only after the
// whole collection has been sent, so it is still writing when END goes out.
TEST_CASE("StreamWriterTest_ZMQ_MasterAfterAllDataFiles", "[StreamWriter]") {
RegisterHDF5Filter();
Logger logger("StreamWriterTest_ZMQ_MasterAfterAllDataFiles");
DatasetSettings d;
d.FilePrefix("subdir4/StreamWriterTest4").NumTriggers(1).ImagesPerTrigger(4).ImagesPerFile(2)
.RunName("run1").RunNumber(257);
DiffractionExperiment x(DetJF(2));
x.UseInternalPacketGenerator(true).Raw().PedestalG0Frames(0)
.ImportDatasetSettings(d).OverwriteExistingFiles(true);
PixelMask pixel_mask(x);
AcquisitionDeviceGroup aq_devices;
for (int i = 0; i < x.GetDataStreamsNum(); i++)
aq_devices.Add(std::make_unique<MockAcquisitionDevice>(i, 64));
ZMQStream2Pusher pusher({"ipc://*", "ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
JFJochReceiverService fpga_receiver_service(aq_devices, logger, pusher);
auto pusher_addr = pusher.GetAddress();
REQUIRE(pusher_addr.size() == 2);
ZMQImagePuller puller0(pusher_addr[0]), puller1(pusher_addr[1]);
StreamWriter writer0(logger, puller0), writer1(logger, puller1);
REQUIRE_NOTHROW(fpga_receiver_service.Start(x, pixel_mask, nullptr));
auto w0 = std::async(std::launch::async, [&] { return writer0.Run(); });
for (int i = 0; i < 1200; i++) {
const auto status = fpga_receiver_service.GetStatus();
if (status && status->images_sent == 4)
break;
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
auto stop = std::async(std::launch::async, [&] { return fpga_receiver_service.Stop(); });
std::this_thread::sleep_for(std::chrono::seconds(2));
auto w1 = std::async(std::launch::async, [&] { return writer1.Run(); });
JFJochReceiverOutput r;
REQUIRE_NOTHROW(r = stop.get());
w0.get();
w1.get();
CHECK(r.writer_err.empty());
auto changed = [](const std::string &path) {
struct stat st{};
REQUIRE(stat(path.c_str(), &st) == 0);
return std::make_pair(st.st_ctim.tv_sec, st.st_ctim.tv_nsec);
};
const auto master = changed("subdir4/StreamWriterTest4_master.h5");
for (int f = 1; f <= 2; f++)
CHECK(changed("subdir4/StreamWriterTest4_data_00000" + std::to_string(f) + ".h5") <= master);
std::filesystem::remove_all("subdir4");
}
// A failure that is not a JFJochException must end the run the same way as one that is: error state,
// END processed, state Finalized. Before this test the writer caught only its own exception type at the
// per-image and END sites, and anything else - here the JSON library refusing a sample name that is not
// UTF-8 while publishing a file-done notification - left Run() with the state stuck in Receiving, no END
// acknowledgement and no notification, and on TCP a broker waiting for that acknowledgement for ever.
TEST_CASE("StreamWriterTest_NonJFJochException_EndsRunInErrorState", "[StreamWriter]") {
RegisterHDF5Filter();
Logger logger("StreamWriterTest_NonJFJochException");
DiffractionExperiment x(DetJF(2));
x.FilePrefix("subdir5/StreamWriterTest5").NumTriggers(1).ImagesPerTrigger(2).ImagesPerFile(1)
.Compression(CompressionAlgorithm::NO_COMPRESSION).OverwriteExistingFiles(true)
.SampleName("\xff\xfe not utf-8");
StartMessage start_message;
x.FillMessage(start_message);
start_message.write_master_file = true;
TestImagePuller puller;
// The file-done PUB socket is what makes the writer serialise the sample name to JSON.
StreamWriter writer(logger, puller, "ipc://*");
puller.Put({.cbor = std::make_shared<CBORStream2DeserializerOutput>(start_message)});
std::vector<int16_t> image(x.GetPixelsNum());
for (int i = 0; i < 2; i++) {
DataMessage message{};
message.image = CompressedImage(image, x.GetXPixelsNum(), x.GetYPixelsNum());
message.number = i;
puller.Put({.cbor = std::make_shared<CBORStream2DeserializerOutput>(message)});
}
EndMessage end_message{};
end_message.max_image_number = 2;
puller.Put({.cbor = std::make_shared<CBORStream2DeserializerOutput>(end_message)});
StreamWriterOutput out;
REQUIRE_NOTHROW(out = writer.Run());
// Image 0 filled its one-image data file; closing it published the notification, which threw.
CHECK(out.stats.state == StreamWriterState::Finalized);
CHECK(out.stats.processed_images == 0);
CHECK(out.data_file_stats.empty());
// The master is never written for a run that ended in error; the data file was renamed into
// place before the notification was attempted.
CHECK(!std::filesystem::exists("subdir5/StreamWriterTest5_master.h5"));
CHECK(std::filesystem::exists("subdir5/StreamWriterTest5_data_000001.h5"));
std::filesystem::remove_all("subdir5");
}