Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 16m43s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 28m46s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 32m5s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 39m44s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m7s
Build Packages / Create release (push) Successful in 8s
Build Packages / Build documentation (push) Successful in 49s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m30s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m54s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 11m29s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m51s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m7s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 29m38s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m6s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 29m44s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 33m22s
Build Packages / Generate python client (push) Successful in 47s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 34m6s
Build Packages / Unit tests (push) Successful in 1h2m47s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 35m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 34m8s
Build Packages / Validation Battery (push) Successful in 38m31s
* Rugnux: Refinement several times faster, one protocol on the CPU and the GPU, parts of it on the Apple GPU; its random draws are the same on every platform. * Rugnux: Refinement follows common practice: the free set decides only the restraint weights and the displacement model (isotropic, TLS or anisotropic, by Hamilton's test), everything else is decided on the working set; lower R-free for placed and predicted models. * Rugnux: Restraint dictionaries via `--restraints` (or among the inputs) and more built-in ligands; a model whose space group is a subgroup of the data's is refined in its own group; reflections plus a model select `--mode refine` on their own. * Rugnux: `-C` fixes the crystal system on the axes as typed; `-S` alone fixes the space group only, the cell is still found and reduced; a space group or cell stored in the input file is ignored, with a line in the log. * Rugnux: More reliable multi-sweep merging; `--mode scale` merges several stills runs as one; `_plot.txt` and the report name each input; a second crystal form is reported for stills given a cell. * jfjoch_broker (experimental): Live Rugnux on FPGA detectors (JUNGFRAU) takes the same path as a DECTRIS stream, rotation and serial alike, model step included, with analysis threads per GPU; frames waiting for the analysis can be viewed; no host stage drops a frame - a source faster than the analysis is held at the card. * jfjoch_broker: Summation of more than 256 frames per image is split between the card and the host (`summation_mode` is then `hybrid`), and live Rugnux runs on such collections; pedestal runs take a receiver of their own. * jfjoch_broker (experimental): `stills_downsample` analyses 1 of every N serial images (chosen by a hash; the first 2000 all; the others written without analysis; reset to 1 by `/initialize`); `max_spot_count` left out means 250 for per-image indexing and up to 1000 for live stills; the live status gives the merge before the model work, the fitted resolution and the anomalous peak height of each unexplained blob. * jfjoch_broker: Without a `pcie` entry the broker runs on a mock of the card instead of the HLS simulation; its spot, ROI and azimuthal results are zero then. * Web frontend: Redesigned Rugnux live page; Vite 8, TypeScript 6, Mol* 5.13 and dependency security updates. * Tools: `jfjoch_fpga_stills_test` measures live stills on the cards with a stored dataset; `jfjoch_fpga_test` loses the HLS-simulation option. * Other: FFTW no longer fetched or linked (the CPU FFT indexer keeps the name `FFTW`); PCIe driver now GPL-2.0-or-later and third-party notices complete; several crash fixes. Reviewed-on: #98 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
300 lines
13 KiB
C++
300 lines
13 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include <catch2/catch_all.hpp>
|
|
#include <filesystem>
|
|
#include <future>
|
|
#include <thread>
|
|
#include <sys/stat.h>
|
|
|
|
#include "../writer/StreamWriter.h"
|
|
#include "../image_pusher/ZMQStream2Pusher.h"
|
|
#include "../receiver/JFJochReceiverService.h"
|
|
#include "../image_pusher/ZMQWriterNotificationPuller.h"
|
|
#include "../image_puller/TestImagePuller.h"
|
|
#include "../acquisition_device/MockAcquisitionDevice.h"
|
|
|
|
TEST_CASE("StreamWriterTest_ZMQ", "[StreamWriter]") {
|
|
RegisterHDF5Filter();
|
|
|
|
Logger logger("StreamWriterTest_ZMQ");
|
|
|
|
DiffractionExperiment x(DetJF(2));
|
|
x.FilePrefix("subdir/StreamWriterTest").NumTriggers(1).ImagesPerTrigger(5)
|
|
.UseInternalPacketGenerator(true).Raw().PedestalG0Frames(0).OverwriteExistingFiles(true);
|
|
|
|
PixelMask pixel_mask(x);
|
|
|
|
JFModuleGainCalibration gain;
|
|
AcquisitionDeviceGroup aq_devices;
|
|
for (int i = 0; i < x.GetDataStreamsNum(); i++)
|
|
aq_devices.Add(std::make_unique<MockAcquisitionDevice>(i, 64));
|
|
|
|
ZMQStream2Pusher pusher({"ipc://*"});
|
|
JFJochReceiverService fpga_receiver_service(aq_devices, logger, pusher);
|
|
|
|
std::unique_ptr<StreamWriter> writer;
|
|
REQUIRE(x.GetImageNum() == 5);
|
|
auto pusher_addr = pusher.GetAddress();
|
|
REQUIRE(pusher_addr.size() == 1);
|
|
ZMQImagePuller puller(pusher_addr[0]);
|
|
REQUIRE_NOTHROW(writer = std::make_unique<StreamWriter>(logger, puller));
|
|
CHECK(writer->GetStatistics().state == StreamWriterState::Idle);
|
|
REQUIRE_NOTHROW(fpga_receiver_service.Start(x, pixel_mask, nullptr));
|
|
|
|
REQUIRE_NOTHROW(writer->Run());
|
|
|
|
REQUIRE_NOTHROW(fpga_receiver_service.Stop());
|
|
REQUIRE(fpga_receiver_service.GetStatus()->images_collected == 5);
|
|
REQUIRE(fpga_receiver_service.GetStatus()->images_sent == 5);
|
|
CHECK(writer->GetStatistics().state == StreamWriterState::Finalized);
|
|
CHECK(writer->GetStatistics().processed_images == 5);
|
|
CHECK(writer->GetStatistics().file_prefix == x.GetFilePrefix());
|
|
|
|
// HDF5 file can be opened
|
|
std::unique_ptr<HDF5ReadOnlyFile> file;
|
|
REQUIRE_NOTHROW(file = std::make_unique<HDF5ReadOnlyFile>("subdir/StreamWriterTest_data_000001.h5"));
|
|
std::unique_ptr<HDF5DataSet> dataset;
|
|
REQUIRE_NOTHROW(dataset = std::make_unique<HDF5DataSet>(*file, "/entry/data/data"));
|
|
std::unique_ptr<HDF5DataSpace> dataspace;
|
|
REQUIRE_NOTHROW(dataspace = std::make_unique<HDF5DataSpace>(*dataset));
|
|
|
|
REQUIRE(dataspace->GetNumOfDimensions() == 3);
|
|
REQUIRE(dataspace->GetDimensions()[0] == 5);
|
|
REQUIRE(dataspace->GetDimensions()[1] == RAW_MODULE_COLS);
|
|
REQUIRE(dataspace->GetDimensions()[2] == 2*RAW_MODULE_LINES);
|
|
|
|
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest_master.h5"));
|
|
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest_data_000001.h5"));
|
|
REQUIRE(std::filesystem::remove("subdir"));
|
|
}
|
|
|
|
TEST_CASE("StreamWriterTest_ZMQ_Update", "[StreamWriter]") {
|
|
RegisterHDF5Filter();
|
|
|
|
Logger logger("StreamWriterTest_ZMQ_Update");
|
|
|
|
DatasetSettings d;
|
|
d.FilePrefix("subdir/StreamWriterTest2").NumTriggers(1).ImagesPerTrigger(5).RunName("run1").RunNumber(256);
|
|
DiffractionExperiment x(DetJF(2));
|
|
x.UseInternalPacketGenerator(true).Raw().PedestalG0Frames(0)
|
|
.ImportDatasetSettings(d).OverwriteExistingFiles(true);
|
|
PixelMask pixel_mask(x);
|
|
|
|
JFModuleGainCalibration gain;
|
|
AcquisitionDeviceGroup aq_devices;
|
|
for (int i = 0; i < x.GetDataStreamsNum(); i++)
|
|
aq_devices.Add(std::make_unique<MockAcquisitionDevice>(i, 64));
|
|
|
|
ZMQStream2Pusher pusher({"ipc://*"});
|
|
pusher.WriterNotificationSocket("ipc://*");
|
|
JFJochReceiverService fpga_receiver_service(aq_devices, logger, pusher);
|
|
|
|
std::unique_ptr<StreamWriter> writer;
|
|
REQUIRE(x.GetImageNum() == 5);
|
|
auto pusher_addr = pusher.GetAddress();
|
|
REQUIRE(pusher_addr.size() == 1);
|
|
|
|
ZMQImagePuller puller(pusher_addr[0]);
|
|
REQUIRE_NOTHROW(writer = std::make_unique<StreamWriter>(logger, puller));
|
|
CHECK(writer->GetStatistics().state == StreamWriterState::Idle);
|
|
REQUIRE_NOTHROW(fpga_receiver_service.Start(x, pixel_mask, nullptr));
|
|
|
|
REQUIRE_NOTHROW(writer->Run());
|
|
|
|
REQUIRE_NOTHROW(fpga_receiver_service.Stop());
|
|
REQUIRE(fpga_receiver_service.GetStatus()->images_collected == 5);
|
|
REQUIRE(fpga_receiver_service.GetStatus()->images_sent == 5);
|
|
CHECK(writer->GetStatistics().state == StreamWriterState::Finalized);
|
|
CHECK(writer->GetStatistics().processed_images == 5);
|
|
CHECK(writer->GetStatistics().file_prefix == x.GetFilePrefix());
|
|
|
|
// HDF5 file can be opened
|
|
std::unique_ptr<HDF5ReadOnlyFile> file;
|
|
REQUIRE_NOTHROW(file = std::make_unique<HDF5ReadOnlyFile>("subdir/StreamWriterTest2_data_000001.h5"));
|
|
std::unique_ptr<HDF5DataSet> dataset;
|
|
REQUIRE_NOTHROW(dataset = std::make_unique<HDF5DataSet>(*file, "/entry/data/data"));
|
|
std::unique_ptr<HDF5DataSpace> dataspace;
|
|
REQUIRE_NOTHROW(dataspace = std::make_unique<HDF5DataSpace>(*dataset));
|
|
|
|
REQUIRE(dataspace->GetNumOfDimensions() == 3);
|
|
REQUIRE(dataspace->GetDimensions()[0] == 5);
|
|
REQUIRE(dataspace->GetDimensions()[1] == RAW_MODULE_COLS);
|
|
REQUIRE(dataspace->GetDimensions()[2] == 2*RAW_MODULE_LINES);
|
|
|
|
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest2_master.h5"));
|
|
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest2_data_000001.h5"));
|
|
REQUIRE(std::filesystem::remove("subdir"));
|
|
}
|
|
|
|
TEST_CASE("StreamWriterTest_ZMQ_Update_NoNotification", "[StreamWriter]") {
|
|
// This tests simulates what happens if writer notification about writing end is missing
|
|
// Expected end result: receiver ends with an exception
|
|
|
|
RegisterHDF5Filter();
|
|
|
|
Logger logger("StreamWriterTest_ZMQ_Update_NoNotification");
|
|
|
|
DatasetSettings d;
|
|
d.FilePrefix("subdir/StreamWriterTest3").NumTriggers(1).ImagesPerTrigger(5).RunName("run1").RunNumber(256);
|
|
DiffractionExperiment x(DetJF(2));
|
|
x.UseInternalPacketGenerator(true).Raw().PedestalG0Frames(0)
|
|
.ImportDatasetSettings(d).OverwriteExistingFiles(true);
|
|
PixelMask pixel_mask(x);
|
|
|
|
JFModuleGainCalibration gain;
|
|
AcquisitionDeviceGroup aq_devices;
|
|
for (int i = 0; i < x.GetDataStreamsNum(); i++)
|
|
aq_devices.Add(std::make_unique<MockAcquisitionDevice>(i, 64));
|
|
|
|
ZMQStream2Pusher pusher({"ipc://*"});
|
|
pusher.WriterNotificationSocket("ipc://*");
|
|
JFJochReceiverService fpga_receiver_service(aq_devices, logger, pusher);
|
|
|
|
std::unique_ptr<StreamWriter> writer;
|
|
REQUIRE(x.GetImageNum() == 5);
|
|
auto pusher_addr = pusher.GetAddress();
|
|
REQUIRE(pusher_addr.size() == 1);
|
|
|
|
ZMQImagePuller puller(pusher_addr[0]);
|
|
REQUIRE_NOTHROW(writer = std::make_unique<StreamWriter>(logger, puller));
|
|
writer->DebugSkipWriteNotification(true);
|
|
|
|
CHECK(writer->GetStatistics().state == StreamWriterState::Idle);
|
|
REQUIRE_NOTHROW(fpga_receiver_service.Start(x, pixel_mask, nullptr));
|
|
|
|
REQUIRE_NOTHROW(writer->Run());
|
|
|
|
JFJochReceiverOutput r;
|
|
REQUIRE_NOTHROW(r = fpga_receiver_service.Stop());
|
|
REQUIRE(!r.writer_err.empty());
|
|
REQUIRE(fpga_receiver_service.GetStatus()->images_collected == 5);
|
|
REQUIRE(fpga_receiver_service.GetStatus()->images_sent == 5);
|
|
CHECK(writer->GetStatistics().state == StreamWriterState::Finalized);
|
|
CHECK(writer->GetStatistics().processed_images == 5);
|
|
CHECK(writer->GetStatistics().file_prefix == x.GetFilePrefix());
|
|
|
|
// HDF5 file can be opened
|
|
std::unique_ptr<HDF5ReadOnlyFile> file;
|
|
REQUIRE_NOTHROW(file = std::make_unique<HDF5ReadOnlyFile>("subdir/StreamWriterTest3_data_000001.h5"));
|
|
std::unique_ptr<HDF5DataSet> dataset;
|
|
REQUIRE_NOTHROW(dataset = std::make_unique<HDF5DataSet>(*file, "/entry/data/data"));
|
|
std::unique_ptr<HDF5DataSpace> dataspace;
|
|
REQUIRE_NOTHROW(dataspace = std::make_unique<HDF5DataSpace>(*dataset));
|
|
|
|
REQUIRE(dataspace->GetNumOfDimensions() == 3);
|
|
REQUIRE(dataspace->GetDimensions()[0] == 5);
|
|
REQUIRE(dataspace->GetDimensions()[1] == RAW_MODULE_COLS);
|
|
REQUIRE(dataspace->GetDimensions()[2] == 2*RAW_MODULE_LINES);
|
|
|
|
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest3_master.h5"));
|
|
REQUIRE(std::filesystem::remove("subdir/StreamWriterTest3_data_000001.h5"));
|
|
REQUIRE(std::filesystem::remove("subdir"));
|
|
}
|
|
|
|
// Two writers, the second one writing every other data file: the master, written by the first, must
|
|
// appear only after the second has closed its data files - here the second one starts only after the
|
|
// whole collection has been sent, so it is still writing when END goes out.
|
|
TEST_CASE("StreamWriterTest_ZMQ_MasterAfterAllDataFiles", "[StreamWriter]") {
|
|
RegisterHDF5Filter();
|
|
Logger logger("StreamWriterTest_ZMQ_MasterAfterAllDataFiles");
|
|
|
|
DatasetSettings d;
|
|
d.FilePrefix("subdir4/StreamWriterTest4").NumTriggers(1).ImagesPerTrigger(4).ImagesPerFile(2)
|
|
.RunName("run1").RunNumber(257);
|
|
DiffractionExperiment x(DetJF(2));
|
|
x.UseInternalPacketGenerator(true).Raw().PedestalG0Frames(0)
|
|
.ImportDatasetSettings(d).OverwriteExistingFiles(true);
|
|
PixelMask pixel_mask(x);
|
|
|
|
AcquisitionDeviceGroup aq_devices;
|
|
for (int i = 0; i < x.GetDataStreamsNum(); i++)
|
|
aq_devices.Add(std::make_unique<MockAcquisitionDevice>(i, 64));
|
|
|
|
ZMQStream2Pusher pusher({"ipc://*", "ipc://*"});
|
|
pusher.WriterNotificationSocket("ipc://*");
|
|
JFJochReceiverService fpga_receiver_service(aq_devices, logger, pusher);
|
|
|
|
auto pusher_addr = pusher.GetAddress();
|
|
REQUIRE(pusher_addr.size() == 2);
|
|
ZMQImagePuller puller0(pusher_addr[0]), puller1(pusher_addr[1]);
|
|
StreamWriter writer0(logger, puller0), writer1(logger, puller1);
|
|
|
|
REQUIRE_NOTHROW(fpga_receiver_service.Start(x, pixel_mask, nullptr));
|
|
auto w0 = std::async(std::launch::async, [&] { return writer0.Run(); });
|
|
for (int i = 0; i < 1200; i++) {
|
|
const auto status = fpga_receiver_service.GetStatus();
|
|
if (status && status->images_sent == 4)
|
|
break;
|
|
std::this_thread::sleep_for(std::chrono::milliseconds(100));
|
|
}
|
|
auto stop = std::async(std::launch::async, [&] { return fpga_receiver_service.Stop(); });
|
|
std::this_thread::sleep_for(std::chrono::seconds(2));
|
|
auto w1 = std::async(std::launch::async, [&] { return writer1.Run(); });
|
|
JFJochReceiverOutput r;
|
|
REQUIRE_NOTHROW(r = stop.get());
|
|
w0.get();
|
|
w1.get();
|
|
CHECK(r.writer_err.empty());
|
|
|
|
auto changed = [](const std::string &path) {
|
|
struct stat st{};
|
|
REQUIRE(stat(path.c_str(), &st) == 0);
|
|
return std::make_pair(st.st_ctim.tv_sec, st.st_ctim.tv_nsec);
|
|
};
|
|
const auto master = changed("subdir4/StreamWriterTest4_master.h5");
|
|
for (int f = 1; f <= 2; f++)
|
|
CHECK(changed("subdir4/StreamWriterTest4_data_00000" + std::to_string(f) + ".h5") <= master);
|
|
|
|
std::filesystem::remove_all("subdir4");
|
|
}
|
|
|
|
// A failure that is not a JFJochException must end the run the same way as one that is: error state,
|
|
// END processed, state Finalized. Before this test the writer caught only its own exception type at the
|
|
// per-image and END sites, and anything else - here the JSON library refusing a sample name that is not
|
|
// UTF-8 while publishing a file-done notification - left Run() with the state stuck in Receiving, no END
|
|
// acknowledgement and no notification, and on TCP a broker waiting for that acknowledgement for ever.
|
|
TEST_CASE("StreamWriterTest_NonJFJochException_EndsRunInErrorState", "[StreamWriter]") {
|
|
RegisterHDF5Filter();
|
|
Logger logger("StreamWriterTest_NonJFJochException");
|
|
|
|
DiffractionExperiment x(DetJF(2));
|
|
x.FilePrefix("subdir5/StreamWriterTest5").NumTriggers(1).ImagesPerTrigger(2).ImagesPerFile(1)
|
|
.Compression(CompressionAlgorithm::NO_COMPRESSION).OverwriteExistingFiles(true)
|
|
.SampleName("\xff\xfe not utf-8");
|
|
|
|
StartMessage start_message;
|
|
x.FillMessage(start_message);
|
|
start_message.write_master_file = true;
|
|
|
|
TestImagePuller puller;
|
|
// The file-done PUB socket is what makes the writer serialise the sample name to JSON.
|
|
StreamWriter writer(logger, puller, "ipc://*");
|
|
|
|
puller.Put({.cbor = std::make_shared<CBORStream2DeserializerOutput>(start_message)});
|
|
std::vector<int16_t> image(x.GetPixelsNum());
|
|
for (int i = 0; i < 2; i++) {
|
|
DataMessage message{};
|
|
message.image = CompressedImage(image, x.GetXPixelsNum(), x.GetYPixelsNum());
|
|
message.number = i;
|
|
puller.Put({.cbor = std::make_shared<CBORStream2DeserializerOutput>(message)});
|
|
}
|
|
EndMessage end_message{};
|
|
end_message.max_image_number = 2;
|
|
puller.Put({.cbor = std::make_shared<CBORStream2DeserializerOutput>(end_message)});
|
|
|
|
StreamWriterOutput out;
|
|
REQUIRE_NOTHROW(out = writer.Run());
|
|
|
|
// Image 0 filled its one-image data file; closing it published the notification, which threw.
|
|
CHECK(out.stats.state == StreamWriterState::Finalized);
|
|
CHECK(out.stats.processed_images == 0);
|
|
CHECK(out.data_file_stats.empty());
|
|
// The master is never written for a run that ended in error; the data file was renamed into
|
|
// place before the notification was attempted.
|
|
CHECK(!std::filesystem::exists("subdir5/StreamWriterTest5_master.h5"));
|
|
CHECK(std::filesystem::exists("subdir5/StreamWriterTest5_data_000001.h5"));
|
|
|
|
std::filesystem::remove_all("subdir5");
|
|
}
|