Files
Jungfraujoch/tests/JFJochReceiverProcessingTest.cpp
leonarski_fandClaude Opus 5.5 8bad85f9f4 Live stills in JFJochReceiverRugnux, not in JFJochReceiverFPGA
With rugnux live mode on, a serial collection from a DECTRIS detector now runs the
whole live stills workflow instead of per-image analysis only: the frames wait in the
image buffer for a pre-pass that fixes the beam centre, distance and mask, are analysed
on the GPU at that geometry, and are merged at half and three quarters of the images and
after the last. With a model, it is placed, mapped and refined on each merge. The start
message waits for the pre-pass, the end message carries the merged data, and /rugnux/status
and the frontend report the run. The receiver uses only what a puller delivers, a
DataMessage with a compressed image, so a later FPGA puller feeds the same code.

FPGA users lose this until that puller exists: JFJochReceiverFPGA has no stills path any
more, so a JUNGFRAU or PSI EIGER collection with rugnux on runs image by image on the
card's results, as with the mode off, with /rugnux/status idle, no pre-pass and no live
merge.

From a stream there are no module packets: efficiency is the images received over those
expected, a frame not delivered leaves no entry, and per-image collection efficiency, ADU
histograms and pulse IDs are absent.

The dark frames of the mask sequence always stay with JFJochReceiverLite. Selection used to
look only at the goniometer, so POST /pedestal after a rotation /start with rugnux on went
to the rugnux receiver and returned an empty mask. Frames numbered outside the collection
are dropped: the analysis keeps per-image results by number, and they corrupted the heap.

Carries the behaviour of the live commits of rc184, updated for this receiver: the geometry
fixed when the pre-pass ends (the running refinement is removed), the q range fixed, the
pre-pass holding more frames until the geometry can be fitted, frames arriving during it
waiting in the whole image buffer, the indexer pool's device memory report, the stills model
step on provisional merges and after the last image, /cancel cutting it, and the model and
Mol* viewer of a serial run during the collection.

Rotation on this receiver is unchanged: end to end, the old and new receiver wrote a
byte-identical MTZ.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 14:50:39 +01:00

1881 lines
77 KiB
C++

// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <catch2/catch_all.hpp>
#include <fstream>
#include "../receiver/JFJochReceiverTest.h"
#include "../acquisition_device/HLSSimulatedDevice.h"
#include "../writer/HDF5Objects.h"
#include "../receiver/JFJochReceiverService.h"
#include "../preview/JFJochTIFF.h"
#include "../image_pusher/ZMQStream2Pusher.h"
#include "../writer/StreamWriter.h"
#include "../image_pusher/NonePusher.h"
#include "../image_pusher/HDF5FilePusher.h"
#include "../image_pusher/TCPStreamPusher.h"
#include "../image_puller/TCPImagePuller.h"
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
experiment.SampleTemperature_K(123.0).RingCurrent_mA(115);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_refinement", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_refinement").JungfrauConvPhotonCnt(false).SetFileWriterFormat(
FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90})
.GeomRefinementAlgorithm(GeomRefinementAlgorithmEnum::BeamCenter);
experiment.SampleTemperature_K(123.0).RingCurrent_mA(115);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_refinement_tetragonal", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_refinement_tetragonal").JungfrauConvPhotonCnt(false).SetFileWriterFormat(
FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 78.95, .b = 78.95, .c = 36.9, .alpha = 90, .beta = 90, .gamma = 90})
.GeomRefinementAlgorithm(GeomRefinementAlgorithmEnum::BeamCenter)
.SpaceGroupNumber(96);
experiment.SampleTemperature_K(123.0).RingCurrent_mA(115);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_index_and_integrate", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
settings.quick_integration = true;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
logger.Info("Processing time {}", receiver_out.processing_time.processing);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
auto b_factor = service.GetDataProcessingPlot(
PlotRequest{.type = PlotType::BFactor, .binning = 1});
REQUIRE(b_factor.GetPlots().size() == 1);
REQUIRE(b_factor.GetPlots()[0].x.size() == experiment.GetImageNum());
logger.Info("B-factor {}", b_factor.GetPlots()[0].y[0]);
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_index_and_integrate_no_spots", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 20; // make sure there are zero spots possible
settings.photon_count_threshold = 10000;
settings.min_pix_per_spot = 5;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
settings.quick_integration = true;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 0.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
logger.Info("Processing time {}", receiver_out.processing_time.processing);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_azint_2d", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_azim_2d").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
PixelMask pixel_mask(experiment);
AzimuthalIntegrationSettings azim;
azim.QSpacing_recipA(0.05).QRange_recipA(0.05, 6.05).AzimuthalBinCount(16);
experiment.ImportAzimuthalIntegrationSettings(azim);
REQUIRE(azim.GetBinCount() == 6 * 20 * 16);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
auto azim_plot = service.GetDataProcessingPlot(PlotRequest{.type = PlotType::AzInt, .binning = 1});
REQUIRE(azim_plot.GetPlots().size() == 1);
REQUIRE(azim_plot.GetPlots()[0].x.size() == 120 * 16);
REQUIRE(azim_plot.GetPlots()[0].y.size() == 120 * 16);
REQUIRE(azim_plot.GetPlots()[0].z.size() == 120 * 16);
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_FFT", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_fft").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS)
.OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.IndexingAlgorithm(IndexingAlgorithmEnum::FFT);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_FFTW", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_fftw").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS)
.OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.IndexingAlgorithm(IndexingAlgorithmEnum::FFTW);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_FFT_reference", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_fft_ref").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS)
.OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.IndexingAlgorithm(IndexingAlgorithmEnum::FFT)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_FFT_wrong_reference", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_fft_wrongref").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS)
.OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.IndexingAlgorithm(IndexingAlgorithmEnum::FFT)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 55.0, .alpha =90, .beta = 90, .gamma = 90});
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 0.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_grid", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
GridScanSettings grid_scan(3, 10.0f, 25.0f, false, true);
DatasetSettings dataset_settings;
dataset_settings.ImagesPerTrigger(7).NumTriggers(1).DetectorDistance_mm(75)
.BeamY_pxl(1136).BeamX_pxl(1090)
.PhotonEnergy_keV(12.4).GridScan(grid_scan)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
DiffractionExperiment experiment(DetJF4M());
experiment.UseInternalPacketGenerator(true).ImagesPerFile(2).ImportDatasetSettings(dataset_settings)
.FilePrefix("crystal_test_grid").JungfrauConvPhotonCnt(false)
.SetFileWriterFormat(FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// 3x3 grid = 9 cells, but only 7 images collected; the two uncollected
// placeholder rows (grid indices 5 and 8) are dropped from scan_result.
REQUIRE(receiver_out.scan_result.images.size() == 7);
CHECK(receiver_out.scan_result.images[0].number == 0);
CHECK(receiver_out.scan_result.images[1].number == 3);
CHECK(receiver_out.scan_result.images[3].number == 1);
auto grid_scan_out = service.GetDataProcessingPlot(
PlotRequest{.type = PlotType::IndexingRate, .experimental_coord = true});
CHECK(grid_scan_out.GetUnits() == MultiLinePlotUnits::Grid_um);
REQUIRE(grid_scan_out.GetPlots().size() == 1);
REQUIRE(grid_scan_out.GetPlots()[0].z.size() == 7);
CHECK(grid_scan_out.GetPlots()[0].z[0] == 1.0f);
CHECK(grid_scan_out.GetPlots()[0].z[1] == 1.0f);
CHECK(grid_scan_out.GetPlots()[0].z[2] == 1.0f);
CHECK(grid_scan_out.GetPlots()[0].z[3] == 1.0f);
CHECK(grid_scan_out.GetPlots()[0].z[4] == 1.0f);
CHECK(grid_scan_out.GetPlots()[0].z[5] == 1.0f);
CHECK(grid_scan_out.GetPlots()[0].z[6] == 1.0f);
CHECK(grid_scan_out.GetPlots()[0].x[5] == 1.5 * 10.0f);
CHECK(grid_scan_out.GetPlots()[0].y[5] == 2.5 * 25.0f);
CHECK(grid_scan_out.GetPlots()[0].x[6] == 2.5 * 10.0f);
CHECK(grid_scan_out.GetPlots()[0].y[6] == 0.5 * 25.0f);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_32", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_32").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxLegacy).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4).BitDepthImage(32).PixelSigned(true)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
pusher.WriterNotificationSocket("ipc://*");
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_1.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQ_crystal_spot_and_index_min_pix_2", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).OverwriteExistingFiles(true)
.FilePrefix("crystal_test_min_pix_2").JungfrauConvPhotonCnt(false).SetFileWriterFormat(
FileWriterFormat::NXmxVDS)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90})
.PixelSigned(true);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {2,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 3;
settings.min_pix_per_spot = 2;
settings.max_pix_per_spot = 200;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg, jpeg_indexed;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_min_pix_2.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
REQUIRE_NOTHROW(jpeg_indexed = service.GetJPEGFromBuffer(jpeg_settings, ImageBuffer::MaxIndexedImage));
REQUIRE(!jpeg_indexed.empty());
CHECK(receiver_out.efficiency == 1.0);
CHECK(receiver_out.status.indexing_rate == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("GenerateResolutionMap") {
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true)
.FilePrefix("crystal_test").JungfrauConvPhotonCnt(false)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
std::vector<float> spot_finder_resolution_map(experiment.GetModulesNum() * RAW_MODULE_SIZE, 1.0);
for (int m = 0; m < experiment.GetModulesNum(); m++)
experiment.CalcSpotFinderResolutionMap(spot_finder_resolution_map.data() + m * RAW_MODULE_SIZE, m);
std::vector<uint32_t> spot_finder_resolution_map_int(spot_finder_resolution_map.size());
for (int i = 0; i < spot_finder_resolution_map.size(); i++)
spot_finder_resolution_map_int[i] = static_cast<uint32_t>(spot_finder_resolution_map[i] * 100);
std::vector<uint32_t> spot_finder_resolution_map_int_conv(experiment.GetPixelsNum(), 0);
RawToConvertedGeometry(experiment, spot_finder_resolution_map_int_conv.data(), spot_finder_resolution_map_int.data());
CompressedImage image(spot_finder_resolution_map_int_conv, experiment.GetXPixelsNum(),
experiment.GetYPixelsNum() );
WriteTIFFToFile("ResolutionMap.tiff", image);
}
TEST_CASE("JFJochIntegrationTest_ZMQ_ROI", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true)
.FilePrefix("crystal_test_roi").JungfrauConvPhotonCnt(false)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90})
.SetFileWriterFormat(FileWriterFormat::NXmxVDS).PixelSigned(true).OverwriteExistingFiles(true);
PixelMask pixel_mask(experiment);
experiment.ROI().SetROI(ROIDefinition{
.boxes = {ROIBox("beam", 100, 120, 20, 30)},
.circles = {ROICircle("roi1", 500, 800, 10)},
.azimuthal = {ROIAzimuthal("roi2", 4.0, 7.0)}
});
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {5,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
uint64_t roi_value = 0;
uint64_t pixels = 0;
for (int y = 20; y <= 30; y++) {
for (int x = 100; x <= 120; x++) {
int16_t val = image_conv[experiment.GetXPixelsNum() * y + x];
if ((val != INT16_MIN) && (val != INT16_MAX)) {
pixels += 1;
roi_value += val;
}
}
}
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
auto &mask = pixel_mask.GetMaskRaw();
uint64_t sat_pixels = 0;
uint64_t err_pixels = 0;
for (int i = 0; i < image_raw_geom.size(); i++) {
if (mask[i] == 0) {
if (image_raw_geom[i] == INT16_MAX)
sat_pixels++;
else if (image_raw_geom[i] == INT16_MIN)
err_pixels++;
}
}
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
ZMQStream2Pusher pusher({"ipc://*"});
ZMQImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
std::string jpeg;
PreviewImageSettings jpeg_settings{
.saturation_value = 10,
.jpeg_quality = 100,
.show_spots = true,
.show_roi = true
};
REQUIRE_NOTHROW(jpeg = service.GetJPEGFromBuffer(jpeg_settings));
REQUIRE(!jpeg.empty());
std::ofstream f("crystal_processing_test_roi.jpeg", std::ios::binary);
f.write(jpeg.data(), jpeg.size());
auto plot = service.GetDataProcessingPlot(PlotRequest{.type = PlotType::ROISum, .binning = 1});
REQUIRE(plot.GetPlots().size() == 3);
CHECK(plot.GetPlots()[0].title == "beam");
REQUIRE(!plot.GetPlots()[0].x.empty());
CHECK(plot.GetPlots()[0].x[0] == 0);
CHECK(plot.GetPlots()[0].y[0] == roi_value);
CHECK(plot.GetPlots()[1].title == "roi1");
CHECK(plot.GetPlots()[2].title == "roi2");
CHECK(receiver_out.status.roi_beam_sum == roi_value);
CHECK(receiver_out.status.roi_beam_npixel == pixels);
REQUIRE(receiver_out.status.error_pixels.has_value());
REQUIRE(receiver_out.status.saturated_pixels.has_value());
CHECK(receiver_out.status.error_pixels.value() == err_pixels);
CHECK(receiver_out.status.saturated_pixels.value() == sat_pixels);
CHECK(receiver_out.efficiency == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_ZMQPreview", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true)
.FilePrefix("").JungfrauConvPhotonCnt(false)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4);
PixelMask pixel_mask(experiment);
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
aq_devices.Add(std::make_unique<HLSSimulatedDevice>(0, 64));
NonePusher pusher;
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
service.PreviewSocket("tcp://0.0.0.0:*", 16);
service.PreviewSocketSettings({.period = std::chrono::seconds(0)}); // Send one message
ZMQSocket sub_socket(ZMQSocketType::Sub);
sub_socket.ReceiveWaterMark(10);
sub_socket.Connect(service.GetPreviewSocketAddress());
sub_socket.SubscribeAll();
sub_socket.ReceiveTimeout(std::chrono::seconds(1)); // Ensure Subscriber is connected
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
CHECK(receiver_out.status.images_collected == 5);
CHECK(receiver_out.status.images_sent == 0);
CHECK(!receiver_out.status.cancelled);
ZMQMessage msg1;
REQUIRE(sub_socket.Receive(msg1, true));
// start message
auto out = CBORStream2Deserialize(msg1.data(), msg1.size());
REQUIRE(out != nullptr);
CHECK(out->msg_type == CBORImageType::START);
REQUIRE(sub_socket.Receive(msg1, true));
out = CBORStream2Deserialize(msg1.data(), msg1.size());
REQUIRE(out != nullptr);
CHECK(out->msg_type == CBORImageType::IMAGE);
// 4 other images
REQUIRE(sub_socket.Receive(msg1, true));
REQUIRE(sub_socket.Receive(msg1, true));
REQUIRE(sub_socket.Receive(msg1, true));
REQUIRE(sub_socket.Receive(msg1, true));
// end message
REQUIRE(sub_socket.Receive(msg1, true));
REQUIRE(!sub_socket.Receive(msg1, true));
}
TEST_CASE("JFJochIntegrationTest_ZMQMetadata", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true)
.FilePrefix("").JungfrauConvPhotonCnt(false)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4);
PixelMask pixel_mask(experiment);
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
aq_devices.Add(std::make_unique<HLSSimulatedDevice>(0, 64));
NonePusher pusher;
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
service.MetadataSocket("tcp://0.0.0.0:*");
service.MetadataSocketSettings({.period = std::chrono::seconds(2500)}); // Send all metadata as one message
ZMQSocket sub_socket(ZMQSocketType::Sub);
sub_socket.ReceiveWaterMark(10);
sub_socket.Connect(service.GetMetadataSocketAddress());
sub_socket.SubscribeAll();
sub_socket.ReceiveTimeout(std::chrono::seconds(1)); // Ensure Subscriber is connected
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
CHECK(receiver_out.status.images_collected == 5);
CHECK(receiver_out.status.images_sent == 0);
CHECK(!receiver_out.status.cancelled);
ZMQMessage msg1;
REQUIRE(sub_socket.Receive(msg1, true));
auto out = CBORStream2Deserialize(msg1.data(), msg1.size());
REQUIRE(out != nullptr);
CHECK(out->msg_type == CBORImageType::START);
REQUIRE(sub_socket.Receive(msg1, true));
out = CBORStream2Deserialize(msg1.data(), msg1.size());
REQUIRE(out != nullptr);
CHECK(out->msg_type == CBORImageType::METADATA);
CHECK(out->metadata);
CHECK(out->metadata->images.size() == 5);
// end message
REQUIRE(sub_socket.Receive(msg1, true));
REQUIRE(!sub_socket.Receive(msg1, true));
}
TEST_CASE("JFJochIntegrationTest_HDF5FilePusher", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true)
.FilePrefix("HDF5FilePusherTest").JungfrauConvPhotonCnt(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4).OverwriteExistingFiles(true);
JFCalibration calibration(experiment);
PixelMask pixel_mask(experiment);
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
aq_devices.Add(std::make_unique<HLSSimulatedDevice>(0, 64));
HDF5FilePusher pusher;
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
service.Start(experiment, pixel_mask, &calibration);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
CHECK(receiver_out.status.images_collected == 5);
CHECK(receiver_out.status.images_sent == 5);
CHECK(!receiver_out.status.cancelled);
}
TEST_CASE("JFJochIntegrationTest_HDF5FilePusher_republish", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
const std::string repub_addr = "tcp://127.0.0.1:9132";
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true)
.FilePrefix("HDF5FilePusherRepublishTest").JungfrauConvPhotonCnt(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.OverwriteExistingFiles(true);
PixelMask pixel_mask(experiment);
AcquisitionDeviceGroup aq_devices;
aq_devices.Add(std::make_unique<HLSSimulatedDevice>(0, 64));
HDF5FilePusher pusher(repub_addr);
ZMQSocket pull_socket(ZMQSocketType::Pull);
pull_socket.ReceiveWaterMark(16);
pull_socket.Connect(repub_addr);
pull_socket.ReceiveTimeout(std::chrono::seconds(1));
std::this_thread::sleep_for(std::chrono::milliseconds(100));
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
CHECK(receiver_out.status.images_collected == 5);
CHECK(receiver_out.status.images_sent == 5);
CHECK(!receiver_out.status.cancelled);
ZMQMessage msg;
REQUIRE(pull_socket.Receive(msg, true));
auto out = CBORStream2Deserialize(msg.data(), msg.size());
REQUIRE(out != nullptr);
CHECK(out->msg_type == CBORImageType::START);
for (int i = 0; i < experiment.GetImageNum(); i++) {
REQUIRE(pull_socket.Receive(msg, true));
out = CBORStream2Deserialize(msg.data(), msg.size());
REQUIRE(out != nullptr);
CHECK(out->msg_type == CBORImageType::IMAGE);
}
REQUIRE(pull_socket.Receive(msg, true));
out = CBORStream2Deserialize(msg.data(), msg.size());
REQUIRE(out != nullptr);
CHECK(out->msg_type == CBORImageType::END);
REQUIRE(!pull_socket.Receive(msg, true));
}
TEST_CASE("JFJochIntegrationTest_HDF5FilePusher_Raw", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF(1));
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true)
.FilePrefix("HDF5FilePusherTest_Raw").OverwriteExistingFiles(true);
experiment.Raw();
PixelMask pixel_mask(experiment);
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
aq_devices.Add(std::make_unique<HLSSimulatedDevice>(0, 64));
HDF5FilePusher pusher;
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
CHECK(receiver_out.status.images_collected == 5);
CHECK(receiver_out.status.images_sent == 5);
CHECK(!receiver_out.status.cancelled);
}
TEST_CASE("JFJochIntegrationTest_TCP_crystal_no_processing", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_tcp_no_processing").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
experiment.SampleTemperature_K(123.0).RingCurrent_mA(115);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
TCPStreamPusher pusher("tcp://127.0.0.1:*", 1);
TCPImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.enable = false;
settings.indexing = true;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
CHECK(!receiver_out.status.indexing_rate);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(receiver_out.writer_err.empty());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
auto ack = pusher.GetImagesWritten();
REQUIRE(ack.has_value());
CHECK(ack == experiment.GetImageNum());
}
TEST_CASE("JFJochIntegrationTest_TCP_crystal_spot_and_index", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_tcp").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
experiment.SampleTemperature_K(123.0).RingCurrent_mA(115);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
TCPStreamPusher pusher("tcp://127.0.0.1:*", 1);
TCPImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(receiver_out.writer_err.empty());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
auto ack = pusher.GetImagesWritten();
REQUIRE(ack.has_value());
CHECK(ack == experiment.GetImageNum());
}
TEST_CASE("JFJochIntegrationTest_TCP_calibration", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2).SaveCalibration(true)
.FilePrefix("calib_integration_test").JungfrauConvPhotonCnt(true).SetFileWriterFormat(FileWriterFormat::NXmxVDS).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
experiment.SampleTemperature_K(123.0).RingCurrent_mA(115);
PixelMask pixel_mask(experiment);
JFCalibration calibration(experiment);
for (int i = 0; i < experiment.GetModulesNum(); i++) {
std::vector<int16_t> pedestal_g0(RAW_MODULE_SIZE, i);
std::vector<int16_t> pedestal_g1(RAW_MODULE_SIZE);
std::vector<int16_t> pedestal_g2(RAW_MODULE_SIZE);
for (int j = 0; j < RAW_MODULE_SIZE; j++) {
pedestal_g1[j] = j % RAW_MODULE_COLS;
pedestal_g2[j] = i * (RAW_MODULE_LINES) + j / RAW_MODULE_COLS; // line number
}
calibration.Pedestal(i,0).LoadPedestal(pedestal_g0);
calibration.Pedestal(i,1).LoadPedestal(pedestal_g1);
calibration.Pedestal(i,2).LoadPedestal(pedestal_g2);
}
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
aq_devices.Add(std::move(test));
TCPStreamPusher pusher("tcp://127.0.0.1:*", 1);
TCPImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
service.Start(experiment, pixel_mask, &calibration);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochIntegrationTest_TCP_crystal_spot_and_index_single_file", "[JFJochReceiver]") {
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const uint16_t nthreads = 4;
DiffractionExperiment experiment(DetJF4M());
experiment.ImagesPerTrigger(5).NumTriggers(1).UseInternalPacketGenerator(true).ImagesPerFile(2)
.FilePrefix("crystal_test_tcp_single_file").JungfrauConvPhotonCnt(false).SetFileWriterFormat(FileWriterFormat::NXmxIntegrated).OverwriteExistingFiles(true)
.DetectorDistance_mm(75).BeamY_pxl(1136).BeamX_pxl(1090).IncidentEnergy_keV(12.4)
.SetUnitCell(UnitCell{.a = 36.9, .b = 78.95, .c = 78.95, .alpha =90, .beta = 90, .gamma = 90});
experiment.SampleTemperature_K(123.0).RingCurrent_mA(115);
PixelMask pixel_mask(experiment);
// Load example image
HDF5ReadOnlyFile data("../../tests/test_data/compression_benchmark.h5");
HDF5DataSet dataset(data, "/entry/data/data");
HDF5DataSpace file_space(dataset);
REQUIRE(file_space.GetDimensions()[2] == experiment.GetXPixelsNum());
REQUIRE(file_space.GetDimensions()[1] == experiment.GetYPixelsNum());
std::vector<int16_t> image_conv (file_space.GetDimensions()[1] * file_space.GetDimensions()[2]);
std::vector<hsize_t> start = {4,0,0};
std::vector<hsize_t> file_size = {1, file_space.GetDimensions()[1], file_space.GetDimensions()[2]};
dataset.ReadVector(image_conv, start, file_size);
std::vector<int16_t> image_raw_geom(experiment.GetModulesNum() * RAW_MODULE_SIZE);
ConvertedToRawGeometry(experiment, image_raw_geom.data(), image_conv.data());
logger.Info("Loaded image");
// Setup acquisition device
AcquisitionDeviceGroup aq_devices;
std::unique_ptr<HLSSimulatedDevice> test = std::make_unique<HLSSimulatedDevice>(0, 64);
for (int m = 0; m < experiment.GetModulesNum(); m++)
test->SetInternalGeneratorFrame((uint16_t *) image_raw_geom.data() + m * RAW_MODULE_SIZE, m);
aq_devices.Add(std::move(test));
TCPStreamPusher pusher("tcp://127.0.0.1:*", 1);
TCPImagePuller puller(pusher.GetAddress()[0]);
StreamWriter writer(logger, puller);
auto writer_future = std::async(std::launch::async, &StreamWriter::Run, &writer);
JFJochReceiverService service(aq_devices, logger, pusher);
service.NumThreads(nthreads);
service.Indexing(experiment.GetIndexingSettings());
// No progress value at the start of measurement
REQUIRE(!service.GetProgress().has_value());
SpotFindingSettings settings = DiffractionExperiment::DefaultDataProcessingSettings();
settings.signal_to_noise_threshold = 2.5;
settings.photon_count_threshold = 5;
settings.min_pix_per_spot = 1;
settings.max_pix_per_spot = 200;
settings.high_resolution_limit = 2.0;
settings.low_resolution_limit = 50.0;
service.SetSpotFindingSettings(settings);
service.Start(experiment, pixel_mask, nullptr);
auto receiver_out = service.Stop();
CHECK(receiver_out.efficiency == 1.0);
REQUIRE(receiver_out.status.indexing_rate);
CHECK(receiver_out.status.indexing_rate.value() == 1.0);
CHECK(receiver_out.status.images_sent == experiment.GetImageNum());
CHECK(receiver_out.writer_err.empty());
CHECK(!receiver_out.status.cancelled);
// No progress value at the end of measurement
REQUIRE(!service.GetProgress().has_value());
REQUIRE_NOTHROW(writer_future.get());
auto ack = pusher.GetImagesWritten();
REQUIRE(ack.has_value());
CHECK(ack == experiment.GetImageNum());
}