Build Packages / Build documentation (push) Successful in 48s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 15m47s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 23m19s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 18m47s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m54s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 11m27s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 26m59s
Build Packages / Unit tests (push) Successful in 1h50m25s
Build Packages / Validation Battery (push) Successful in 46m38s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 32m52s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 32m32s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 28m34s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 29m19s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 30m11s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 33m58s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 31m26s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 34m4s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 32m17s
Build Packages / Generate python client (push) Successful in 32s
Build Packages / Create release (push) Successful in 9s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 31m50s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 19m38s
* jfjoch_broker (experimental): Live Rugnux covers serial collections from DECTRIS detectors; a rotation collection too large for the image buffer falls back to standard per-image processing with a warning, and a stream that ends without its end message finishes the collection instead of hanging it. * jfjoch_broker (experimental): Live Rugnux settings to build the solvent during refinement and to process a rotation collection as stills, a Clear action for the last result, a `_report.txt` for live rotation runs, R-free reported for the refined model, and ice-ring detection on by default for rotation as in the command line. * jfjoch_broker (experimental): More complete and reproducible live serial merges, analysed for twinning, pseudo-translation and anisotropy, with merging statistics by resolution shell in the live status. * jfjoch_writer: Live results with large models are written reliably; update jfjoch_writer together with jfjoch_broker. * Rugnux: Several masters can be processed together with `--force-still`, including grid scans; more robust and faster merging of several sweeps; improved twinning, pseudo-translation, anisotropy, anomalous-signal and space-group decisions. * Rugnux: Molecular replacement places the copies of a multi-copy solution together; the pseudo-translation warning quotes the Patterson peak and is not raised once a model is placed; the report advises on extending the rotation or reorienting the crystal where the coverage would gain from it. * Web frontend: Light and dark modes, a phone layout, a hideable page list, a redesigned Rugnux live page with step-through of the Fo-Fc peaks in the structure view. * jfjoch_viewer: A broker URL given on the command line follows the running collection. Reviewed-on: #97 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
265 lines
11 KiB
C++
265 lines
11 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include "JFJochReceiverStream.h"
|
|
|
|
void JFJochReceiverStream::ApplyStartMessage(DiffractionExperiment &experiment, const StartMessage &msg) {
|
|
// A PSI detector's images are what the experiment set the card up to produce: nothing to learn here.
|
|
if (experiment.GetDetectorType() != DetectorType::DECTRIS)
|
|
return;
|
|
|
|
if ((experiment.GetXPixelsNum() != msg.image_size_x)
|
|
|| (experiment.GetYPixelsNum() != msg.image_size_y))
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
|
|
"Mismatch in detector size");
|
|
if (fabs(experiment.GetPixelSize_mm() - msg.pixel_size_x * 1e3) > 1e-7)
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
|
|
"Mismatch in pixel size");
|
|
|
|
experiment.Detector().Description(msg.detector_description);
|
|
experiment.Detector().SerialNumber(msg.detector_serial_number);
|
|
experiment.Detector().SensorMaterial(msg.sensor_material);
|
|
experiment.Detector().SensorThickness_um(msg.sensor_thickness * 1e6);
|
|
experiment.Detector().SaturationLimit(SaturationLimitFromValue(msg.saturation_value));
|
|
// The outgoing start message is rebuilt from the experiment, NXmx would record these as not applied
|
|
experiment.Detector().CountRateCorrectionApplied(msg.countrate_correction_enabled);
|
|
experiment.Detector().CountRateCorrectionLookupTable(msg.countrate_correction_lookup_table);
|
|
experiment.Detector().FlatfieldApplied(msg.flatfield_enabled);
|
|
experiment.Detector().VirtualPixelInterpolationApplied(msg.virtual_pixel_interpolation_enabled);
|
|
// The mask written with the data is ours, never uploaded to the detector
|
|
experiment.ApplyPixelMask(false);
|
|
// Images are forwarded byte-for-byte, so the stream decides both width and sign
|
|
experiment.Detector().BitDepthImage(msg.bit_depth_image);
|
|
experiment.PixelSigned(msg.pixel_signed);
|
|
}
|
|
|
|
JFJochReceiverStream::JFJochReceiverStream(const DiffractionExperiment &in_experiment,
|
|
const PixelMask &in_pixel_mask,
|
|
ImagePuller &in_image_puller,
|
|
ImagePusher &in_image_pusher,
|
|
Logger &in_logger,
|
|
int64_t in_nthreads,
|
|
const SpotFindingSettings &in_spot_finding_settings,
|
|
JFJochReceiverCurrentStatus &in_current_status,
|
|
JFJochReceiverPlots &in_plots,
|
|
ImageBuffer &in_image_buffer,
|
|
ZMQPreviewSocket *in_zmq_preview_socket,
|
|
ZMQMetadataSocket *in_zmq_metadata_socket)
|
|
: JFJochReceiver(in_experiment,
|
|
in_image_buffer,
|
|
in_image_pusher,
|
|
in_current_status,
|
|
in_plots,
|
|
in_spot_finding_settings,
|
|
in_logger,
|
|
in_pixel_mask,
|
|
in_zmq_preview_socket,
|
|
in_zmq_metadata_socket),
|
|
image_puller(in_image_puller),
|
|
nthreads(in_nthreads),
|
|
measurement_started(1),
|
|
images_started(nthreads) {
|
|
}
|
|
|
|
void JFJochReceiverStream::StartThreads() {
|
|
logger.Debug("Starting {} image threads", nthreads);
|
|
for (int i = 0; i < nthreads; i++)
|
|
image_futures.emplace_back(std::async(std::launch::async, &JFJochReceiverStream::ImageThread, this, i));
|
|
measurement = std::async(std::launch::async, &JFJochReceiverStream::MeasurementThread, this);
|
|
images_started.wait();
|
|
logger.Debug("Image threads ready");
|
|
}
|
|
|
|
void JFJochReceiverStream::JoinFuture(std::future<void> &future) {
|
|
try {
|
|
if (future.valid())
|
|
future.get();
|
|
} catch (const std::exception &e) {
|
|
logger.Error("Receiver teardown failed: {}", e.what());
|
|
} catch (...) {
|
|
logger.Error("Receiver teardown failed with unknown exception");
|
|
}
|
|
}
|
|
|
|
void JFJochReceiverStream::JoinThreads() {
|
|
JoinFuture(measurement);
|
|
// The measurement thread joins them, unless it left with an error
|
|
for (auto &f: image_futures)
|
|
JoinFuture(f);
|
|
}
|
|
|
|
void JFJochReceiverStream::MeasurementThread() {
|
|
try {
|
|
// Once the detector has reported it is done, a start message in flight may still land, then give up
|
|
std::optional<std::chrono::steady_clock::time_point> give_up_at;
|
|
auto msg = image_puller.PollImage();
|
|
|
|
while (!cancelled && (!msg.has_value() || !msg->cbor || !msg->cbor->start_message)) {
|
|
if (detector_finished && !give_up_at)
|
|
give_up_at = std::chrono::steady_clock::now() + StartMessageGrace;
|
|
if (give_up_at && (std::chrono::steady_clock::now() > *give_up_at)) {
|
|
logger.Error("Detector finished, but no start message arrived - abandoning measurement");
|
|
cancelled = true;
|
|
break;
|
|
}
|
|
msg = image_puller.PollImage();
|
|
}
|
|
|
|
if (cancelled) {
|
|
current_status.SetProgress({});
|
|
current_status.SetStatus(GetStatus());
|
|
measurement_started.count_down();
|
|
return; // just quit the function - no measurement, no results
|
|
}
|
|
|
|
last_message = std::chrono::steady_clock::now();
|
|
ApplyStartMessage(experiment, msg->cbor->start_message.value());
|
|
image_buffer.StartMeasurement(experiment); // Only at this point we know bit-depth of the images
|
|
start_time = std::chrono::system_clock::now();
|
|
OnStart(msg->received);
|
|
measurement_started.count_down();
|
|
} catch (const JFJochException &e) {
|
|
logger.ErrorException(e);
|
|
Cancel(e);
|
|
measurement_started.count_down();
|
|
throw;
|
|
} catch (const std::exception &e) {
|
|
logger.Error("Cancelling data collection due to exception: {}", e.what());
|
|
cancelled = true;
|
|
measurement_started.count_down();
|
|
throw;
|
|
}
|
|
|
|
logger.Info("Receiving started");
|
|
|
|
for (auto &f: image_futures)
|
|
f.get();
|
|
|
|
logger.Debug("Image threads finished");
|
|
|
|
FinishCollection();
|
|
}
|
|
|
|
void JFJochReceiverStream::OnStart(std::chrono::steady_clock::time_point received) {
|
|
SendStartMessage();
|
|
}
|
|
|
|
void JFJochReceiverStream::ImageThread(uint32_t id) {
|
|
WaitForImages();
|
|
PollImages(id);
|
|
}
|
|
|
|
void JFJochReceiverStream::WaitForImages() {
|
|
images_started.count_down();
|
|
measurement_started.wait();
|
|
}
|
|
|
|
void JFJochReceiverStream::PollImages(uint32_t id) {
|
|
while (!cancelled && !end_message_received && !end_message_lost) {
|
|
try {
|
|
auto msg = image_puller.PollImage(std::chrono::milliseconds(5));
|
|
|
|
if (!msg.has_value() || !msg->cbor) {
|
|
// Message not received or not parsed. Once the detector has finished, a silent stream means
|
|
// the end message is not coming (lost, or not sent): the collection ends with the images it has.
|
|
if (detector_finished && (std::chrono::steady_clock::now() > last_message.load() + EndMessageGrace)
|
|
&& !end_message_lost.exchange(true))
|
|
logger.Error("Detector finished, but no end message arrived in {} s - ending the collection",
|
|
std::chrono::duration_cast<std::chrono::seconds>(EndMessageGrace).count());
|
|
continue;
|
|
}
|
|
last_message = std::chrono::steady_clock::now();
|
|
if (msg->cbor->end_message) {
|
|
end_adu_histogram = msg->cbor->end_message->adu_histogram;
|
|
end_message_received = true;
|
|
logger.Debug("Thread {} end message received", id);
|
|
} else if (msg->cbor->data_message) {
|
|
DataMessage message = msg->cbor->data_message.value();
|
|
OnImage(id, message, msg->received);
|
|
}
|
|
UpdateMaxDelay(image_puller.GetCurrentFifoUtilization());
|
|
current_status.SetProgress(GetProgress());
|
|
current_status.SetStatus(GetStatus());
|
|
} catch (const JFJochException &e) {
|
|
logger.ErrorException(e);
|
|
Cancel(e);
|
|
} catch (const std::exception &e) {
|
|
logger.Error("Cancelling data collection due to exception: {}", e.what());
|
|
cancelled = true;
|
|
}
|
|
}
|
|
}
|
|
|
|
void JFJochReceiverStream::CheckIfImagesSent() {
|
|
if (!image_buffer.CheckIfBufferReturned(SendFinalizeTimeout)) {
|
|
logger.Error("Send commands not finalized in {} seconds", SendFinalizeTimeout.count());
|
|
throw JFJochException(JFJochExceptionCategory::ZeroMQ, "Send commands not finalized in "
|
|
+ std::to_string(SendFinalizeTimeout.count()) + " seconds");
|
|
}
|
|
}
|
|
|
|
void JFJochReceiverStream::WriterFinalize() {
|
|
if (push_images_to_writer)
|
|
writer_error = image_pusher.Finalize();
|
|
}
|
|
|
|
void JFJochReceiverStream::FinishCollection() {
|
|
current_status.SetProgress(1.0);
|
|
current_status.SetStatus(GetStatus());
|
|
|
|
SendEndMessage();
|
|
CheckIfImagesSent();
|
|
logger.Debug("All images sent");
|
|
|
|
WriterFinalize();
|
|
logger.Debug("Writing process finalized");
|
|
|
|
end_time = std::chrono::system_clock::now();
|
|
|
|
current_status.SetProgress({});
|
|
current_status.SetStatus(GetStatus());
|
|
}
|
|
|
|
void JFJochReceiverStream::StopReceiver() {
|
|
if (measurement.valid()) {
|
|
measurement.get();
|
|
logger.Debug("Receiver stopped");
|
|
}
|
|
}
|
|
|
|
void JFJochReceiverStream::Cancel(bool silent) {
|
|
// A silent cancel is not a request to abort, it says the detector has finished. It bounds the wait for the start
|
|
// and for the end message.
|
|
last_message = std::chrono::steady_clock::now();
|
|
detector_finished = true;
|
|
if (silent)
|
|
return;
|
|
// Once the end message is in (or taken as lost), the images are all there: a cancel only cuts the analysis that
|
|
// goes on after them.
|
|
if (!end_message_received && !end_message_lost)
|
|
JFJochReceiver::Cancel(silent);
|
|
Cut("cancel");
|
|
}
|
|
|
|
void JFJochReceiverStream::AddToEndMessage(EndMessage &message) {
|
|
if (!end_adu_histogram.empty())
|
|
message.adu_histogram = end_adu_histogram;
|
|
}
|
|
|
|
// The collection is counted in images, not frames. When the card sums frames, one image is many frames.
|
|
int64_t JFJochReceiverStream::ExpectedImages() const {
|
|
return experiment.GetImageNum();
|
|
}
|
|
|
|
float JFJochReceiverStream::GetEfficiency() const {
|
|
if (ExpectedImages() == 0)
|
|
return 0;
|
|
return static_cast<float>(collection_efficiency_sum / static_cast<double>(ExpectedImages()));
|
|
}
|
|
|
|
float JFJochReceiverStream::GetProgress() const {
|
|
if (ExpectedImages() == 0)
|
|
return 0.0;
|
|
return static_cast<float>(max_image_number_received) / static_cast<float>(ExpectedImages());
|
|
}
|