Files
Jungfraujoch/tools/jfjoch_stills_gpu_replay.cpp
leonarski_f 4809c1622c
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 16m43s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 28m46s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 32m5s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 39m44s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m7s
Build Packages / Create release (push) Successful in 8s
Build Packages / Build documentation (push) Successful in 49s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m30s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m54s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 11m29s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m51s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m7s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 29m38s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m6s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 29m44s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 33m22s
Build Packages / Generate python client (push) Successful in 47s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 34m6s
Build Packages / Unit tests (push) Successful in 1h2m47s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 35m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 34m8s
Build Packages / Validation Battery (push) Successful in 38m31s
v1.0.0-rc.187 (#98)
* Rugnux: Refinement several times faster, one protocol on the CPU and the GPU, parts of it on the Apple GPU; its random draws are the same on every platform.
* Rugnux: Refinement follows common practice: the free set decides only the restraint weights and the displacement model (isotropic, TLS or anisotropic, by Hamilton's test), everything else is decided on the working set; lower R-free for placed and predicted models.
* Rugnux: Restraint dictionaries via `--restraints` (or among the inputs) and more built-in ligands; a model whose space group is a subgroup of the data's is refined in its own group; reflections plus a model select `--mode refine` on their own.
* Rugnux: `-C` fixes the crystal system on the axes as typed; `-S` alone fixes the space group only, the cell is still found and reduced; a space group or cell stored in the input file is ignored, with a line in the log.
* Rugnux: More reliable multi-sweep merging; `--mode scale` merges several stills runs as one; `_plot.txt` and the report name each input; a second crystal form is reported for stills given a cell.
* jfjoch_broker (experimental): Live Rugnux on FPGA detectors (JUNGFRAU) takes the same path as a DECTRIS stream, rotation and serial alike, model step included, with analysis threads per GPU; frames waiting for the analysis can be viewed; no host stage drops a frame - a source faster than the analysis is held at the card.
* jfjoch_broker: Summation of more than 256 frames per image is split between the card and the host (`summation_mode` is then `hybrid`), and live Rugnux runs on such collections; pedestal runs take a receiver of their own.
* jfjoch_broker (experimental): `stills_downsample` analyses 1 of every N serial images (chosen by a hash; the first 2000 all; the others written without analysis; reset to 1 by `/initialize`); `max_spot_count` left out means 250 for per-image indexing and up to 1000 for live stills; the live status gives the merge before the model work, the fitted resolution and the anomalous peak height of each unexplained blob.
* jfjoch_broker: Without a `pcie` entry the broker runs on a mock of the card instead of the HLS simulation; its spot, ROI and azimuthal results are zero then.
* Web frontend: Redesigned Rugnux live page; Vite 8, TypeScript 6, Mol* 5.13 and dependency security updates.
* Tools: `jfjoch_fpga_stills_test` measures live stills on the cards with a stored dataset; `jfjoch_fpga_test` loses the HLS-simulation option.
* Other: FFTW no longer fetched or linked (the CPU FFT indexer keeps the name `FFTW`); PCIe driver now GPL-2.0-or-later and third-party notices complete; several crash fixes.

Reviewed-on: #98
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-11-19 20:49:28 +01:00

323 lines
17 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
// Replays the frames of a stored serial collection through the broker's live analysis of serial
// frames (StillsGPUAnalysis): the pre-pass on the first images, then every frame analysed with the
// beam centre and mask it decided. Reports the pre-pass, the indexed fraction and the sustained rate, and
// writes the mask the frames were analysed with (<prefix>_mask.u32, uint32, row by row), and per frame its spot
// count, indexed spot count and outcome (<prefix>_images.tsv) and its spots (<prefix>_spots.tsv). With a model (-m), the
// model step of the merges as the broker makes it, and the files the end message would carry: <prefix>.mtz (the
// merge, in the model's frame), <prefix>_maps.mtz, <prefix>_model.cif, <prefix>_refined_model.cif,
// <prefix>_refined_maps.mtz.
#include <atomic>
#include <chrono>
#include <cstring>
#include <fstream>
#include <functional>
#include <iostream>
#include <sstream>
#include <thread>
#include <getopt.h>
#include <memory>
#include <mutex>
#include "../common/ImageBuffer.h"
#include "../common/Logger.h"
#include "../image_analysis/indexing/IndexerThreadPool.h"
#include "../reader/JFJochHDF5Reader.h"
#include "../receiver/StillsGPUAnalysis.h"
#include "../receiver/StillsSelection.h"
#include "../writer/HDF5Objects.h"
#include "../writer/RugnuxModelFile.h"
#include "../writer/WriteModel.h"
#include "../writer/WriteRugnuxFiles.h"
#include "../image_analysis/WriteReflections.h"
struct ReplaySink : public StillsSink {
std::function<void(const StillsSetup &)> ready;
std::function<void(DataMessage &, const AzimuthalIntegrationProfile &)> analysed;
SpotFindingSettings settings;
void StillsReady(const StillsSetup &setup) override { ready(setup); }
void SendAnalysedImage(DataMessage &message, const AzimuthalIntegrationProfile &profile) override { analysed(message, profile); }
SpotFindingSettings AnalysisSettings() override { return settings; }
void StillsSetupFailed(const JFJochException &) override {}
};
void print_usage() {
std::cout << "Usage: jfjoch_stills_gpu_replay [options] <input master>" << std::endl;
std::cout << " -o <prefix> output prefix (default stills_gpu_replay)" << std::endl;
std::cout << " -s <image> first image (default 0)" << std::endl;
std::cout << " -n <images> images to replay (default all)" << std::endl;
std::cout << " -r <Hz> images given per second (default 0 = as fast as they are read)" << std::endl;
std::cout << " -N <num> analysis threads (default 16)" << std::endl;
std::cout << " -b <MiB> image buffer the frames wait in (default 8192)" << std::endl;
std::cout << " -x <px>, -y <px> beam centre, overriding the file" << std::endl;
std::cout << " -C <a,b,c,alpha,beta,gamma> unit cell of the collection" << std::endl;
std::cout << " -F the broker's default fixed-threshold spot finder (default: adaptive, as the receiver)" << std::endl;
std::cout << " -c leave the file's unit cell out of the collection" << std::endl;
std::cout << " -S <group> space group of the collection, the merge's (a space group stored in the file is ignored)" << std::endl;
std::cout << " -M no merge (StillsGPUAnalysis merge = false)" << std::endl;
std::cout << " -e <images> the first images given as empty frames (no diffraction)" << std::endl;
std::cout << " -m <file> model (PDB/mmCIF) placed and refined against the merges" << std::endl;
std::cout << " -p <num> spots kept per image (default: as rugnux live stills where none is given, 1000)" << std::endl;
std::cout << " -k <num> analyse 1 of every <num> images on average, as rugnux live stills_downsample" << std::endl;
std::cout << " -P <num> pre-pass on the first <num> images, as rugnux live stills_prepass_images (default "
<< STILLS_PREPASS_IMAGES << ")" << std::endl;
std::cout << " -D <s> deadline of the model step after the last image (default 60, 0 = none)" << std::endl;
}
int main(int argc, char **argv) {
Logger logger("jfjoch_stills_gpu_replay");
RegisterHDF5Filter();
std::string prefix = "stills_gpu_replay";
int64_t first = 0;
std::optional<int64_t> images;
double rate = 0.0;
int nthreads = 16;
size_t buffer_MiB = 8192;
std::optional<float> beam_x, beam_y;
std::optional<int64_t> max_spots;
std::optional<UnitCell> cell;
bool no_cell = false;
bool fixed_threshold = false;
std::optional<std::string> space_group;
bool merge = true;
int64_t empty_frames = 0;
std::shared_ptr<const RugnuxModel> model;
double deadline_s = 60;
int64_t downsample = 1;
int64_t prepass_images = STILLS_PREPASS_IMAGES;
int opt;
while ((opt = getopt(argc, argv, "o:s:n:r:N:b:x:y:C:cFS:Me:m:D:p:k:P:")) != -1) {
switch (opt) {
case 'o': prefix = optarg; break;
case 's': first = atoll(optarg); break;
case 'n': images = atoll(optarg); break;
case 'r': rate = atof(optarg); break;
case 'N': nthreads = atoi(optarg); break;
case 'b': buffer_MiB = atoll(optarg); break;
case 'x': beam_x = atof(optarg); break;
case 'y': beam_y = atof(optarg); break;
case 'c': no_cell = true; break;
case 'F': fixed_threshold = true; break;
case 'S': space_group = optarg; break;
case 'M': merge = false; break;
case 'p': max_spots = atoll(optarg); break;
case 'e': empty_frames = atoll(optarg); break;
case 'm': model = std::make_shared<const RugnuxModel>(ReadRugnuxModelFile(optarg)); break;
case 'D': deadline_s = atof(optarg); break;
case 'k': downsample = atoll(optarg); break;
case 'P': prepass_images = atoll(optarg); break;
case 'C': {
UnitCell c{};
if (sscanf(optarg, "%f,%f,%f,%f,%f,%f", &c.a, &c.b, &c.c, &c.alpha, &c.beta, &c.gamma) != 6) {
print_usage();
return EXIT_FAILURE;
}
cell = c;
break;
}
default: print_usage(); return EXIT_FAILURE;
}
}
if (optind + 1 != argc) {
print_usage();
return EXIT_FAILURE;
}
JFJochHDF5Reader reader;
reader.ReadFile(argv[optind]);
const auto dataset = reader.GetDataset();
const int64_t n = std::min<int64_t>(images.value_or(reader.GetNumberOfImages() - first),
reader.GetNumberOfImages() - first);
const auto stored = reader.GetStoredPixelFormat();
DiffractionExperiment x(dataset->experiment);
x.ImagesPerTrigger(n).NumTriggers(1);
if (beam_x)
x.BeamX_pxl(*beam_x);
if (beam_y)
x.BeamY_pxl(*beam_y);
if (no_cell) {
// Indexed de novo, as the broker does where no cell is stated.
x.SetUnitCell(std::nullopt);
IndexingSettings indexing = x.GetIndexingSettings();
indexing.Algorithm(IndexingAlgorithmEnum::Auto);
x.ImportIndexingSettings(indexing);
}
if (cell)
x.SetUnitCell(cell);
if (max_spots)
x.MaxSpotCount(*max_spots);
// A space group stored in the file is ignored: only -S gives one, as the user gives it for a collection
if (!space_group && x.GetGemmiSpaceGroup())
logger.Info("Space group {} stored in the file is ignored; -S gives the collection one", x.GetGemmiSpaceGroup()->hm);
x.SetSpaceGroup(space_group ? std::optional(*gemmi::find_spacegroup_by_name(*space_group)) : std::nullopt);
const PixelMask &mask = *dataset->pixel_mask;
StartMessage start;
x.FillMessage(start);
start.number_of_images = n;
start.bit_depth_image = stored.bit_depth;
start.pixel_signed = stored.is_signed;
start.pixel_mask["default"] = mask.GetMask(x);
SpotFindingSettings spot_finding = DiffractionExperiment::DefaultDataProcessingSettings();
// As JFJochReceiverFPGA analyses converted frames.
if (!fixed_threshold) {
spot_finding.adaptive_threshold = true;
spot_finding.min_pix_per_spot = std::nullopt;
}
IndexerThreadPool pool(x.GetIndexingSettings());
// The frames wait in an image buffer, as in the receiver: a slot each, at most half of the slots.
ImageBuffer buffer(buffer_MiB * 1024 * 1024);
buffer.StartMeasurement(x);
const int64_t slots = buffer.GetStatus().total_slots;
try {
CheckStillsPrePassFits(slots, x.GetImageBufferLocationSize(), std::min(prepass_images, n));
} catch (const JFJochException &e) {
logger.Error("{}", e.what());
return EXIT_FAILURE;
}
std::atomic<int64_t> analysed = 0, indexed = 0, spots = 0, given = 0;
std::atomic<bool> caught_up = false;
double buffer_full_s = 0.0;
std::atomic<double> processing_s = 0.0;
std::chrono::steady_clock::time_point t_ready;
const auto t0 = std::chrono::steady_clock::now();
{
// As the receiver: half the buffer for the frames waiting, the other half for those sent.
std::mutex run_profile_mutex;
std::unique_ptr<AzimuthalIntegrationProfile> run_profile;
RugnuxLiveStatus status; // nobody reads it here
ReplaySink sink;
std::mutex image_log_mutex;
std::ofstream image_log(prefix + "_images.tsv"), spot_log(prefix + "_spots.tsv");
image_log << "image\tspots\tindexed_spots\tindexed\n";
spot_log << "image\tx\ty\tintensity\tindexed\n";
sink.ready = [&](const StillsSetup &setup) {
t_ready = std::chrono::steady_clock::now();
run_profile = std::make_unique<AzimuthalIntegrationProfile>(*setup.mapping);
const auto &m = setup.mask.GetMask();
std::ofstream f(prefix + "_mask.u32", std::ios::binary);
f.write(reinterpret_cast<const char *>(m.data()), m.size() * sizeof(uint32_t));
logger.Info("Ready after {:.1f} s: beam centre {:.2f} {:.2f}",
std::chrono::duration<double>(t_ready - t0).count(),
setup.experiment.GetBeamX_pxl(), setup.experiment.GetBeamY_pxl());
};
sink.analysed = [&](DataMessage &m, const AzimuthalIntegrationProfile &profile) {
{
// One line per frame and per spot, to compare frame by frame with an offline run's _process.h5
std::unique_lock ul(image_log_mutex);
image_log << m.number << '\t' << m.spots.size() << '\t' << m.spot_count_indexed.value_or(0) << '\t'
<< m.indexing_result.value_or(false) << '\n';
for (const auto &s: m.spots)
spot_log << m.number << '\t' << s.x << '\t' << s.y << '\t' << s.intensity << '\t' << s.indexed
<< '\n';
}
{
// As the receiver's plots do: the frames' profiles add up over the run.
std::unique_lock ul(run_profile_mutex);
*run_profile += profile;
}
// The analysis has caught up with the frames given.
if (++analysed + nthreads >= given && !caught_up.exchange(true))
logger.Info("Analysis caught up {:.1f} s after it started, {} frames analysed",
std::chrono::duration<double>(std::chrono::steady_clock::now()
- t_ready).count(), analysed.load());
spots += static_cast<int64_t>(m.spots.size());
processing_s += m.processing_time_s.value_or(0.0f);
if (m.indexing_result.value_or(false))
++indexed;
};
sink.settings = spot_finding;
StillsGPUAnalysis gpu(x, mask, start, &pool, nthreads, prepass_images, logger, sink, status,
merge, model, deadline_s);
JFJochReaderRawImage raw;
std::vector<uint8_t> empty;
for (int64_t i = 0; i < n; i++) {
if (!StillsImageSelected(i, downsample, prepass_images))
continue;
if (rate > 0)
std::this_thread::sleep_until(t0 + std::chrono::duration<double>(static_cast<double>(i) / rate));
reader.ReadRawImage(first + i, raw);
if (i < empty_frames) {
// An empty frame of the same size and pixel type: no compression, every pixel zero.
empty.assign(raw.image.GetUncompressedSize(), 0);
raw.image = CompressedImage(empty.data(), empty.size(), raw.image.GetWidth(), raw.image.GetHeight(),
raw.image.GetMode());
}
auto held = std::make_shared<HeldImage>();
// As the receiver: a slot each, and the frame waits for room where there is none.
const size_t size = raw.image.GetCompressedSize();
const auto t_slot = std::chrono::steady_clock::now();
held->slot = buffer.GetImageSlot(slots - slots / 2);
while (!held->slot) {
std::this_thread::sleep_for(std::chrono::milliseconds(1));
held->slot = buffer.GetImageSlot(slots - slots / 2);
}
buffer_full_s += std::chrono::duration<double>(std::chrono::steady_clock::now() - t_slot).count();
auto data = static_cast<uint8_t *>(held->slot->GetImage());
memcpy(data, raw.image.GetCompressed(), size);
held->message.number = i;
held->message.image = CompressedImage(data, size,
raw.image.GetWidth(), raw.image.GetHeight(), raw.image.GetMode(),
raw.image.GetCompressionAlgorithm());
held->message.image_collection_efficiency = 1.0f;
gpu.Hold(held);
++given;
}
const auto t_given = std::chrono::steady_clock::now();
// The replay waits for a slot where a receiver would drop the frame.
logger.Info("Buffer full for {:.1f} s: at -r {} Hz about {:.0f} frames would be dropped", buffer_full_s, rate,
buffer_full_s * rate);
logger.Info("Gave {} images in {:.1f} s", n, std::chrono::duration<double>(t_given - t0).count());
gpu.Finish();
if (const auto c = gpu.GetConsensusUnitCell())
logger.Info("Consensus cell {:.3f} {:.3f} {:.3f} {:.2f} {:.2f} {:.2f}", c->a, c->b, c->c, c->alpha, c->beta,
c->gamma);
if (const auto &m = gpu.GetMerged()) {
const auto &o = m->merge.statistics.overall;
logger.Info("Merge: {} crystals, {} unique to {:.2f} A, CC1/2 {:.4f}, R_split {:.4f}, <I/sigma> {:.2f}, "
"{:.2f} s after the last frame was analysed", m->images, o.unique_reflections, o.d_min,
o.cc_half, o.r_split, o.mean_i_over_sigma, m->seconds);
std::ostringstream os;
os << m->merge.statistics;
logger.Info("Merging statistics:\n{}", os.str());
}
logger.Info("Last image given -> merge and model step done: {:.2f} s", std::chrono::duration<double>(
std::chrono::steady_clock::now() - t_given).count());
if (const auto &mx = gpu.GetModel()) {
DiffractionExperiment mx_x(gpu.GetMergeExperiment());
mx_x.SetSpaceGroup(mx->space_group);
WriteMtz(MergedMtzTable(mx->merged, mx->cell, mx_x, OutputProvenance{}), prefix + ".mtz");
if (mx->maps)
WriteMtz(mx->maps->coefficients, prefix + "_maps.mtz");
if (mx->placed_model)
WritePlacedModel(*mx->placed_model->model, mx->cell, mx->space_group, prefix, logger);
if (const auto &rf = mx->refinement; rf && rf->ok && rf->model) {
WritePlacedModel(*rf->model, mx->cell, mx->space_group, prefix + "_refined", logger);
if (rf->maps)
WriteMtz(rf->maps->coefficients, prefix + "_refined_maps.mtz");
}
}
}
const auto t1 = std::chrono::steady_clock::now();
const double after_ready = std::chrono::duration<double>(t1 - t_ready).count();
logger.Info("Analysed {} images, {} indexed ({:.1f}%); {:.1f} s after the pre-pass = {:.1f} Hz; total {:.1f} s",
analysed.load(), indexed.load(), 100.0 * indexed / std::max<int64_t>(1, analysed), after_ready,
analysed / after_ready, std::chrono::duration<double>(t1 - t0).count());
logger.Info("Per image: {:.1f} ms analysis (one thread), {:.1f} spots",
1e3 * processing_s / std::max<int64_t>(1, analysed),
static_cast<double>(spots) / std::max<int64_t>(1, analysed));
return EXIT_SUCCESS;
}