v1.0.0-rc.173 (#83)
Build Packages / Create release (push) Successful in 24s
Build Packages / build:viewer:macos-arm64:nocuda (push) Successful in 3m29s
Build Packages / build:rugnux:macos-arm64:nocuda (push) Successful in 2m43s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 8m27s
Build Packages / build:rugnux:linux-x86_64:cuda (push) Successful in 9m53s
Build Packages / build:viewer:linux-x86_64:nocuda (push) Successful in 9m58s
Build Packages / build:viewer:linux-x86_64:cuda (push) Successful in 11m22s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 13m39s
Build Packages / build:viewer:windows-x86_64:nocuda (push) Successful in 18m37s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 16m32s
Build Packages / build:viewer:windows-x86_64:cuda (push) Successful in 24m11s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 25m30s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 19m3s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 20m23s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 19m41s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / Build documentation (push) Successful in 1m16s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 21m0s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 18m38s
Build Packages / build:rugnux:windows-x86_64:cuda (push) Successful in 14m33s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 17m55s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 20m50s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 18m38s
Build Packages / Unit tests (push) Successful in 1h46m14s

* jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports.
* jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls.
* Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results.
* Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable.
* Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate.
* Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do.
* Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence.
* Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags.
* Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check.
* Rugnux: Clear error messages when a data set needs more GPU or host memory than is available.

Reviewed-on: #83
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This commit was merged in pull request #83.
This commit is contained in:
leonarski_f authored and leonarski_f committed 2026-09-29 15:57:32 +02:00
1 parent 6dfe065365
commit 84228bf8be
452 files changed
+23762 -3779

No files matched your search

+82 -15
View File
@@ -50,7 +50,9 @@ MXAnalysisWithoutFPGA::MXAnalysisWithoutFPGA(const DiffractionExperiment &in_exp
if (get_gpu_count() == 0) {
#endif
preprocessor_buffer = std::make_unique<ImagePreprocessorBuffer>(experiment.GetPixelsNum());
preprocessor = std::make_unique<ImagePreprocessorCPU>(in_experiment, in_mask);
auto cpu_preprocessor = std::make_unique<ImagePreprocessorCPU>(in_experiment, in_mask);
preprocessor_cpu = cpu_preprocessor.get();
preprocessor = std::move(cpu_preprocessor);
bragg_engine = std::make_unique<BraggIntegrationEngineCPU>(in_experiment);
if (experiment.ROI().size() >= 1)
roi = std::make_unique<ROIIntegrationCPU>(experiment);
@@ -80,8 +82,20 @@ MXAnalysisWithoutFPGA::MXAnalysisWithoutFPGA(const DiffractionExperiment &in_exp
}
}
#endif
if (!adaptiveSpotFinder)
adaptiveSpotFinder = std::make_unique<AdaptiveSpotFinderCPU>(integration);
if (!adaptiveSpotFinder) {
auto cpu = std::make_unique<AdaptiveSpotFinderCPU>(integration);
// Where the azimuthal integration is the CPU engine too, the finder's first ring pass is the
// same pass over the same pixels, and gives the profile (see AdaptiveSpotFinderCPU).
bool cpu_azint = true;
#ifdef JFJOCH_USE_CUDA
cpu_azint = !stream;
#endif
if (cpu_azint) {
cpu->FuseAzimuthalIntegration(true);
fused_adaptive_cpu = cpu.get();
}
adaptiveSpotFinder = std::move(cpu);
}
}
void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
@@ -105,7 +119,8 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
// The device route must never be the reason a frame fails: whatever it could not handle, the
// host decoder gets its turn. If the data really is bad the host throws too and the caller
// sees the same error it saw before any of this existed - but a GPU-side problem costs speed
// rather than the acquisition.
// rather than the acquisition. Unless the context is lost, which the host route cannot survive.
cuda_throw_if_context_lost();
spdlog::warn("Device decoding failed ({}), falling back to host decompression", e.what());
decoded_on_device = false;
// Handled here, so it must not stay behind as the thread's last error: the host route below
@@ -115,7 +130,23 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
}
const auto compression_end_time = std::chrono::steady_clock::now();
if (!decoded_on_device) {
// The fused GPU engine (rugnux offline, GPU, adaptive detection) produces the azimuthal profile as
// a byproduct of spot finding, so the separate azint pass is skipped in that case and the profile is
// lifted out of the finder below.
const bool fused = enable_fused_adaptive_gpu && spot_finding_settings.enable
&& spot_finding_settings.adaptive_threshold && fused_adaptive != nullptr;
const bool fused_cpu = spot_finding_settings.enable && spot_finding_settings.adaptive_threshold
&& fused_adaptive_cpu != nullptr;
if (preprocessor_cpu) {
const auto preprocessing_start_time = std::chrono::steady_clock::now();
ret = PreprocessCPU(output.image, fused_cpu);
const auto preprocessing_end_time = std::chrono::steady_clock::now();
output.preprocessing_time_s = std::chrono::duration<float>(preprocessing_end_time - preprocessing_start_time).count();
// Decoded inside the preprocessing pass, whose time includes it.
if (output.image.GetCompressionAlgorithm() != CompressionAlgorithm::NO_COMPRESSION)
output.compression_time_s = 0.0f;
} else if (!decoded_on_device) {
const uint8_t *image_ptr = Decompress(output.image);
const auto decompressed_time = std::chrono::steady_clock::now();
if (output.image.GetCompressionAlgorithm() != CompressionAlgorithm::NO_COMPRESSION)
@@ -136,13 +167,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
output.preprocessing_time_s = total_s - decompress_s;
}
// The fused GPU engine (rugnux offline, GPU, adaptive detection) produces the azimuthal profile as
// a byproduct of spot finding, so the separate azint pass is skipped in that case and the profile is
// lifted out of the finder below.
const bool fused = enable_fused_adaptive_gpu && spot_finding_settings.enable
&& spot_finding_settings.adaptive_threshold && fused_adaptive != nullptr;
if (!fused) {
if (!fused && !fused_cpu) {
const auto azint_start_time = std::chrono::steady_clock::now();
AzInt().Run(*preprocessor_buffer, profile);
const auto azint_end_time = std::chrono::steady_clock::now();
@@ -165,6 +190,19 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
int64_t image_number) {
return bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number);
};
// The supercell probe integrates on the same engine, but what the engine counts is the run's
// measurement of its own stencil, and the probe's reflections are not the run's: keep them out.
const auto probe_integrate_fn = [this](const std::vector<Reflection> &predicted, size_t npredicted,
int64_t image_number) {
const BraggIntegrationCounts before = bragg_engine->Counts();
auto ret = bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number);
const BraggIntegrationCounts after = bragg_engine->Counts();
probe_counts.predicted += after.predicted - before.predicted;
probe_counts.bkg_starved += after.bkg_starved - before.bkg_starved;
probe_counts.bkg_starved_by_neighbour += after.bkg_starved_by_neighbour - before.bkg_starved_by_neighbour;
probe_counts.profile_fallback += after.profile_fallback - before.profile_fallback;
return ret;
};
// The radial background correction has to be decided BEFORE this image is integrated, so the
// ice score is taken here rather than with the other per-image quantities at the end of the
@@ -240,7 +278,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
s.min_pix_per_spot = best_mp;
SpotAnalyze(experiment, s, best_spots, output);
decide_radial_background();
indexer.ProcessImage(output, s, *prediction, integrate_fn);
indexer.ProcessImage(output, s, *prediction, integrate_fn, probe_integrate_fn);
indexing_time_s += output.indexing_time_s.value_or(0.0f);
}
// Each indexer call reports only its own time, so the escalation's total is summed here.
@@ -253,7 +291,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
output.spot_finding_time_s = std::chrono::duration<float>(std::chrono::steady_clock::now() - spot_finding_start_time).count();
decide_radial_background();
if (spot_finding_settings.indexing)
indexer.ProcessImage(output, spot_finding_settings, *prediction, integrate_fn);
indexer.ProcessImage(output, spot_finding_settings, *prediction, integrate_fn, probe_integrate_fn);
}
// Recorded whichever way the frame went. A frame holding StrongPixelLimit of them is given up
@@ -270,6 +308,10 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
output.azint_time_s = 0.0f;
}
#endif
if (fused_cpu) {
fused_adaptive_cpu->GetProfile(profile);
output.azint_time_s = 0.0f;
}
}
output.max_viable_pixel_value = ret.max_value;
@@ -365,6 +407,24 @@ const uint8_t *MXAnalysisWithoutFPGA::Decompress(const CompressedImage &image) {
return image.GetUncompressedPtr(decompression_buffer);
}
ImageStatistics MXAnalysisWithoutFPGA::PreprocessCPU(const CompressedImage &image, bool ring_pass) {
ImageStatistics ret{};
if (ring_pass)
fused_adaptive_cpu->BeginRings();
const auto process_block = [&](size_t first, const uint8_t *data, size_t n) {
preprocessor_cpu->AnalyzeBlock(*preprocessor_buffer, first, data, n, image.GetMode(), ret);
if (ring_pass)
fused_adaptive_cpu->AccumulateRingsBlock(*preprocessor_buffer, first, n);
};
if (JFJochIsBitshuffle(image.GetCompressionAlgorithm()))
JFJochDecompressBlocks(image.GetCompressionAlgorithm(), image.GetCompressed(), image.GetCompressedSize(),
image.GetWidth() * image.GetHeight() * image.GetNumChannels(), image.GetByteDepth(),
process_block);
else
process_block(0, Decompress(image), npixels);
return ret;
}
void MXAnalysisWithoutFPGA::RunROIOnly(DataMessage &output) {
output.roi.clear();
if (roi)
@@ -372,7 +432,14 @@ void MXAnalysisWithoutFPGA::RunROIOnly(DataMessage &output) {
}
BraggIntegrationCounts MXAnalysisWithoutFPGA::BraggCounts() const {
return bragg_engine ? bragg_engine->Counts() : BraggIntegrationCounts{};
if (!bragg_engine)
return {};
BraggIntegrationCounts c = bragg_engine->Counts();
c.predicted -= probe_counts.predicted;
c.bkg_starved -= probe_counts.bkg_starved;
c.bkg_starved_by_neighbour -= probe_counts.bkg_starved_by_neighbour;
c.profile_fallback -= probe_counts.profile_fallback;
return c;
}
void MXAnalysisWithoutFPGA::UpdateMaskResolution(const SpotFindingSettings &settings) {