mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 68 Wiki Activity

Build the GPU engines a worker never uses on first use, not always

Browse Source 
Every worker thread built a full set of analysis engines. Two of them are never
asked for on the offline path: the fixed-threshold spot finder, because
detection is adaptive by default, and the azimuthal integrator, because the
fused adaptive finder produces the profile as a by-product. They are still
needed elsewhere - the broker defaults to non-adaptive detection, and
--no-adaptive-spots asks for the finder - so they are built on first use rather
than removed. A lazily built finder takes the current resolution mask on
construction; without that it would find spots outside the limits it was never
told about.

The bitshuffle decoder sized its output buffer for the widest pixel type there
is rather than the one the images actually have, holding a second full frame per
worker on 16-bit data. It is sized from the image now and grows if a later frame
needs more.

The shared-table checksum runs over eight interleaved lanes. FNV's multiply is a
loop-carried dependency, so one chain retires a byte every few cycles whatever
memory bandwidth is spare, and every worker hashes tens of megabytes of geometry
tables as it builds its engines - about 5% of all CPU samples on a 16M-pixel
detector.

Measured on a 16M-pixel rotation dataset: cudaMalloc 11314 -> 9474 calls and,
with cudaFree, 117 s -> 78 s of aggregate thread time; both synchronise the
whole device, so that time is spent blocking every other worker. Whole battery
15m32s -> 12m30s.

Data quality against main, over 24 crystals and eight statistics each: the same
space group on all 24, and every difference smaller than what two runs of an
IDENTICAL binary produce (measured: 13 of 24 crystals reproduce exactly run to
run, worst R_meas swing 5.5 points, against 4.6 points for main vs this branch).
The float atomics in the reductions have always made this so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
jungfrau
2026-08-15 21:42:41 -04:00
co-authored by Claude Opus 5
parent eb634400ea
commit 2cbb3fc8b4
6 changed files with 84 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
image_analysis/MXAnalysisWithoutFPGA.cpp
+34 -7
View File
@@ -50,8 +50,6 @@ MXAnalysisWithoutFPGA::MXAnalysisWithoutFPGA(const DiffractionExperiment &in_exp
if (get_gpu_count() == 0) {
#endif
preprocessor_buffer = std::make_unique<ImagePreprocessorBuffer>(experiment.GetPixelsNum());
spotFinder = std::make_unique<ImageSpotFinderCPU>(experiment.GetXPixelsNum(), experiment.GetYPixelsNum());
azint = std::make_unique<AzIntEngineCPU>(integration);
preprocessor = std::make_unique<ImagePreprocessorCPU>(in_experiment, in_mask);
bragg_engine = std::make_unique<BraggIntegrationEngineCPU>(in_experiment);
if (experiment.ROI().size() >= 1)
@@ -70,8 +68,6 @@ MXAnalysisWithoutFPGA::MXAnalysisWithoutFPGA(const DiffractionExperiment &in_exp
// enable_fused_adaptive_gpu = true, so on the GPU path the copy is off in practice.
preprocessor = std::make_unique<ImagePreprocessorGPU>(in_experiment, in_mask, stream,
/*copy_image_to_host=*/!enable_fused_adaptive_gpu);
spotFinder = std::make_unique<ImageSpotFinderGPU>(experiment.GetXPixelsNum(), experiment.GetYPixelsNum(), stream);
azint = std::make_unique<AzIntEngineGPU>(integration, stream);
bragg_engine = std::make_unique<BraggIntegrationEngineGPU>(in_experiment, stream);
if (experiment.ROI().size() >= 1)
roi = std::make_unique<ROIIntegrationGPU>(experiment, stream);
@@ -144,7 +140,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
if (!fused) {
const auto azint_start_time = std::chrono::steady_clock::now();
azint->Run(*preprocessor_buffer, profile);
AzInt().Run(*preprocessor_buffer, profile);
const auto azint_end_time = std::chrono::steady_clock::now();
output.azint_time_s = std::chrono::duration<float>(azint_end_time - azint_start_time).count();
}
@@ -160,7 +156,7 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
ImageSpotFinder &finder = spot_finding_settings.adaptive_threshold
? static_cast<ImageSpotFinder &>(*adaptiveSpotFinder)
: *spotFinder;
: FixedThresholdFinder();
const auto integrate_fn = [this](const std::vector<Reflection> &predicted, size_t npredicted,
int64_t image_number) {
return bragg_engine->Run(*preprocessor_buffer, predicted, npredicted, image_number);
@@ -293,6 +289,36 @@ void MXAnalysisWithoutFPGA::Analyze(DataMessage &output,
integration.Settings(), spot_finding_settings.ice_ring_width_Q_recipA);
}
ImageSpotFinder &MXAnalysisWithoutFPGA::FixedThresholdFinder() {
if (!spotFinder) {
#ifdef JFJOCH_USE_CUDA
if (stream)
spotFinder = std::make_unique<ImageSpotFinderGPU>(experiment.GetXPixelsNum(),
experiment.GetYPixelsNum(), stream);
else
#endif
spotFinder = std::make_unique<ImageSpotFinderCPU>(experiment.GetXPixelsNum(),
experiment.GetYPixelsNum());
// It missed every mask update that happened before it existed, so it takes the current one
// now. Without this it would find spots outside the resolution limits.
if (mask_high_res.has_value() || mask_low_res.has_value())
spotFinder->SetResolutionMask(mask_resolution);
}
return *spotFinder;
}
AzIntEngine &MXAnalysisWithoutFPGA::AzInt() {
if (!azint) {
#ifdef JFJOCH_USE_CUDA
if (stream)
azint = std::make_unique<AzIntEngineGPU>(integration, stream);
else
#endif
azint = std::make_unique<AzIntEngineCPU>(integration);
}
return *azint;
}
void MXAnalysisWithoutFPGA::RebuildROI() {
if (experiment.ROI().empty()) {
roi.reset();
@@ -346,6 +372,7 @@ void MXAnalysisWithoutFPGA::UpdateMaskResolution(const SpotFindingSettings &sett
// The finders keep their own copy (the GPU ones a bit-packed device copy), so the mask is handed
// over here - when the limits change - rather than with every image.
spotFinder->SetResolutionMask(mask_resolution);
if (spotFinder)
spotFinder->SetResolutionMask(mask_resolution);
adaptiveSpotFinder->SetResolutionMask(mask_resolution);
}