leonarski_fandClaude Opus 5
bef0c6f390
The offline rotation pipeline held every partial observation in up to three forms at once at its peak: the per-image Reflection vectors (88 B/obs), RotationScaleMerge's flat Obs array plus its full-length GPU staging (80 + 57 B/obs), and the geometry post-refinement's own gather (2 x 32 B/obs while it bucket-sorts). On a rotation crystal with a ~640 A axis, 3600 finely-sliced images and 95.7 M first-pass partials that peaked at 28-30 GB of RSS and OOMed a 32 GB machine. Four reductions, none of which changes an output byte: * The post-refine gather scatters each partial straight into its h bucket (histogram pass first), so the gathered-order copy of the whole array - one of two full-size copies alive through the sort - no longer exists. * The gather is split from the fit (GatherPostRefineObservations), and the geometry pre-pass takes it right after Ingest, handing back each image's reflection vector as it is read: nothing later in the pre-pass reads them (its reflections are never written, and the scaling engine reads back only per-frame fields). A run with a reference MTZ keeps them, since resolving the indexing ambiguity re-reads them. * The pre-pass frees the whole RotationScaleMerge engine - about twice the partial payload again - before the post-refinement fit, which sat exactly on the pass's peak. The canonical pass keeps everything, as before. * Ingest's transients are bounded: the sort keys are consumed before the flat Obs array is built instead of coexisting with it, and the GPU upload stages bounded slices (under half a GB) instead of fourteen full-length arrays at once (5 GB on that crystal). Measured there: peak RSS 28.3 / 30.0 GB (two baseline runs) -> 20.3-22.3 GB (four runs of the fix), wall clock unchanged within run-to-run variance, and byte-identical output - merged/unmerged/P1 MTZ data blocks, .hkl, .cif, per-image table and report all equal, on this crystal and on two ordinary rotation sentinels (which also keep their wall clock and drop slightly in peak). What remains of the peak is the canonical pass, which cannot hand its reflections back (they are re-seated and written); going lower means removing the Reflection/Obs duplication itself, which this deliberately does not attempt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Jungfraujoch
Application to receive data from the PSI JUNGFRAU and EIGER detectors.
All documentation is now placed in docs/ subdirectory and for the current version hosted on Jungfraujoch Read The Docs page.