v1.0.0.rc-162 (#72)

**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/WriteReflections.cpp

@@ -5,6 +5,7 @@
 #include "scale_merge/Merge.h"
 #include "scale_merge/HKLKey.h"
 #include "scale_merge/TwinningAnalysis.h"
+#include "../common/ParallelFor.h"
 
 #include <algorithm>
 #include <cmath>
@@ -150,7 +151,8 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
                            const MergeStatistics &statistics,
                            const ErrorModelReport &error_model,
                            const TwinningAnalysisResult &twinning,
-                           const std::string &filename) {
+                           const std::string &filename,
+                           size_t nthreads) {
 
     std::ofstream out(filename);
     if (!out)
@@ -332,25 +334,50 @@ void WriteMmcifReflections(const std::vector<MergedReflection> &reflections,
     out << "_refln.status_free\n";
     out << "_refln.status\n";
 
-    for (const auto& r : reflections) {
-        out << std::setw(5) << r.h << " "
-            << std::setw(5) << r.k << " "
-            << std::setw(5) << r.l << " "
-            << std::setw(14) << Fmt(r.I, 4) << " "
-            << std::setw(14) << Fmt(r.sigma, 4) << " "
-            << std::setw(14) << Fmt(r.I_plus, 4) << " "
-            << std::setw(14) << Fmt(r.sigma_plus, 4) << " "
-            << std::setw(14) << Fmt(r.I_minus, 4) << " "
-            << std::setw(14) << Fmt(r.sigma_minus, 4) << " "
-            << std::setw(14) << Fmt(r.F, 4) << " "
-            << std::setw(14) << Fmt(r.sigmaF, 4) << " "
-            << std::setw(14) << Fmt(r.F_plus, 4) << " "
-            << std::setw(14) << Fmt(r.sigmaF_plus, 4) << " "
-            << std::setw(14) << Fmt(r.F_minus, 4) << " "
-            << std::setw(14) << Fmt(r.sigmaF_minus, 4) << " "
-            << (r.rfree_flag ? 1 : 0) << " "
-            << "o"  // 'o' = observed
-            << "\n";
+    // One row per unique reflection, twelve formatted floats each - tens of megabytes on a crowded
+    // crystal, and the largest single-threaded stretch left in a run. Nothing about a row depends on
+    // any other, so each worker formats its own block into its own string and the blocks go to the
+    // file in order. The columns are written exactly as the stream wrote them: the same "%.4f" (or
+    // "?" where the value is not finite), right-aligned in the same width.
+    {
+        const size_t nrow = reflections.size();
+        const size_t nw = std::max<size_t>(nthreads, 1);
+        const int nch = static_cast<int>(ThreadsForWork(nrow, nw, 4096));
+        std::vector<std::string> block(nch);
+        ParallelChunks(nch, nw, [&](int tlo, int thi) {
+            for (int t = tlo; t < thi; ++t) {
+                const size_t lo = nrow * t / nch, hi = nrow * (t + 1) / nch;
+                std::string &s = block[t];
+                s.reserve((hi - lo) * 208);
+                const auto column = [&s](const std::string &v, size_t width) {
+                    if (v.size() < width) s.append(width - v.size(), ' ');
+                    s.append(v);
+                    s.push_back(' ');
+                };
+                for (size_t i = lo; i < hi; ++i) {
+                    const auto &r = reflections[i];
+                    column(std::to_string(r.h), 5);
+                    column(std::to_string(r.k), 5);
+                    column(std::to_string(r.l), 5);
+                    column(Fmt(r.I, 4), 14);
+                    column(Fmt(r.sigma, 4), 14);
+                    column(Fmt(r.I_plus, 4), 14);
+                    column(Fmt(r.sigma_plus, 4), 14);
+                    column(Fmt(r.I_minus, 4), 14);
+                    column(Fmt(r.sigma_minus, 4), 14);
+                    column(Fmt(r.F, 4), 14);
+                    column(Fmt(r.sigmaF, 4), 14);
+                    column(Fmt(r.F_plus, 4), 14);
+                    column(Fmt(r.sigmaF_plus, 4), 14);
+                    column(Fmt(r.F_minus, 4), 14);
+                    column(Fmt(r.sigmaF_minus, 4), 14);
+                    s.push_back(r.rfree_flag ? '1' : '0');
+                    s.append(" o\n");   // 'o' = observed
+                }
+            }
+        });
+        for (const std::string &s : block)
+            out.write(s.data(), static_cast<std::streamsize>(s.size()));
     }
 
     out << "#\n";
@@ -435,7 +462,8 @@ void WriteMtzReflections(const std::vector<MergedReflection> &reflections,
 
 void WriteShelxHklReflections(const std::vector<MergedReflection> &reflections,
                               const DiffractionExperiment &experiment,
-                              const std::string &filename) {
+                              const std::string &filename,
+                              size_t nthreads) {
     bool has_anom = true;
     const std::vector<MergedOutRow> rows = BuildMergedRows(reflections, experiment, has_anom);
 
@@ -459,20 +487,53 @@ void WriteShelxHklReflections(const std::vector<MergedReflection> &reflections,
     std::ofstream out(filename);
     if (!out)
         throw std::runtime_error("WriteShelxHklReflections: cannot open " + filename);
-    out << std::fixed << std::setprecision(2);
-    const auto emit = [&out, scale](int h, int k, int l, float I, float sigma) {
-        out << std::setw(4) << h << std::setw(4) << k << std::setw(4) << l
-            << std::setw(8) << scale * I << std::setw(8) << scale * sigma << "\n";
+    // Up to two records per reflection, five formatted numbers each. Built in parallel into per-worker
+    // blocks and handed to the file in order, exactly as the mmCIF rows are; "%.2f" right-aligned in
+    // the fixed field is what `fixed` + `setprecision(2)` + `setw` made the stream write.
+    const auto column = [](std::string &s, const std::string &v, size_t width) {
+        if (v.size() < width) s.append(width - v.size(), ' ');
+        s.append(v);
     };
-    for (const auto& r : rows) {
-        const bool plus = usable(r.Ip, r.sIp);
-        const bool minus = usable(r.Im, r.sIm);
-        if (plus)  emit(r.h, r.k, r.l, r.Ip, r.sIp);
-        if (minus) emit(-r.h, -r.k, -r.l, r.Im, r.sIm);
-        if (!plus && !minus && usable(r.Imean, r.sImean))
-            emit(r.h, r.k, r.l, r.Imean, r.sImean);
+    const auto num2 = [](double v) {
+        char b[64];
+        const int n = std::snprintf(b, sizeof b, "%.2f", v);
+        return std::string(b, static_cast<size_t>(std::clamp(n, 0, static_cast<int>(sizeof b) - 1)));
+    };
+    const auto emit = [&column, &num2, scale](std::string &s, int h, int k, int l, float I, float sigma) {
+        column(s, std::to_string(h), 4);
+        column(s, std::to_string(k), 4);
+        column(s, std::to_string(l), 4);
+        column(s, num2(scale * I), 8);
+        column(s, num2(scale * sigma), 8);
+        s.push_back('\n');
+    };
+    {
+        const size_t nrow = rows.size();
+        const size_t nw = std::max<size_t>(nthreads, 1);
+        const int nch = static_cast<int>(ThreadsForWork(nrow, nw, 4096));
+        std::vector<std::string> block(nch);
+        ParallelChunks(nch, nw, [&](int tlo, int thi) {
+            for (int t = tlo; t < thi; ++t) {
+                const size_t lo = nrow * t / nch, hi = nrow * (t + 1) / nch;
+                std::string &s = block[t];
+                s.reserve((hi - lo) * 2 * 29);
+                for (size_t i = lo; i < hi; ++i) {
+                    const auto &r = rows[i];
+                    const bool plus = usable(r.Ip, r.sIp);
+                    const bool minus = usable(r.Im, r.sIm);
+                    if (plus)  emit(s, r.h, r.k, r.l, r.Ip, r.sIp);
+                    if (minus) emit(s, -r.h, -r.k, -r.l, r.Im, r.sIm);
+                    if (!plus && !minus && usable(r.Imean, r.sImean))
+                        emit(s, r.h, r.k, r.l, r.Imean, r.sImean);
+                }
+            }
+        });
+        for (const std::string &s : block)
+            out.write(s.data(), static_cast<std::streamsize>(s.size()));
     }
-    emit(0, 0, 0, 0.0f, 0.0f);   // HKLF-4 end-of-data marker
+    std::string tail;
+    emit(tail, 0, 0, 0, 0.0f, 0.0f);   // HKLF-4 end-of-data marker
+    out.write(tail.data(), static_cast<std::streamsize>(tail.size()));
     out.close();
 }
 
@@ -482,11 +543,13 @@ void WriteReflections(const std::vector<MergedReflection> &reflections,
                       const MergeStatistics &statistics,
                       const ErrorModelReport &error_model,
                       const TwinningAnalysisResult &twinning,
-                      const std::string &filename) {
+                      const std::string &filename,
+                      size_t nthreads) {
     // Write an MTZ, an mmCIF and a SHELX HKLF-4 .hkl - each has its uses downstream (MTZ for the CCP4 /
     // phenix reflection tools, mmCIF for deposition and as the self-describing native format, HKLF-4 as
     // the SHELXC / ANODE substructure-solution input).
     WriteMtzReflections(reflections, unitCell, experiment, filename + ".mtz");
-    WriteMmcifReflections(reflections, unitCell, experiment, statistics, error_model, twinning, filename + ".cif");
-    WriteShelxHklReflections(reflections, experiment, filename + ".hkl");
+    WriteMmcifReflections(reflections, unitCell, experiment, statistics, error_model, twinning,
+                          filename + ".cif", nthreads);
+    WriteShelxHklReflections(reflections, experiment, filename + ".hkl", nthreads);
 }