diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index 9d8e23b80..586f316f0 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -311,6 +311,32 @@ from the one each image was indexed in — the file records the change of basis describe the same frame. A file whose stored cell and reflections disagree stops with a message naming both cells and the exact `-S`/`-C` override to merge it in its own setting. +### Several files + +``` +rugnux --mode scale -o combined run1_process.h5 run2_process.h5 run3_process.h5 +``` + +The `_process.h5` files of several stills runs of one crystal are merged as one run, as serial data +from several collections usually are. Each frame keeps the geometry, wavelength and detector of its own +file; the frames are taken file after file, in the order given, and numbered on from one file to the +next, so the frames of different files stay distinct. The union is then merged exactly as one file is: +the frames of the run's crystal form, each put on the run's cell, the indexing ambiguity resolved from +the data over all the crystals together - so crystals of different files indexed in different +alternative indexings, or on permuted axes of the same lattice, end up in one - and one scaling, +post-refinement and merge. The cell, space group, given cell (`/entry/MX/stillsGivenUnitCell`) and +rocking curve merged with are the first file's; the ice-ring gate pools the frames of all of them. + +The files must be of one crystal: each file's cell the first's lattice (in its setting or another) and +the same space group. A run stops with a message naming both cells, or both groups, where they are not - +unless the cell is given (`-C`, or a reference `-z`) and, for the group, `-S`: then everything is merged +in what is given, and the crystals of a file not on that cell are left out. A mix of stills and rotation +files is refused, and so are several rotation files (`--mode scale` merges one sweep at a time; several +sweeps of one crystal are merged together from their masters by `--mode mx`). Under `--force-still` a +rotation file counts as stills. `-s`/`-e` select images of one file and are refused with several. The +log and the report ([`INPUT_nn`](RUGNUX_REPORT.md#the-headline-keys)) give each file's images, indexed +crystals, merged crystals and those left out; the statistics are those of the merge of all of them. + ## The merged mmCIF (`.cif`) The `.cif` follows the PDBx/mmCIF dictionary and holds one data block, `data_sample`: the merged diff --git a/docs/RUGNUX_CLI.md b/docs/RUGNUX_CLI.md index 7da5065ff..f195be852 100644 --- a/docs/RUGNUX_CLI.md +++ b/docs/RUGNUX_CLI.md @@ -84,10 +84,14 @@ The binning is set by the [azimuthal-integration options](#azimuthal-integration ``` rugnux -o myrun --write-process-h5 dataset_master.h5 # integrate and merge once rugnux --mode scale -o remerged -A myrun_process.h5 # re-merge (on stills, -A keeps the Bijvoet hands apart) +rugnux --mode scale -o combined run1_process.h5 run2_process.h5 run3_process.h5 # stills of several runs, as one ``` Re-scales and merges the reflections already integrated into a `_process.h5`, without re-running spot finding or integration ([Re-scaling and re-merging](RUGNUX_ADVANCED.md#re-scaling-and-re-merging-rugnux---mode-scale)). +Several `_process.h5` files of stills are merged as one run, their crystals scaled together and the indexing +ambiguity resolved over all of them ([Several files](RUGNUX_ADVANCED.md#several-files)); rotation files are +taken one at a time. ## Inputs @@ -97,8 +101,8 @@ one dataset - `rugnux run*.h5` (SACLA), `*.cxi`, `*.xtc` (LCLS) - or several Jun masters of stills - or of rotation sweeps under `--force-still` - read as one run: their images one after another, merged together, with the indexing rate reported per master as well. Such masters must share the detector, wavelength, distance and beam centre. A grid scan an input holds (a fixed-target chip) is not carried into the run: several chips are -not one grid, so the run has no grid map. Several inputs must be one format; any `--mode` other than `mx`, and -following a broker, take one. +not one grid, so the run has no grid map. Several inputs must be one format; `--mode scale` takes several `_process.h5` files of +stills (above), and any other `--mode`, and following a broker, take one. Several rotation sweeps are told apart from the files: one sweep per CBF/marCCD/SMV/SFRM file-name template (the name with its frame number taken out - any frame of a sweep names the whole sweep, so @@ -296,7 +300,7 @@ serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rota | `--no-scaling-corrections` | rot3d: disable the default-on decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls | | `--relative-b[=deg]` | rot3d: fit a per-batch relative-*B* beyond the single decay slope over deg-degree batches, cross-validated (default 10° when bare; off otherwise) | | `--simple-stills` | Stills: treat every reflection as a full (*p* = 1, single-pass scale/merge) — disables the default-on physical partiality post-refinement | -| `--live-checkpoints [,...]` | `--mode scale`, stills (for developers): replay the stored frames in image order as the broker's live merge of a serial collection merges them - a merge at each fraction of the images (e.g. `0.5,0.75`), each crystal after the first checkpoint scaled, post-refined and placed in its indexing against the latest checkpoint's merge as it comes, and after the last frame one merge of all of them without a scaling or post-refinement over the whole run. The log gives each checkpoint's statistics and time, and the time of the merge after the last frame; the merge written is that last one. Without it `--mode scale` merges cold, as the broker does when no checkpoint is reached Each fraction 0.01-0.99. | +| `--live-checkpoints [,...]` | `--mode scale`, stills (for developers): replay the stored frames in image order as the broker's live merge of a serial collection merges them - a merge at each fraction of the images (e.g. `0.5,0.75`), each crystal after the first checkpoint scaled, post-refined and placed in its indexing against the latest checkpoint's merge as it comes, and after the last frame one merge of all of them without a scaling or post-refinement over the whole run. The log gives each checkpoint's statistics and time, and the time of the merge after the last frame; the merge written is that last one. With several `_process.h5` files the frames are replayed file after file, in the order given, as one collection. Without it `--mode scale` merges cold, as the broker does when no checkpoint is reached Each fraction 0.01-0.99. | | `--live-final-full` | With `--live-checkpoints`: the merge after the last frame scales and post-refines every crystal over the whole run, as a cold merge does, instead of merging the crystals as they were prepared against the checkpoints - what the broker's live merge would do with a full scaling pass at the end; its time is in the log beside the incremental one's. | | `--no-expected-variance-merge` | Stills: disable the default expected-variance merge weighting (which rebuilds each weak observation's signal variance at the reflection mean to de-bias the inverse-variance merge); restores observed-sigma weighting | | `--capture-uncertainty ` | Systematic sigma num·(1−captured fraction)·⟨I⟩: on rot3d fulls the share of the rocking curve not captured, on stills the share of each partial its partiality does not see (default: 1.0 for rotation, 0 otherwise) | diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index fd07ac934..64af80d53 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -238,6 +238,7 @@ The keys most scripts read, by section; the rest of this page explains the ones |---|---| | `EXPERIMENT_TYPE` | `ROTATION` or `STILLS` (section 1) | | `IMAGES_PROCESSED`, `WAVELENGTH` | Images the run read, and the wavelength (Å) | +| `INPUT_nn` | `--mode scale` of several `_process.h5` files: one row per file, in the order given - its images, the crystals indexed in it, those of them merged and those left out (off the run's cell form, a collapsed scale, `--min-image-cc`), and the file ([Several files](RUGNUX_ADVANCED.md#several-files)) | | `OSCILLATION_RANGE`, `STARTING_ANGLE`, `ROTATION_AXIS` | Rotation only: the per-image rotation the run integrated at (deg: the file's, scaled where the run adopted a goniometer rotation scale), the first image's angle and the axis | | `DETECTOR_DISTANCE`, `BEAM_CENTRE`, `DETECTOR_TILT` | The geometry the run integrated at (mm; the point of normal incidence in pixels; the three PONI rotations in degrees) - on a rotation run the post-refined one, not the header's | | `DIRECT_BEAM` | Where the direct beam meets the detector under that tilt (pixels) | diff --git a/image_analysis/scale_merge/Merge.h b/image_analysis/scale_merge/Merge.h index 96947669a..1625414c7 100644 --- a/image_analysis/scale_merge/Merge.h +++ b/image_analysis/scale_merge/Merge.h @@ -424,8 +424,6 @@ class MergeOnTheFly { double reject_up = 0.0, reject_down = 0.0; size_t reject_count = 0; - bool Mask(const IntegrationOutcome &outcome) const; - // One accepted observation, reduced to what the accumulator adds for it. struct Contribution { uint64_t key; @@ -442,6 +440,8 @@ class MergeOnTheFly { void Accumulate(const std::vector &contributions, int64_t image_id); public: MergeOnTheFly(const DiffractionExperiment &x); + // Whether the merge leaves this crystal out: off the reference cell, or below the per-image CC limit. + bool Mask(const IntegrationOutcome &outcome) const; MergeOnTheFly& ReferenceCell(const std::optional &cell); MergeOnTheFly& ExcludeIceRings(bool input) { exclude_ice_rings = input; return *this; } MergeOnTheFly& FilterByImageCC(bool input) { filter_by_image_cc = input; return *this; } diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index b331b29da..53308e245 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -910,6 +910,13 @@ ReportDocument BuildReportDocument(const std::string &output_prefix, s.title = "1. DATA SET AND GEOMETRY"; Add(s, KeyEnum("EXPERIMENT_TYPE", rotation ? "ROTATION" : "STILLS", {"ROTATION", "STILLS"})); Add(s, KeyInt("IMAGES_PROCESSED", static_cast(result.images_processed))); + // --mode scale of several _process.h5 files: images, indexed, merged and left-out crystals, file. + for (size_t i = 0; i < result.scale_inputs.size(); ++i) { + const auto &in = result.scale_inputs[i]; + Add(s, KeyText(fmt::format("INPUT_{:02}", i + 1).c_str(), + fmt::format("{} {} {} {} {}", in.images, in.indexed, in.merged, in.indexed - in.merged, + in.file))); + } Add(s, KeyReal("WAVELENGTH", experiment.GetWavelength_A(), "{:.5f}")); if (const auto gonio = experiment.GetGoniometer()) { // The oscillation the run integrated at: the file's, scaled where the run adopted a diff --git a/rugnux/Rugnux.h b/rugnux/Rugnux.h index 5a18f9f5c..411554c38 100644 --- a/rugnux/Rugnux.h +++ b/rugnux/Rugnux.h @@ -582,9 +582,17 @@ struct RugnuxOutputs { std::optional process_start; }; +// One of several _process.h5 files --mode scale merged as one: its frames, those of them indexed, and those of +// them that went into the merge (MergedCrystals). +struct ScaleInputCount { + std::string file; + int64_t images = 0, indexed = 0, merged = 0; +}; + struct ProcessResult { bool cancelled = false; uint64_t images_processed = 0; + std::vector scale_inputs; // --mode scale of several files, in the order given // The file image of processed-image ordinal f is first_image + f * image_stride (-s, --stride). int first_image = 0; int image_stride = 1; diff --git a/rugnux/StillsLiveMerge.cpp b/rugnux/StillsLiveMerge.cpp index d60be6483..8dd019516 100644 --- a/rugnux/StillsLiveMerge.cpp +++ b/rugnux/StillsLiveMerge.cpp @@ -306,3 +306,79 @@ ProcessResult StillsLiveResult(const DiffractionExperiment &experiment, const St result.twinning = out.twinning; return result; } + +namespace { + std::string CellText(const UnitCell &c) { + return fmt::format("a={:.3f} b={:.3f} c={:.3f} alpha={:.2f} beta={:.2f} gamma={:.2f}", c.a, c.b, c.c, c.alpha, + c.beta, c.gamma); + } + + // The same lattice in this setting or another: a change of basis of determinant one carries `cell` onto + // `reference` (an axis multiple, a basis of larger determinant, is a different lattice). + bool SameLattice(const UnitCell &cell, const UnitCell &reference, const IndexingSettings &settings) { + for (const auto &m : FindSublatticeBases(CrystalLattice(cell), reference, settings)) { + const int det = m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1]) + - m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0]) + + m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0]); + if (std::abs(det) == 1) + return true; + } + return false; + } +} + +std::string ScaleInputsRefusal(const std::vector &inputs, bool cell_given, bool space_group_given, + const IndexingSettings &settings) { + size_t rotation = 0; + for (const auto &in : inputs) + rotation += in.rotation; + if (rotation > 0 && rotation < inputs.size()) { + const auto rot = std::find_if(inputs.begin(), inputs.end(), [](const ScaleInput &i) { return i.rotation; }); + const auto still = std::find_if(inputs.begin(), inputs.end(), [](const ScaleInput &i) { return !i.rotation; }); + return fmt::format("{} is a rotation sweep and {} is stills: --mode scale merges several _process.h5 files " + "of stills only", rot->file, still->file); + } + if (rotation > 0) + return fmt::format("{} _process.h5 files of rotation sweeps: --mode scale merges several files of stills " + "only. To merge sweeps of one crystal together, give their masters to --mode mx", + rotation); + const ScaleInput &first = inputs.front(); + for (const auto &in : inputs) { + if (!cell_given) { + if (!in.cell) + return fmt::format("{} states no unit cell: give the cell to merge in with -C", in.file); + if (!SameLattice(*in.cell, *first.cell, settings)) + return fmt::format("{} was integrated in {} and {} in {}, not one lattice, so their crystals are " + "not merged together. If they are one crystal, give the cell to merge in with -C", + first.file, CellText(*first.cell), in.file, CellText(*in.cell)); + } + if (!space_group_given && in.space_group_number != first.space_group_number) + return fmt::format("{} is in space group {} and {} in {}: give the group to merge in with -S", first.file, + first.space_group_number ? std::to_string(*first.space_group_number) : "none", + in.file, in.space_group_number ? std::to_string(*in.space_group_number) : "none"); + } + return {}; +} + +void AppendScaleInput(std::vector &outcomes, std::vector &&more) { + const auto offset = static_cast(outcomes.size()); + for (auto &o : more) { + for (auto &r : o.reflections) + r.image_number += offset; + outcomes.push_back(std::move(o)); + } +} + +size_t MergedCrystals(const DiffractionExperiment &experiment, const std::vector &outcomes, + size_t first, size_t last) { + MergeOnTheFly merge(experiment); + merge.ReferenceCell(experiment.GetUnitCell()); + merge.FilterByImageCC(experiment.GetScalingSettings().GetMinCCForImage() > 0.0); + size_t n = 0; + for (size_t i = first; i < last; ++i) + n += !merge.Mask(outcomes[i]) + && std::any_of(outcomes[i].reflections.begin(), outcomes[i].reflections.end(), [](const Reflection &r) { + return r.image_scale_corr > 0.0f && std::isfinite(r.image_scale_corr); + }); + return n; +} diff --git a/rugnux/StillsLiveMerge.h b/rugnux/StillsLiveMerge.h index df62e0ee5..dcb569aea 100644 --- a/rugnux/StillsLiveMerge.h +++ b/rugnux/StillsLiveMerge.h @@ -4,6 +4,7 @@ #pragma once #include +#include #include #include "Rugnux.h" @@ -125,3 +126,27 @@ StillsMergeOutput ReplayLiveStillsMerge(const DiffractionExperiment &experiment, // it, never a verdict. ProcessResult StillsLiveResult(const DiffractionExperiment &experiment, const StillsMergeOutput &out, int64_t images_analysed, int64_t images_indexed); + +// One _process.h5 of several that --mode scale merges together, as the file states it. +struct ScaleInput { + std::string file; + bool rotation = false; // a rotation sweep (not under --force-still) + std::optional cell; + std::optional space_group_number; +}; + +// Why --mode scale cannot merge these files together; empty where it can. Several files are merged as one +// stills run: all of them stills, of one lattice - each file's cell the first's, in that setting or another +// (an axis permutation: the merge puts every crystal on the first file's axes, RelabelOnSublattice) - unless +// the cell is given (cell_given, -C), and of one space group unless it is given (space_group_given, -S). +std::string ScaleInputsRefusal(const std::vector &inputs, bool cell_given, bool space_group_given, + const IndexingSettings &settings); + +// The frames of one more file put after those of the files before it: their image numbers moved on by the +// frames already there, so that frames of different files stay distinct. +void AppendScaleInput(std::vector &outcomes, std::vector &&more); + +// Of the frames [first, last) as MergeStills left them, the crystals that went into the merge: on the cell +// merged in, scaled, and not left out by --min-image-cc. +size_t MergedCrystals(const DiffractionExperiment &experiment, const std::vector &outcomes, + size_t first, size_t last); diff --git a/rugnux/rugnux_cli.cpp b/rugnux/rugnux_cli.cpp index 5d1cb6a2a..da59aa7c7 100644 --- a/rugnux/rugnux_cli.cpp +++ b/rugnux/rugnux_cli.cpp @@ -176,6 +176,7 @@ Azimuthal integration (--mode azint) Re-merging (--mode scale) rugnux --mode scale -o remerged -A myrun_process.h5 + rugnux --mode scale -o combined run1_process.h5 run2_process.h5 stills of several runs, as one Re-scales and merges the reflections of a _process.h5 written with --write-process-h5. Basic options @@ -1800,10 +1801,11 @@ int RugnuxCli::Parse(int argc, char **argv) { exit(EXIT_FAILURE); } if (input_files.size() > 1) { - // Several files are one stills dataset (JFJochMultiReader.h for NXmx masters); every way of - // reading that is not a set of stored files is refused here, before anything is opened. - if (mode != RugnuxMode::MX) { - logger.Error("Several inputs are read together in --mode mx only"); + // Several files are one stills dataset (JFJochMultiReader.h for NXmx masters), or in --mode scale the + // _process.h5 files of several stills runs, merged as one; every way of reading that is not a set of + // stored files is refused here, before anything is opened. + if (mode != RugnuxMode::MX && mode != RugnuxMode::Scale) { + logger.Error("Several inputs are read together in --mode mx and --mode scale only"); exit(EXIT_FAILURE); } if (!broker_url.empty() || simulate_live) { @@ -1978,7 +1980,8 @@ int RugnuxInputRun::PrepareRugnuxInput() { return JFJochXFELReader::CanReadPanelTable(f) || JFJochXFELReader::CanReadCXI(f) || JFJochXTCReader::CanRead(f); }; input_is_crystfel = !crystfel_geometry.empty(); - if (input_files.size() > 1 && !input_is_crystfel && !is_foreign(input_file)) { + // --mode scale reads the first of several _process.h5 files here, and the others with its reflections. + if (input_files.size() > 1 && mode != RugnuxMode::Scale && !input_is_crystfel && !is_foreign(input_file)) { // One frame of a CBF, marCCD or SMV sweep names the whole sweep. if (JFJochCBFReader::CanRead(input_file) || JFJochMarCCDReader::CanRead(input_file) || JFJochSMVReader::CanRead(input_file) || JFJochSFRMReader::CanRead(input_file)) { @@ -2372,7 +2375,69 @@ int RugnuxInputRun::PrepareRugnuxInput() { } const auto total_images = static_cast(reader.GetNumberOfImages()); const int last_image = (end_image < 0 || end_image >= total_images) ? total_images - 1 : end_image; + // Several files: the stills of several runs merged as one, in the order given, each frame with the + // geometry of its own file. The cell, the group, the given cell and the rocking curve merged in are + // the first file's, unless -C / -S give them; the ice gate pools every file's frames. + std::vector ice_ring_score = dataset->ice_ring_score; + std::vector spot_count_ice_rings = dataset->spot_count_ice_rings; + std::vector spot_count_ice_control = dataset->spot_count_ice_control; + std::vector> more; // the files after the first + if (input_files.size() > 1) { + if (start_image != 0 || end_image >= 0) { + logger.Error("-s / -e select images of one file; with several _process.h5 files each is merged whole"); + throw RugnuxInputFailed{EXIT_FAILURE}; + } + const auto describe = [this](const std::string &f, const JFJochReaderDataset &d) { + const auto sg = d.experiment.GetGemmiSpaceGroup(); + return ScaleInput{f, d.experiment.GetGoniometer().has_value() && !force_still, + d.experiment.GetUnitCell(), + sg ? std::optional(sg->number) : std::nullopt}; + }; + std::vector inputs = {describe(input_file, *dataset)}; + for (size_t k = 1; k < input_files.size(); ++k) { + auto r = std::make_unique(); + try { + r->ReadFile(input_files[k]); + } catch (const std::exception &e) { + logger.Error("{} cannot be read: {}", input_files[k], e.what()); + throw RugnuxInputFailed{EXIT_FAILURE}; + } + if (!r->GetDataset()) { + logger.Error("No experiment dataset found in {}", input_files[k]); + throw RugnuxInputFailed{EXIT_FAILURE}; + } + inputs.push_back(describe(input_files[k], *r->GetDataset())); + more.push_back(std::move(r)); + } + if (const auto refusal = ScaleInputsRefusal(inputs, fixed_reference_unit_cell.has_value(), + space_group != nullptr, IndexingSettings()); + !refusal.empty()) { + logger.Error("{}", refusal); + throw RugnuxInputFailed{EXIT_FAILURE}; + } + } auto reflections = hdf5_reader.ReadReflections(start_image, last_image); + std::vector input_first_frame = {0}; // of each file, in `reflections` + for (const auto &r : more) { + input_first_frame.push_back(reflections.size()); + AppendScaleInput(reflections, r->ReadReflections()); + const auto &d = *r->GetDataset(); + ice_ring_score.insert(ice_ring_score.end(), d.ice_ring_score.begin(), d.ice_ring_score.end()); + spot_count_ice_rings.insert(spot_count_ice_rings.end(), d.spot_count_ice_rings.begin(), + d.spot_count_ice_rings.end()); + spot_count_ice_control.insert(spot_count_ice_control.end(), d.spot_count_ice_control.begin(), + d.spot_count_ice_control.end()); + } + input_first_frame.push_back(reflections.size()); + std::vector input_counts; + if (input_files.size() > 1) + for (size_t k = 0; k < input_files.size(); ++k) { + ScaleInputCount c{input_files[k], static_cast(input_first_frame[k + 1] - input_first_frame[k])}; + for (size_t i = input_first_frame[k]; i < input_first_frame[k + 1]; ++i) + c.indexed += !reflections[i].reflections.empty(); + logger.Info("Input {}: {} images, {} indexed, from {}", k + 1, c.images, c.indexed, c.file); + input_counts.push_back(c); + } DiffractionExperiment experiment(dataset->experiment); configure_offline_output(experiment, output_prefix); @@ -2547,7 +2612,7 @@ int RugnuxInputRun::PrepareRugnuxInput() { // --scale reaches the same verdict on the same data as the pipeline that produced it. double ice_sum = 0.0; size_t ice_n = 0; - for (const float s : dataset->ice_ring_score) + for (const float s : ice_ring_score) if (std::isfinite(s)) { ice_sum += s; ++ice_n; @@ -2555,9 +2620,9 @@ int RugnuxInputRun::PrepareRugnuxInput() { const float ice_min_score = experiment.GetScalingSettings().GetIceMinScore(); // ...and the spot channel, pooled over the run exactly as the full pipeline pools it. double ring_sum = 0.0, ctrl_sum = 0.0; - for (const float v : dataset->spot_count_ice_rings) + for (const float v : spot_count_ice_rings) if (std::isfinite(v)) ring_sum += v; - for (const float v : dataset->spot_count_ice_control) + for (const float v : spot_count_ice_control) if (std::isfinite(v)) ctrl_sum += v; // Empty control + spots on the rings = the strongest ice evidence, not its absence. const double ice_spot_ratio = ctrl_sum > 0.0 ? ring_sum / ctrl_sum : (ring_sum > 0.0 ? 1.0e3 : 0.0); @@ -2657,6 +2722,13 @@ int RugnuxInputRun::PrepareRugnuxInput() { dataset->stills_run_width, flag_ice, ice_width, live_checkpoints, live_final_full, nthreads, logger); experiment.ImagesPerTrigger(static_cast(so.images)); + for (size_t k = 0; k < input_counts.size(); ++k) { + auto &c = input_counts[k]; + c.merged = static_cast(MergedCrystals(experiment, reflections, input_first_frame[k], + input_first_frame[k + 1])); + logger.Info("Input {}: {} of {} indexed crystals merged, {} left out, from {}", k + 1, c.merged, + c.indexed, c.indexed - c.merged, c.file); + } auto &sm = so.merge; merged_reflections = std::move(sm.merged); merged_statistics = std::move(sm.statistics); @@ -2869,7 +2941,8 @@ int RugnuxInputRun::PrepareRugnuxInput() { // --mode scale re-merges stored reflections, so it determines a merging result and gets the // same report. It has no ProcessResult of its own; fill the parts that exist here. ProcessResult scale_result; - scale_result.images_processed = reader.GetNumberOfImages(); + scale_result.images_processed = input_files.size() > 1 ? reflections.size() : reader.GetNumberOfImages(); + scale_result.scale_inputs = input_counts; scale_result.consensus_cell = experiment.GetUnitCell(); scale_result.space_group = experiment.GetGemmiSpaceGroup(); scale_result.used_beam_x_pxl = experiment.GetBeamX_pxl(); @@ -3984,6 +4057,14 @@ int RugnuxCli::Run() { || JFJochXTCReader::CanRead(f); return d; }; + // --mode scale: several _process.h5 files are one stills merge, checked where they are read. + if (mode == RugnuxMode::Scale) { + try { + return RugnuxInputRun(*this).Process(); + } catch (const RugnuxInputFailed &f) { + return f.code; + } + } // --force-still: every input is stills, so several masters are one stills run, rotation masters // included. Frame files still go through the plan: their frames name one sweep. if (force_still && std::none_of(input_files.begin(), input_files.end(), [&](const std::string &f) { diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 0bc1bafd5..eb86705e1 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -128,6 +128,7 @@ ADD_EXECUTABLE(jfjoch_test LiveCollectionTest.cpp RugnuxLiveStatusTest.cpp RugnuxLiveTriageTest.cpp + RugnuxScaleInputsTest.cpp StillsGPUAnalysisTest.cpp StillsGPUAnalysisFPGATest.cpp RugnuxStructureTest.cpp diff --git a/tests/RugnuxScaleInputsTest.cpp b/tests/RugnuxScaleInputsTest.cpp new file mode 100644 index 000000000..37ac4e893 --- /dev/null +++ b/tests/RugnuxScaleInputsTest.cpp @@ -0,0 +1,92 @@ +// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute +// SPDX-License-Identifier: GPL-3.0-only + +#include + +#include "../common/DiffractionExperiment.h" +#include "../rugnux/ResultReport.h" +#include "../rugnux/StillsLiveMerge.h" + +namespace { + const UnitCell orthorhombic_uc{.a = 40.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + + ScaleInput Still(const std::string &file, const UnitCell &cell, int space_group = 19) { + return ScaleInput{file, false, cell, space_group}; + } +} + +TEST_CASE("ScaleInputsRefusal_OneCrystal", "[RugnuxScale]") { + const IndexingSettings settings; + // The same cell, and the same lattice on permuted axes (the merge relabels those onto the first's). + const UnitCell permuted{.a = 60.0f, .b = 90.0f, .c = 40.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + const UnitCell refined{.a = 40.05f, .b = 59.95f, .c = 90.1f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", refined), Still("c", permuted)}, false, false, + settings).empty()); +} + +TEST_CASE("ScaleInputsRefusal_CellsDisagree", "[RugnuxScale]") { + const IndexingSettings settings; + const UnitCell other{.a = 45.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + // An axis doubled: a lattice that contains the first, but is not it. + const UnitCell doubled{.a = 80.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f}; + const auto refusal = ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", other)}, false, false, settings); + CHECK(refusal.find("not one lattice") != std::string::npos); + CHECK(refusal.find("a=40.000 b=60.000 c=90.000") != std::string::npos); + CHECK(refusal.find("a=45.000 b=60.000 c=90.000") != std::string::npos); + CHECK(refusal.find("-C") != std::string::npos); + CHECK(!ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", doubled)}, false, false, settings).empty()); + // A cell given on the command line is what they are merged in. + CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", other)}, true, false, settings).empty()); + // A file without a cell, unless one is given. + ScaleInput no_cell{"b", false, std::nullopt, 19}; + CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), no_cell}, false, false, settings).find("no unit cell") + != std::string::npos); + CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), no_cell}, true, false, settings).empty()); +} + +TEST_CASE("ScaleInputsRefusal_SpaceGroupsDisagree", "[RugnuxScale]") { + const IndexingSettings settings; + const auto refusal = ScaleInputsRefusal({Still("a", orthorhombic_uc, 19), Still("b", orthorhombic_uc, 16)}, + false, false, settings); + CHECK(refusal.find("space group 19") != std::string::npos); + CHECK(refusal.find("-S") != std::string::npos); + CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc, 19), Still("b", orthorhombic_uc, 16)}, false, true, + settings).empty()); +} + +TEST_CASE("ScaleInputsRefusal_Rotation", "[RugnuxScale]") { + const IndexingSettings settings; + const ScaleInput sweep{"r", true, orthorhombic_uc, 19}; + // Stills and a rotation sweep: refused, whatever is given. + const auto mixed = ScaleInputsRefusal({Still("a", orthorhombic_uc), sweep}, true, true, settings); + CHECK(mixed.find("r is a rotation sweep and a is stills") != std::string::npos); + // Several sweeps: refused, pointing to --mode mx. + const auto sweeps = ScaleInputsRefusal({sweep, sweep}, false, false, settings); + CHECK(sweeps.find("--mode mx") != std::string::npos); +} + +TEST_CASE("AppendScaleInput_FramesStayDistinct", "[RugnuxScale]") { + std::vector all(3); + all[1].reflections.push_back(Reflection{.image_number = 1.0f}); + std::vector more(2); + more[0].reflections.push_back(Reflection{.image_number = 0.0f}); + more[1].reflections.push_back(Reflection{.image_number = 1.0f}); + AppendScaleInput(all, std::move(more)); + REQUIRE(all.size() == 5); + CHECK(all[1].reflections[0].image_number == 1.0f); + CHECK(all[3].reflections[0].image_number == 3.0f); + CHECK(all[4].reflections[0].image_number == 4.0f); +} + +TEST_CASE("ScaleInputs_Report", "[RugnuxScale]") { + DiffractionExperiment x(DetJF(1)); + ProcessResult result; + result.images_processed = 300; + result.scale_inputs = {{"run1_process.h5", 100, 90, 85}, {"run2_process.h5", 200, 150, 140}}; + const std::string text = RenderResultReport("p", "run1_process.h5", x, result); + CHECK(text.find("\nINPUT_01= 100 90 85 5 run1_process.h5\n") != std::string::npos); + CHECK(text.find("\nINPUT_02= 200 150 140 10 run2_process.h5\n") != std::string::npos); + // One file: no rows. + result.scale_inputs.clear(); + CHECK(RenderResultReport("p", "in.h5", x, result).find("INPUT_01") == std::string::npos); +}