rugnux --mode scale: merge several stills _process.h5 files as one run

`rugnux --mode scale -o combined run1_process.h5 run2_process.h5 ...` reads
every file's reflections and per-image metadata, keeps each frame's own
geometry, numbers the frames on from file to file, and runs the existing
stills merge (MergeStills) on the union - so the cell-form selection, the
relabel onto the run's cell (RelabelOnSublattice, which also takes crystals
of a file indexed on permuted axes) and the data-only indexing-ambiguity
resolution act over all crystals of all files at once. The cell, group,
given cell and rocking curve are the first file's; the ice gate pools all.

Checks (ScaleInputsRefusal): every file stills; each file's cell the
first's lattice in some setting (a change of basis of determinant one),
unless -C/-z gives the cell; one space-group number unless -S. A mix of
stills and rotation files is refused, and so are several rotation files
(pointing to --mode mx, which merges sweeps from their masters); the
multi-sweep machinery is not touched. -s/-e are refused with several
files; --live-checkpoints replays the files one after another.

Per file the log and the report (INPUT_nn= images indexed merged
left-out file) give the counts; MergeOnTheFly::Mask is made public so
"merged" is the merge's own definition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-14 20:41:45 +01:00
1 parent d8a6ce3a74
commit 74acfc433e
11 files changed
+335 -14

No files matched your search

+26
View File
@@ -311,6 +311,32 @@ from the one each image was indexed in — the file records the change of basis
describe the same frame. A file whose stored cell and reflections disagree stops with a message
naming both cells and the exact `-S`/`-C` override to merge it in its own setting.
### Several files
```
rugnux --mode scale -o combined run1_process.h5 run2_process.h5 run3_process.h5
```
The `_process.h5` files of several stills runs of one crystal are merged as one run, as serial data
from several collections usually are. Each frame keeps the geometry, wavelength and detector of its own
file; the frames are taken file after file, in the order given, and numbered on from one file to the
next, so the frames of different files stay distinct. The union is then merged exactly as one file is:
the frames of the run's crystal form, each put on the run's cell, the indexing ambiguity resolved from
the data over all the crystals together - so crystals of different files indexed in different
alternative indexings, or on permuted axes of the same lattice, end up in one - and one scaling,
post-refinement and merge. The cell, space group, given cell (`/entry/MX/stillsGivenUnitCell`) and
rocking curve merged with are the first file's; the ice-ring gate pools the frames of all of them.
The files must be of one crystal: each file's cell the first's lattice (in its setting or another) and
the same space group. A run stops with a message naming both cells, or both groups, where they are not -
unless the cell is given (`-C`, or a reference `-z`) and, for the group, `-S`: then everything is merged
in what is given, and the crystals of a file not on that cell are left out. A mix of stills and rotation
files is refused, and so are several rotation files (`--mode scale` merges one sweep at a time; several
sweeps of one crystal are merged together from their masters by `--mode mx`). Under `--force-still` a
rotation file counts as stills. `-s`/`-e` select images of one file and are refused with several. The
log and the report ([`INPUT_nn`](RUGNUX_REPORT.md#the-headline-keys)) give each file's images, indexed
crystals, merged crystals and those left out; the statistics are those of the merge of all of them.
## The merged mmCIF (`<prefix>.cif`)
The `.cif` follows the PDBx/mmCIF dictionary and holds one data block, `data_sample`: the merged
+7 -3
View File
@@ -84,10 +84,14 @@ The binning is set by the [azimuthal-integration options](#azimuthal-integration
```
rugnux -o myrun --write-process-h5 dataset_master.h5 # integrate and merge once
rugnux --mode scale -o remerged -A myrun_process.h5 # re-merge (on stills, -A keeps the Bijvoet hands apart)
rugnux --mode scale -o combined run1_process.h5 run2_process.h5 run3_process.h5 # stills of several runs, as one
```
Re-scales and merges the reflections already integrated into a `_process.h5`, without re-running
spot finding or integration ([Re-scaling and re-merging](RUGNUX_ADVANCED.md#re-scaling-and-re-merging-rugnux---mode-scale)).
Several `_process.h5` files of stills are merged as one run, their crystals scaled together and the indexing
ambiguity resolved over all of them ([Several files](RUGNUX_ADVANCED.md#several-files)); rotation files are
taken one at a time.
## Inputs
@@ -97,8 +101,8 @@ one dataset - `rugnux run*.h5` (SACLA), `*.cxi`, `*.xtc` (LCLS) - or several Jun
masters of stills - or of rotation sweeps under `--force-still` - read as one run: their images one after
another, merged together, with the indexing rate reported per master as well. Such masters must share the detector, wavelength,
distance and beam centre. A grid scan an input holds (a fixed-target chip) is not carried into the run: several chips are
not one grid, so the run has no grid map. Several inputs must be one format; any `--mode` other than `mx`, and
following a broker, take one.
not one grid, so the run has no grid map. Several inputs must be one format; `--mode scale` takes several `_process.h5` files of
stills (above), and any other `--mode`, and following a broker, take one.
Several rotation sweeps are told apart from the files: one sweep per CBF/marCCD/SMV/SFRM file-name
template (the name with its frame number taken out - any frame of a sweep names the whole sweep, so
@@ -296,7 +300,7 @@ serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rota
| `--no-scaling-corrections` | rot3d: disable the default-on decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls |
| `--relative-b[=deg]` | rot3d: fit a per-batch relative-*B* beyond the single decay slope over deg-degree batches, cross-validated (default 10° when bare; off otherwise) |
| `--simple-stills` | Stills: treat every reflection as a full (*p* = 1, single-pass scale/merge) — disables the default-on physical partiality post-refinement |
| `--live-checkpoints <f1>[,<f2>...]` | `--mode scale`, stills (for developers): replay the stored frames in image order as the broker's live merge of a serial collection merges them - a merge at each fraction of the images (e.g. `0.5,0.75`), each crystal after the first checkpoint scaled, post-refined and placed in its indexing against the latest checkpoint's merge as it comes, and after the last frame one merge of all of them without a scaling or post-refinement over the whole run. The log gives each checkpoint's statistics and time, and the time of the merge after the last frame; the merge written is that last one. Without it `--mode scale` merges cold, as the broker does when no checkpoint is reached Each fraction 0.01-0.99. |
| `--live-checkpoints <f1>[,<f2>...]` | `--mode scale`, stills (for developers): replay the stored frames in image order as the broker's live merge of a serial collection merges them - a merge at each fraction of the images (e.g. `0.5,0.75`), each crystal after the first checkpoint scaled, post-refined and placed in its indexing against the latest checkpoint's merge as it comes, and after the last frame one merge of all of them without a scaling or post-refinement over the whole run. The log gives each checkpoint's statistics and time, and the time of the merge after the last frame; the merge written is that last one. With several `_process.h5` files the frames are replayed file after file, in the order given, as one collection. Without it `--mode scale` merges cold, as the broker does when no checkpoint is reached Each fraction 0.01-0.99. |
| `--live-final-full` | With `--live-checkpoints`: the merge after the last frame scales and post-refines every crystal over the whole run, as a cold merge does, instead of merging the crystals as they were prepared against the checkpoints - what the broker's live merge would do with a full scaling pass at the end; its time is in the log beside the incremental one's. |
| `--no-expected-variance-merge` | Stills: disable the default expected-variance merge weighting (which rebuilds each weak observation's signal variance at the reflection mean to de-bias the inverse-variance merge); restores observed-sigma weighting |
| `--capture-uncertainty <num>` | Systematic sigma num·(1−captured fraction)·⟨I⟩: on rot3d fulls the share of the rocking curve not captured, on stills the share of each partial its partiality does not see (default: 1.0 for rotation, 0 otherwise) |
+1
View File
@@ -238,6 +238,7 @@ The keys most scripts read, by section; the rest of this page explains the ones
|---|---|
| `EXPERIMENT_TYPE` | `ROTATION` or `STILLS` (section 1) |
| `IMAGES_PROCESSED`, `WAVELENGTH` | Images the run read, and the wavelength (Å) |
| `INPUT_nn` | `--mode scale` of several `_process.h5` files: one row per file, in the order given - its images, the crystals indexed in it, those of them merged and those left out (off the run's cell form, a collapsed scale, `--min-image-cc`), and the file ([Several files](RUGNUX_ADVANCED.md#several-files)) |
| `OSCILLATION_RANGE`, `STARTING_ANGLE`, `ROTATION_AXIS` | Rotation only: the per-image rotation the run integrated at (deg: the file's, scaled where the run adopted a goniometer rotation scale), the first image's angle and the axis |
| `DETECTOR_DISTANCE`, `BEAM_CENTRE`, `DETECTOR_TILT` | The geometry the run integrated at (mm; the point of normal incidence in pixels; the three PONI rotations in degrees) - on a rotation run the post-refined one, not the header's |
| `DIRECT_BEAM` | Where the direct beam meets the detector under that tilt (pixels) |
+2 -2
View File
@@ -424,8 +424,6 @@ class MergeOnTheFly {
double reject_up = 0.0, reject_down = 0.0;
size_t reject_count = 0;
bool Mask(const IntegrationOutcome &outcome) const;
// One accepted observation, reduced to what the accumulator adds for it.
struct Contribution {
uint64_t key;
@@ -442,6 +440,8 @@ class MergeOnTheFly {
void Accumulate(const std::vector<Contribution> &contributions, int64_t image_id);
public:
MergeOnTheFly(const DiffractionExperiment &x);
// Whether the merge leaves this crystal out: off the reference cell, or below the per-image CC limit.
bool Mask(const IntegrationOutcome &outcome) const;
MergeOnTheFly& ReferenceCell(const std::optional<UnitCell> &cell);
MergeOnTheFly& ExcludeIceRings(bool input) { exclude_ice_rings = input; return *this; }
MergeOnTheFly& FilterByImageCC(bool input) { filter_by_image_cc = input; return *this; }
+7
View File
@@ -910,6 +910,13 @@ ReportDocument BuildReportDocument(const std::string &output_prefix,
s.title = "1. DATA SET AND GEOMETRY";
Add(s, KeyEnum("EXPERIMENT_TYPE", rotation ? "ROTATION" : "STILLS", {"ROTATION", "STILLS"}));
Add(s, KeyInt("IMAGES_PROCESSED", static_cast<int64_t>(result.images_processed)));
// --mode scale of several _process.h5 files: images, indexed, merged and left-out crystals, file.
for (size_t i = 0; i < result.scale_inputs.size(); ++i) {
const auto &in = result.scale_inputs[i];
Add(s, KeyText(fmt::format("INPUT_{:02}", i + 1).c_str(),
fmt::format("{} {} {} {} {}", in.images, in.indexed, in.merged, in.indexed - in.merged,
in.file)));
}
Add(s, KeyReal("WAVELENGTH", experiment.GetWavelength_A(), "{:.5f}"));
if (const auto gonio = experiment.GetGoniometer()) {
// The oscillation the run integrated at: the file's, scaled where the run adopted a
+8
View File
@@ -582,9 +582,17 @@ struct RugnuxOutputs {
std::optional<StartMessage> process_start;
};
// One of several _process.h5 files --mode scale merged as one: its frames, those of them indexed, and those of
// them that went into the merge (MergedCrystals).
struct ScaleInputCount {
std::string file;
int64_t images = 0, indexed = 0, merged = 0;
};
struct ProcessResult {
bool cancelled = false;
uint64_t images_processed = 0;
std::vector<ScaleInputCount> scale_inputs; // --mode scale of several files, in the order given
// The file image of processed-image ordinal f is first_image + f * image_stride (-s, --stride).
int first_image = 0;
int image_stride = 1;
+76
View File
@@ -306,3 +306,79 @@ ProcessResult StillsLiveResult(const DiffractionExperiment &experiment, const St
result.twinning = out.twinning;
return result;
}
namespace {
std::string CellText(const UnitCell &c) {
return fmt::format("a={:.3f} b={:.3f} c={:.3f} alpha={:.2f} beta={:.2f} gamma={:.2f}", c.a, c.b, c.c, c.alpha,
c.beta, c.gamma);
}
// The same lattice in this setting or another: a change of basis of determinant one carries `cell` onto
// `reference` (an axis multiple, a basis of larger determinant, is a different lattice).
bool SameLattice(const UnitCell &cell, const UnitCell &reference, const IndexingSettings &settings) {
for (const auto &m : FindSublatticeBases(CrystalLattice(cell), reference, settings)) {
const int det = m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1])
- m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0])
+ m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0]);
if (std::abs(det) == 1)
return true;
}
return false;
}
}
std::string ScaleInputsRefusal(const std::vector<ScaleInput> &inputs, bool cell_given, bool space_group_given,
const IndexingSettings &settings) {
size_t rotation = 0;
for (const auto &in : inputs)
rotation += in.rotation;
if (rotation > 0 && rotation < inputs.size()) {
const auto rot = std::find_if(inputs.begin(), inputs.end(), [](const ScaleInput &i) { return i.rotation; });
const auto still = std::find_if(inputs.begin(), inputs.end(), [](const ScaleInput &i) { return !i.rotation; });
return fmt::format("{} is a rotation sweep and {} is stills: --mode scale merges several _process.h5 files "
"of stills only", rot->file, still->file);
}
if (rotation > 0)
return fmt::format("{} _process.h5 files of rotation sweeps: --mode scale merges several files of stills "
"only. To merge sweeps of one crystal together, give their masters to --mode mx",
rotation);
const ScaleInput &first = inputs.front();
for (const auto &in : inputs) {
if (!cell_given) {
if (!in.cell)
return fmt::format("{} states no unit cell: give the cell to merge in with -C", in.file);
if (!SameLattice(*in.cell, *first.cell, settings))
return fmt::format("{} was integrated in {} and {} in {}, not one lattice, so their crystals are "
"not merged together. If they are one crystal, give the cell to merge in with -C",
first.file, CellText(*first.cell), in.file, CellText(*in.cell));
}
if (!space_group_given && in.space_group_number != first.space_group_number)
return fmt::format("{} is in space group {} and {} in {}: give the group to merge in with -S", first.file,
first.space_group_number ? std::to_string(*first.space_group_number) : "none",
in.file, in.space_group_number ? std::to_string(*in.space_group_number) : "none");
}
return {};
}
void AppendScaleInput(std::vector<IntegrationOutcome> &outcomes, std::vector<IntegrationOutcome> &&more) {
const auto offset = static_cast<float>(outcomes.size());
for (auto &o : more) {
for (auto &r : o.reflections)
r.image_number += offset;
outcomes.push_back(std::move(o));
}
}
size_t MergedCrystals(const DiffractionExperiment &experiment, const std::vector<IntegrationOutcome> &outcomes,
size_t first, size_t last) {
MergeOnTheFly merge(experiment);
merge.ReferenceCell(experiment.GetUnitCell());
merge.FilterByImageCC(experiment.GetScalingSettings().GetMinCCForImage() > 0.0);
size_t n = 0;
for (size_t i = first; i < last; ++i)
n += !merge.Mask(outcomes[i])
&& std::any_of(outcomes[i].reflections.begin(), outcomes[i].reflections.end(), [](const Reflection &r) {
return r.image_scale_corr > 0.0f && std::isfinite(r.image_scale_corr);
});
return n;
}
+25
View File
@@ -4,6 +4,7 @@
#pragma once
#include <optional>
#include <string>
#include <vector>
#include "Rugnux.h"
@@ -125,3 +126,27 @@ StillsMergeOutput ReplayLiveStillsMerge(const DiffractionExperiment &experiment,
// it, never a verdict.
ProcessResult StillsLiveResult(const DiffractionExperiment &experiment, const StillsMergeOutput &out,
int64_t images_analysed, int64_t images_indexed);
// One _process.h5 of several that --mode scale merges together, as the file states it.
struct ScaleInput {
std::string file;
bool rotation = false; // a rotation sweep (not under --force-still)
std::optional<UnitCell> cell;
std::optional<int> space_group_number;
};
// Why --mode scale cannot merge these files together; empty where it can. Several files are merged as one
// stills run: all of them stills, of one lattice - each file's cell the first's, in that setting or another
// (an axis permutation: the merge puts every crystal on the first file's axes, RelabelOnSublattice) - unless
// the cell is given (cell_given, -C), and of one space group unless it is given (space_group_given, -S).
std::string ScaleInputsRefusal(const std::vector<ScaleInput> &inputs, bool cell_given, bool space_group_given,
const IndexingSettings &settings);
// The frames of one more file put after those of the files before it: their image numbers moved on by the
// frames already there, so that frames of different files stay distinct.
void AppendScaleInput(std::vector<IntegrationOutcome> &outcomes, std::vector<IntegrationOutcome> &&more);
// Of the frames [first, last) as MergeStills left them, the crystals that went into the merge: on the cell
// merged in, scaled, and not left out by --min-image-cc.
size_t MergedCrystals(const DiffractionExperiment &experiment, const std::vector<IntegrationOutcome> &outcomes,
size_t first, size_t last);
+90 -9
View File
@@ -176,6 +176,7 @@ Azimuthal integration (--mode azint)
Re-merging (--mode scale)
rugnux --mode scale -o remerged -A myrun_process.h5
rugnux --mode scale -o combined run1_process.h5 run2_process.h5 stills of several runs, as one
Re-scales and merges the reflections of a _process.h5 written with --write-process-h5.
Basic options
@@ -1800,10 +1801,11 @@ int RugnuxCli::Parse(int argc, char **argv) {
exit(EXIT_FAILURE);
}
if (input_files.size() > 1) {
// Several files are one stills dataset (JFJochMultiReader.h for NXmx masters); every way of
// reading that is not a set of stored files is refused here, before anything is opened.
if (mode != RugnuxMode::MX) {
logger.Error("Several inputs are read together in --mode mx only");
// Several files are one stills dataset (JFJochMultiReader.h for NXmx masters), or in --mode scale the
// _process.h5 files of several stills runs, merged as one; every way of reading that is not a set of
// stored files is refused here, before anything is opened.
if (mode != RugnuxMode::MX && mode != RugnuxMode::Scale) {
logger.Error("Several inputs are read together in --mode mx and --mode scale only");
exit(EXIT_FAILURE);
}
if (!broker_url.empty() || simulate_live) {
@@ -1978,7 +1980,8 @@ int RugnuxInputRun::PrepareRugnuxInput() {
return JFJochXFELReader::CanReadPanelTable(f) || JFJochXFELReader::CanReadCXI(f) || JFJochXTCReader::CanRead(f);
};
input_is_crystfel = !crystfel_geometry.empty();
if (input_files.size() > 1 && !input_is_crystfel && !is_foreign(input_file)) {
// --mode scale reads the first of several _process.h5 files here, and the others with its reflections.
if (input_files.size() > 1 && mode != RugnuxMode::Scale && !input_is_crystfel && !is_foreign(input_file)) {
// One frame of a CBF, marCCD or SMV sweep names the whole sweep.
if (JFJochCBFReader::CanRead(input_file) || JFJochMarCCDReader::CanRead(input_file)
|| JFJochSMVReader::CanRead(input_file) || JFJochSFRMReader::CanRead(input_file)) {
@@ -2372,7 +2375,69 @@ int RugnuxInputRun::PrepareRugnuxInput() {
}
const auto total_images = static_cast<int>(reader.GetNumberOfImages());
const int last_image = (end_image < 0 || end_image >= total_images) ? total_images - 1 : end_image;
// Several files: the stills of several runs merged as one, in the order given, each frame with the
// geometry of its own file. The cell, the group, the given cell and the rocking curve merged in are
// the first file's, unless -C / -S give them; the ice gate pools every file's frames.
std::vector<float> ice_ring_score = dataset->ice_ring_score;
std::vector<float> spot_count_ice_rings = dataset->spot_count_ice_rings;
std::vector<float> spot_count_ice_control = dataset->spot_count_ice_control;
std::vector<std::unique_ptr<JFJochHDF5Reader>> more; // the files after the first
if (input_files.size() > 1) {
if (start_image != 0 || end_image >= 0) {
logger.Error("-s / -e select images of one file; with several _process.h5 files each is merged whole");
throw RugnuxInputFailed{EXIT_FAILURE};
}
const auto describe = [this](const std::string &f, const JFJochReaderDataset &d) {
const auto sg = d.experiment.GetGemmiSpaceGroup();
return ScaleInput{f, d.experiment.GetGoniometer().has_value() && !force_still,
d.experiment.GetUnitCell(),
sg ? std::optional<int>(sg->number) : std::nullopt};
};
std::vector<ScaleInput> inputs = {describe(input_file, *dataset)};
for (size_t k = 1; k < input_files.size(); ++k) {
auto r = std::make_unique<JFJochHDF5Reader>();
try {
r->ReadFile(input_files[k]);
} catch (const std::exception &e) {
logger.Error("{} cannot be read: {}", input_files[k], e.what());
throw RugnuxInputFailed{EXIT_FAILURE};
}
if (!r->GetDataset()) {
logger.Error("No experiment dataset found in {}", input_files[k]);
throw RugnuxInputFailed{EXIT_FAILURE};
}
inputs.push_back(describe(input_files[k], *r->GetDataset()));
more.push_back(std::move(r));
}
if (const auto refusal = ScaleInputsRefusal(inputs, fixed_reference_unit_cell.has_value(),
space_group != nullptr, IndexingSettings());
!refusal.empty()) {
logger.Error("{}", refusal);
throw RugnuxInputFailed{EXIT_FAILURE};
}
}
auto reflections = hdf5_reader.ReadReflections(start_image, last_image);
std::vector<size_t> input_first_frame = {0}; // of each file, in `reflections`
for (const auto &r : more) {
input_first_frame.push_back(reflections.size());
AppendScaleInput(reflections, r->ReadReflections());
const auto &d = *r->GetDataset();
ice_ring_score.insert(ice_ring_score.end(), d.ice_ring_score.begin(), d.ice_ring_score.end());
spot_count_ice_rings.insert(spot_count_ice_rings.end(), d.spot_count_ice_rings.begin(),
d.spot_count_ice_rings.end());
spot_count_ice_control.insert(spot_count_ice_control.end(), d.spot_count_ice_control.begin(),
d.spot_count_ice_control.end());
}
input_first_frame.push_back(reflections.size());
std::vector<ScaleInputCount> input_counts;
if (input_files.size() > 1)
for (size_t k = 0; k < input_files.size(); ++k) {
ScaleInputCount c{input_files[k], static_cast<int64_t>(input_first_frame[k + 1] - input_first_frame[k])};
for (size_t i = input_first_frame[k]; i < input_first_frame[k + 1]; ++i)
c.indexed += !reflections[i].reflections.empty();
logger.Info("Input {}: {} images, {} indexed, from {}", k + 1, c.images, c.indexed, c.file);
input_counts.push_back(c);
}
DiffractionExperiment experiment(dataset->experiment);
configure_offline_output(experiment, output_prefix);
@@ -2547,7 +2612,7 @@ int RugnuxInputRun::PrepareRugnuxInput() {
// --scale reaches the same verdict on the same data as the pipeline that produced it.
double ice_sum = 0.0;
size_t ice_n = 0;
for (const float s : dataset->ice_ring_score)
for (const float s : ice_ring_score)
if (std::isfinite(s)) {
ice_sum += s;
++ice_n;
@@ -2555,9 +2620,9 @@ int RugnuxInputRun::PrepareRugnuxInput() {
const float ice_min_score = experiment.GetScalingSettings().GetIceMinScore();
// ...and the spot channel, pooled over the run exactly as the full pipeline pools it.
double ring_sum = 0.0, ctrl_sum = 0.0;
for (const float v : dataset->spot_count_ice_rings)
for (const float v : spot_count_ice_rings)
if (std::isfinite(v)) ring_sum += v;
for (const float v : dataset->spot_count_ice_control)
for (const float v : spot_count_ice_control)
if (std::isfinite(v)) ctrl_sum += v;
// Empty control + spots on the rings = the strongest ice evidence, not its absence.
const double ice_spot_ratio = ctrl_sum > 0.0 ? ring_sum / ctrl_sum : (ring_sum > 0.0 ? 1.0e3 : 0.0);
@@ -2657,6 +2722,13 @@ int RugnuxInputRun::PrepareRugnuxInput() {
dataset->stills_run_width, flag_ice, ice_width, live_checkpoints,
live_final_full, nthreads, logger);
experiment.ImagesPerTrigger(static_cast<int64_t>(so.images));
for (size_t k = 0; k < input_counts.size(); ++k) {
auto &c = input_counts[k];
c.merged = static_cast<int64_t>(MergedCrystals(experiment, reflections, input_first_frame[k],
input_first_frame[k + 1]));
logger.Info("Input {}: {} of {} indexed crystals merged, {} left out, from {}", k + 1, c.merged,
c.indexed, c.indexed - c.merged, c.file);
}
auto &sm = so.merge;
merged_reflections = std::move(sm.merged);
merged_statistics = std::move(sm.statistics);
@@ -2869,7 +2941,8 @@ int RugnuxInputRun::PrepareRugnuxInput() {
// --mode scale re-merges stored reflections, so it determines a merging result and gets the
// same report. It has no ProcessResult of its own; fill the parts that exist here.
ProcessResult scale_result;
scale_result.images_processed = reader.GetNumberOfImages();
scale_result.images_processed = input_files.size() > 1 ? reflections.size() : reader.GetNumberOfImages();
scale_result.scale_inputs = input_counts;
scale_result.consensus_cell = experiment.GetUnitCell();
scale_result.space_group = experiment.GetGemmiSpaceGroup();
scale_result.used_beam_x_pxl = experiment.GetBeamX_pxl();
@@ -3984,6 +4057,14 @@ int RugnuxCli::Run() {
|| JFJochXTCReader::CanRead(f);
return d;
};
// --mode scale: several _process.h5 files are one stills merge, checked where they are read.
if (mode == RugnuxMode::Scale) {
try {
return RugnuxInputRun(*this).Process();
} catch (const RugnuxInputFailed &f) {
return f.code;
}
}
// --force-still: every input is stills, so several masters are one stills run, rotation masters
// included. Frame files still go through the plan: their frames name one sweep.
if (force_still && std::none_of(input_files.begin(), input_files.end(), [&](const std::string &f) {
+1
View File
@@ -128,6 +128,7 @@ ADD_EXECUTABLE(jfjoch_test
LiveCollectionTest.cpp
RugnuxLiveStatusTest.cpp
RugnuxLiveTriageTest.cpp
RugnuxScaleInputsTest.cpp
StillsGPUAnalysisTest.cpp
StillsGPUAnalysisFPGATest.cpp
RugnuxStructureTest.cpp
+92
View File
@@ -0,0 +1,92 @@
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <catch2/catch_all.hpp>
#include "../common/DiffractionExperiment.h"
#include "../rugnux/ResultReport.h"
#include "../rugnux/StillsLiveMerge.h"
namespace {
const UnitCell orthorhombic_uc{.a = 40.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
ScaleInput Still(const std::string &file, const UnitCell &cell, int space_group = 19) {
return ScaleInput{file, false, cell, space_group};
}
}
TEST_CASE("ScaleInputsRefusal_OneCrystal", "[RugnuxScale]") {
const IndexingSettings settings;
// The same cell, and the same lattice on permuted axes (the merge relabels those onto the first's).
const UnitCell permuted{.a = 60.0f, .b = 90.0f, .c = 40.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
const UnitCell refined{.a = 40.05f, .b = 59.95f, .c = 90.1f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", refined), Still("c", permuted)}, false, false,
settings).empty());
}
TEST_CASE("ScaleInputsRefusal_CellsDisagree", "[RugnuxScale]") {
const IndexingSettings settings;
const UnitCell other{.a = 45.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
// An axis doubled: a lattice that contains the first, but is not it.
const UnitCell doubled{.a = 80.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
const auto refusal = ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", other)}, false, false, settings);
CHECK(refusal.find("not one lattice") != std::string::npos);
CHECK(refusal.find("a=40.000 b=60.000 c=90.000") != std::string::npos);
CHECK(refusal.find("a=45.000 b=60.000 c=90.000") != std::string::npos);
CHECK(refusal.find("-C") != std::string::npos);
CHECK(!ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", doubled)}, false, false, settings).empty());
// A cell given on the command line is what they are merged in.
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", other)}, true, false, settings).empty());
// A file without a cell, unless one is given.
ScaleInput no_cell{"b", false, std::nullopt, 19};
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), no_cell}, false, false, settings).find("no unit cell")
!= std::string::npos);
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), no_cell}, true, false, settings).empty());
}
TEST_CASE("ScaleInputsRefusal_SpaceGroupsDisagree", "[RugnuxScale]") {
const IndexingSettings settings;
const auto refusal = ScaleInputsRefusal({Still("a", orthorhombic_uc, 19), Still("b", orthorhombic_uc, 16)},
false, false, settings);
CHECK(refusal.find("space group 19") != std::string::npos);
CHECK(refusal.find("-S") != std::string::npos);
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc, 19), Still("b", orthorhombic_uc, 16)}, false, true,
settings).empty());
}
TEST_CASE("ScaleInputsRefusal_Rotation", "[RugnuxScale]") {
const IndexingSettings settings;
const ScaleInput sweep{"r", true, orthorhombic_uc, 19};
// Stills and a rotation sweep: refused, whatever is given.
const auto mixed = ScaleInputsRefusal({Still("a", orthorhombic_uc), sweep}, true, true, settings);
CHECK(mixed.find("r is a rotation sweep and a is stills") != std::string::npos);
// Several sweeps: refused, pointing to --mode mx.
const auto sweeps = ScaleInputsRefusal({sweep, sweep}, false, false, settings);
CHECK(sweeps.find("--mode mx") != std::string::npos);
}
TEST_CASE("AppendScaleInput_FramesStayDistinct", "[RugnuxScale]") {
std::vector<IntegrationOutcome> all(3);
all[1].reflections.push_back(Reflection{.image_number = 1.0f});
std::vector<IntegrationOutcome> more(2);
more[0].reflections.push_back(Reflection{.image_number = 0.0f});
more[1].reflections.push_back(Reflection{.image_number = 1.0f});
AppendScaleInput(all, std::move(more));
REQUIRE(all.size() == 5);
CHECK(all[1].reflections[0].image_number == 1.0f);
CHECK(all[3].reflections[0].image_number == 3.0f);
CHECK(all[4].reflections[0].image_number == 4.0f);
}
TEST_CASE("ScaleInputs_Report", "[RugnuxScale]") {
DiffractionExperiment x(DetJF(1));
ProcessResult result;
result.images_processed = 300;
result.scale_inputs = {{"run1_process.h5", 100, 90, 85}, {"run2_process.h5", 200, 150, 140}};
const std::string text = RenderResultReport("p", "run1_process.h5", x, result);
CHECK(text.find("\nINPUT_01= 100 90 85 5 run1_process.h5\n") != std::string::npos);
CHECK(text.find("\nINPUT_02= 200 150 140 10 run2_process.h5\n") != std::string::npos);
// One file: no rows.
result.scale_inputs.clear();
CHECK(RenderResultReport("p", "in.h5", x, result).find("INPUT_01") == std::string::npos);
}