rugnux --mode scale: merge several stills _process.h5 files as one run
`rugnux --mode scale -o combined run1_process.h5 run2_process.h5 ...` reads every file's reflections and per-image metadata, keeps each frame's own geometry, numbers the frames on from file to file, and runs the existing stills merge (MergeStills) on the union - so the cell-form selection, the relabel onto the run's cell (RelabelOnSublattice, which also takes crystals of a file indexed on permuted axes) and the data-only indexing-ambiguity resolution act over all crystals of all files at once. The cell, group, given cell and rocking curve are the first file's; the ice gate pools all. Checks (ScaleInputsRefusal): every file stills; each file's cell the first's lattice in some setting (a change of basis of determinant one), unless -C/-z gives the cell; one space-group number unless -S. A mix of stills and rotation files is refused, and so are several rotation files (pointing to --mode mx, which merges sweeps from their masters); the multi-sweep machinery is not touched. -s/-e are refused with several files; --live-checkpoints replays the files one after another. Per file the log and the report (INPUT_nn= images indexed merged left-out file) give the counts; MergeOnTheFly::Mask is made public so "merged" is the merge's own definition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
This commit is contained in:
1 parent
d8a6ce3a74
commit
74acfc433e
11 files changed
+335
-14
No files matched your search
@@ -311,6 +311,32 @@ from the one each image was indexed in — the file records the change of basis
|
||||
describe the same frame. A file whose stored cell and reflections disagree stops with a message
|
||||
naming both cells and the exact `-S`/`-C` override to merge it in its own setting.
|
||||
|
||||
### Several files
|
||||
|
||||
```
|
||||
rugnux --mode scale -o combined run1_process.h5 run2_process.h5 run3_process.h5
|
||||
```
|
||||
|
||||
The `_process.h5` files of several stills runs of one crystal are merged as one run, as serial data
|
||||
from several collections usually are. Each frame keeps the geometry, wavelength and detector of its own
|
||||
file; the frames are taken file after file, in the order given, and numbered on from one file to the
|
||||
next, so the frames of different files stay distinct. The union is then merged exactly as one file is:
|
||||
the frames of the run's crystal form, each put on the run's cell, the indexing ambiguity resolved from
|
||||
the data over all the crystals together - so crystals of different files indexed in different
|
||||
alternative indexings, or on permuted axes of the same lattice, end up in one - and one scaling,
|
||||
post-refinement and merge. The cell, space group, given cell (`/entry/MX/stillsGivenUnitCell`) and
|
||||
rocking curve merged with are the first file's; the ice-ring gate pools the frames of all of them.
|
||||
|
||||
The files must be of one crystal: each file's cell the first's lattice (in its setting or another) and
|
||||
the same space group. A run stops with a message naming both cells, or both groups, where they are not -
|
||||
unless the cell is given (`-C`, or a reference `-z`) and, for the group, `-S`: then everything is merged
|
||||
in what is given, and the crystals of a file not on that cell are left out. A mix of stills and rotation
|
||||
files is refused, and so are several rotation files (`--mode scale` merges one sweep at a time; several
|
||||
sweeps of one crystal are merged together from their masters by `--mode mx`). Under `--force-still` a
|
||||
rotation file counts as stills. `-s`/`-e` select images of one file and are refused with several. The
|
||||
log and the report ([`INPUT_nn`](RUGNUX_REPORT.md#the-headline-keys)) give each file's images, indexed
|
||||
crystals, merged crystals and those left out; the statistics are those of the merge of all of them.
|
||||
|
||||
## The merged mmCIF (`<prefix>.cif`)
|
||||
|
||||
The `.cif` follows the PDBx/mmCIF dictionary and holds one data block, `data_sample`: the merged
|
||||
|
||||
+7
-3
@@ -84,10 +84,14 @@ The binning is set by the [azimuthal-integration options](#azimuthal-integration
|
||||
```
|
||||
rugnux -o myrun --write-process-h5 dataset_master.h5 # integrate and merge once
|
||||
rugnux --mode scale -o remerged -A myrun_process.h5 # re-merge (on stills, -A keeps the Bijvoet hands apart)
|
||||
rugnux --mode scale -o combined run1_process.h5 run2_process.h5 run3_process.h5 # stills of several runs, as one
|
||||
```
|
||||
|
||||
Re-scales and merges the reflections already integrated into a `_process.h5`, without re-running
|
||||
spot finding or integration ([Re-scaling and re-merging](RUGNUX_ADVANCED.md#re-scaling-and-re-merging-rugnux---mode-scale)).
|
||||
Several `_process.h5` files of stills are merged as one run, their crystals scaled together and the indexing
|
||||
ambiguity resolved over all of them ([Several files](RUGNUX_ADVANCED.md#several-files)); rotation files are
|
||||
taken one at a time.
|
||||
|
||||
## Inputs
|
||||
|
||||
@@ -97,8 +101,8 @@ one dataset - `rugnux run*.h5` (SACLA), `*.cxi`, `*.xtc` (LCLS) - or several Jun
|
||||
masters of stills - or of rotation sweeps under `--force-still` - read as one run: their images one after
|
||||
another, merged together, with the indexing rate reported per master as well. Such masters must share the detector, wavelength,
|
||||
distance and beam centre. A grid scan an input holds (a fixed-target chip) is not carried into the run: several chips are
|
||||
not one grid, so the run has no grid map. Several inputs must be one format; any `--mode` other than `mx`, and
|
||||
following a broker, take one.
|
||||
not one grid, so the run has no grid map. Several inputs must be one format; `--mode scale` takes several `_process.h5` files of
|
||||
stills (above), and any other `--mode`, and following a broker, take one.
|
||||
|
||||
Several rotation sweeps are told apart from the files: one sweep per CBF/marCCD/SMV/SFRM file-name
|
||||
template (the name with its frame number taken out - any frame of a sweep names the whole sweep, so
|
||||
@@ -296,7 +300,7 @@ serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rota
|
||||
| `--no-scaling-corrections` | rot3d: disable the default-on decay + absorption + modulation correction surfaces fitted on the fulls after scale-fulls |
|
||||
| `--relative-b[=deg]` | rot3d: fit a per-batch relative-*B* beyond the single decay slope over deg-degree batches, cross-validated (default 10° when bare; off otherwise) |
|
||||
| `--simple-stills` | Stills: treat every reflection as a full (*p* = 1, single-pass scale/merge) — disables the default-on physical partiality post-refinement |
|
||||
| `--live-checkpoints <f1>[,<f2>...]` | `--mode scale`, stills (for developers): replay the stored frames in image order as the broker's live merge of a serial collection merges them - a merge at each fraction of the images (e.g. `0.5,0.75`), each crystal after the first checkpoint scaled, post-refined and placed in its indexing against the latest checkpoint's merge as it comes, and after the last frame one merge of all of them without a scaling or post-refinement over the whole run. The log gives each checkpoint's statistics and time, and the time of the merge after the last frame; the merge written is that last one. Without it `--mode scale` merges cold, as the broker does when no checkpoint is reached Each fraction 0.01-0.99. |
|
||||
| `--live-checkpoints <f1>[,<f2>...]` | `--mode scale`, stills (for developers): replay the stored frames in image order as the broker's live merge of a serial collection merges them - a merge at each fraction of the images (e.g. `0.5,0.75`), each crystal after the first checkpoint scaled, post-refined and placed in its indexing against the latest checkpoint's merge as it comes, and after the last frame one merge of all of them without a scaling or post-refinement over the whole run. The log gives each checkpoint's statistics and time, and the time of the merge after the last frame; the merge written is that last one. With several `_process.h5` files the frames are replayed file after file, in the order given, as one collection. Without it `--mode scale` merges cold, as the broker does when no checkpoint is reached Each fraction 0.01-0.99. |
|
||||
| `--live-final-full` | With `--live-checkpoints`: the merge after the last frame scales and post-refines every crystal over the whole run, as a cold merge does, instead of merging the crystals as they were prepared against the checkpoints - what the broker's live merge would do with a full scaling pass at the end; its time is in the log beside the incremental one's. |
|
||||
| `--no-expected-variance-merge` | Stills: disable the default expected-variance merge weighting (which rebuilds each weak observation's signal variance at the reflection mean to de-bias the inverse-variance merge); restores observed-sigma weighting |
|
||||
| `--capture-uncertainty <num>` | Systematic sigma num·(1−captured fraction)·⟨I⟩: on rot3d fulls the share of the rocking curve not captured, on stills the share of each partial its partiality does not see (default: 1.0 for rotation, 0 otherwise) |
|
||||
|
||||
@@ -238,6 +238,7 @@ The keys most scripts read, by section; the rest of this page explains the ones
|
||||
|---|---|
|
||||
| `EXPERIMENT_TYPE` | `ROTATION` or `STILLS` (section 1) |
|
||||
| `IMAGES_PROCESSED`, `WAVELENGTH` | Images the run read, and the wavelength (Å) |
|
||||
| `INPUT_nn` | `--mode scale` of several `_process.h5` files: one row per file, in the order given - its images, the crystals indexed in it, those of them merged and those left out (off the run's cell form, a collapsed scale, `--min-image-cc`), and the file ([Several files](RUGNUX_ADVANCED.md#several-files)) |
|
||||
| `OSCILLATION_RANGE`, `STARTING_ANGLE`, `ROTATION_AXIS` | Rotation only: the per-image rotation the run integrated at (deg: the file's, scaled where the run adopted a goniometer rotation scale), the first image's angle and the axis |
|
||||
| `DETECTOR_DISTANCE`, `BEAM_CENTRE`, `DETECTOR_TILT` | The geometry the run integrated at (mm; the point of normal incidence in pixels; the three PONI rotations in degrees) - on a rotation run the post-refined one, not the header's |
|
||||
| `DIRECT_BEAM` | Where the direct beam meets the detector under that tilt (pixels) |
|
||||
|
||||
@@ -424,8 +424,6 @@ class MergeOnTheFly {
|
||||
double reject_up = 0.0, reject_down = 0.0;
|
||||
size_t reject_count = 0;
|
||||
|
||||
bool Mask(const IntegrationOutcome &outcome) const;
|
||||
|
||||
// One accepted observation, reduced to what the accumulator adds for it.
|
||||
struct Contribution {
|
||||
uint64_t key;
|
||||
@@ -442,6 +440,8 @@ class MergeOnTheFly {
|
||||
void Accumulate(const std::vector<Contribution> &contributions, int64_t image_id);
|
||||
public:
|
||||
MergeOnTheFly(const DiffractionExperiment &x);
|
||||
// Whether the merge leaves this crystal out: off the reference cell, or below the per-image CC limit.
|
||||
bool Mask(const IntegrationOutcome &outcome) const;
|
||||
MergeOnTheFly& ReferenceCell(const std::optional<UnitCell> &cell);
|
||||
MergeOnTheFly& ExcludeIceRings(bool input) { exclude_ice_rings = input; return *this; }
|
||||
MergeOnTheFly& FilterByImageCC(bool input) { filter_by_image_cc = input; return *this; }
|
||||
|
||||
@@ -910,6 +910,13 @@ ReportDocument BuildReportDocument(const std::string &output_prefix,
|
||||
s.title = "1. DATA SET AND GEOMETRY";
|
||||
Add(s, KeyEnum("EXPERIMENT_TYPE", rotation ? "ROTATION" : "STILLS", {"ROTATION", "STILLS"}));
|
||||
Add(s, KeyInt("IMAGES_PROCESSED", static_cast<int64_t>(result.images_processed)));
|
||||
// --mode scale of several _process.h5 files: images, indexed, merged and left-out crystals, file.
|
||||
for (size_t i = 0; i < result.scale_inputs.size(); ++i) {
|
||||
const auto &in = result.scale_inputs[i];
|
||||
Add(s, KeyText(fmt::format("INPUT_{:02}", i + 1).c_str(),
|
||||
fmt::format("{} {} {} {} {}", in.images, in.indexed, in.merged, in.indexed - in.merged,
|
||||
in.file)));
|
||||
}
|
||||
Add(s, KeyReal("WAVELENGTH", experiment.GetWavelength_A(), "{:.5f}"));
|
||||
if (const auto gonio = experiment.GetGoniometer()) {
|
||||
// The oscillation the run integrated at: the file's, scaled where the run adopted a
|
||||
|
||||
@@ -582,9 +582,17 @@ struct RugnuxOutputs {
|
||||
std::optional<StartMessage> process_start;
|
||||
};
|
||||
|
||||
// One of several _process.h5 files --mode scale merged as one: its frames, those of them indexed, and those of
|
||||
// them that went into the merge (MergedCrystals).
|
||||
struct ScaleInputCount {
|
||||
std::string file;
|
||||
int64_t images = 0, indexed = 0, merged = 0;
|
||||
};
|
||||
|
||||
struct ProcessResult {
|
||||
bool cancelled = false;
|
||||
uint64_t images_processed = 0;
|
||||
std::vector<ScaleInputCount> scale_inputs; // --mode scale of several files, in the order given
|
||||
// The file image of processed-image ordinal f is first_image + f * image_stride (-s, --stride).
|
||||
int first_image = 0;
|
||||
int image_stride = 1;
|
||||
|
||||
@@ -306,3 +306,79 @@ ProcessResult StillsLiveResult(const DiffractionExperiment &experiment, const St
|
||||
result.twinning = out.twinning;
|
||||
return result;
|
||||
}
|
||||
|
||||
namespace {
|
||||
std::string CellText(const UnitCell &c) {
|
||||
return fmt::format("a={:.3f} b={:.3f} c={:.3f} alpha={:.2f} beta={:.2f} gamma={:.2f}", c.a, c.b, c.c, c.alpha,
|
||||
c.beta, c.gamma);
|
||||
}
|
||||
|
||||
// The same lattice in this setting or another: a change of basis of determinant one carries `cell` onto
|
||||
// `reference` (an axis multiple, a basis of larger determinant, is a different lattice).
|
||||
bool SameLattice(const UnitCell &cell, const UnitCell &reference, const IndexingSettings &settings) {
|
||||
for (const auto &m : FindSublatticeBases(CrystalLattice(cell), reference, settings)) {
|
||||
const int det = m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1])
|
||||
- m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0])
|
||||
+ m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0]);
|
||||
if (std::abs(det) == 1)
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
std::string ScaleInputsRefusal(const std::vector<ScaleInput> &inputs, bool cell_given, bool space_group_given,
|
||||
const IndexingSettings &settings) {
|
||||
size_t rotation = 0;
|
||||
for (const auto &in : inputs)
|
||||
rotation += in.rotation;
|
||||
if (rotation > 0 && rotation < inputs.size()) {
|
||||
const auto rot = std::find_if(inputs.begin(), inputs.end(), [](const ScaleInput &i) { return i.rotation; });
|
||||
const auto still = std::find_if(inputs.begin(), inputs.end(), [](const ScaleInput &i) { return !i.rotation; });
|
||||
return fmt::format("{} is a rotation sweep and {} is stills: --mode scale merges several _process.h5 files "
|
||||
"of stills only", rot->file, still->file);
|
||||
}
|
||||
if (rotation > 0)
|
||||
return fmt::format("{} _process.h5 files of rotation sweeps: --mode scale merges several files of stills "
|
||||
"only. To merge sweeps of one crystal together, give their masters to --mode mx",
|
||||
rotation);
|
||||
const ScaleInput &first = inputs.front();
|
||||
for (const auto &in : inputs) {
|
||||
if (!cell_given) {
|
||||
if (!in.cell)
|
||||
return fmt::format("{} states no unit cell: give the cell to merge in with -C", in.file);
|
||||
if (!SameLattice(*in.cell, *first.cell, settings))
|
||||
return fmt::format("{} was integrated in {} and {} in {}, not one lattice, so their crystals are "
|
||||
"not merged together. If they are one crystal, give the cell to merge in with -C",
|
||||
first.file, CellText(*first.cell), in.file, CellText(*in.cell));
|
||||
}
|
||||
if (!space_group_given && in.space_group_number != first.space_group_number)
|
||||
return fmt::format("{} is in space group {} and {} in {}: give the group to merge in with -S", first.file,
|
||||
first.space_group_number ? std::to_string(*first.space_group_number) : "none",
|
||||
in.file, in.space_group_number ? std::to_string(*in.space_group_number) : "none");
|
||||
}
|
||||
return {};
|
||||
}
|
||||
|
||||
void AppendScaleInput(std::vector<IntegrationOutcome> &outcomes, std::vector<IntegrationOutcome> &&more) {
|
||||
const auto offset = static_cast<float>(outcomes.size());
|
||||
for (auto &o : more) {
|
||||
for (auto &r : o.reflections)
|
||||
r.image_number += offset;
|
||||
outcomes.push_back(std::move(o));
|
||||
}
|
||||
}
|
||||
|
||||
size_t MergedCrystals(const DiffractionExperiment &experiment, const std::vector<IntegrationOutcome> &outcomes,
|
||||
size_t first, size_t last) {
|
||||
MergeOnTheFly merge(experiment);
|
||||
merge.ReferenceCell(experiment.GetUnitCell());
|
||||
merge.FilterByImageCC(experiment.GetScalingSettings().GetMinCCForImage() > 0.0);
|
||||
size_t n = 0;
|
||||
for (size_t i = first; i < last; ++i)
|
||||
n += !merge.Mask(outcomes[i])
|
||||
&& std::any_of(outcomes[i].reflections.begin(), outcomes[i].reflections.end(), [](const Reflection &r) {
|
||||
return r.image_scale_corr > 0.0f && std::isfinite(r.image_scale_corr);
|
||||
});
|
||||
return n;
|
||||
}
|
||||
@@ -4,6 +4,7 @@
|
||||
#pragma once
|
||||
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "Rugnux.h"
|
||||
@@ -125,3 +126,27 @@ StillsMergeOutput ReplayLiveStillsMerge(const DiffractionExperiment &experiment,
|
||||
// it, never a verdict.
|
||||
ProcessResult StillsLiveResult(const DiffractionExperiment &experiment, const StillsMergeOutput &out,
|
||||
int64_t images_analysed, int64_t images_indexed);
|
||||
|
||||
// One _process.h5 of several that --mode scale merges together, as the file states it.
|
||||
struct ScaleInput {
|
||||
std::string file;
|
||||
bool rotation = false; // a rotation sweep (not under --force-still)
|
||||
std::optional<UnitCell> cell;
|
||||
std::optional<int> space_group_number;
|
||||
};
|
||||
|
||||
// Why --mode scale cannot merge these files together; empty where it can. Several files are merged as one
|
||||
// stills run: all of them stills, of one lattice - each file's cell the first's, in that setting or another
|
||||
// (an axis permutation: the merge puts every crystal on the first file's axes, RelabelOnSublattice) - unless
|
||||
// the cell is given (cell_given, -C), and of one space group unless it is given (space_group_given, -S).
|
||||
std::string ScaleInputsRefusal(const std::vector<ScaleInput> &inputs, bool cell_given, bool space_group_given,
|
||||
const IndexingSettings &settings);
|
||||
|
||||
// The frames of one more file put after those of the files before it: their image numbers moved on by the
|
||||
// frames already there, so that frames of different files stay distinct.
|
||||
void AppendScaleInput(std::vector<IntegrationOutcome> &outcomes, std::vector<IntegrationOutcome> &&more);
|
||||
|
||||
// Of the frames [first, last) as MergeStills left them, the crystals that went into the merge: on the cell
|
||||
// merged in, scaled, and not left out by --min-image-cc.
|
||||
size_t MergedCrystals(const DiffractionExperiment &experiment, const std::vector<IntegrationOutcome> &outcomes,
|
||||
size_t first, size_t last);
|
||||
+90
-9
@@ -176,6 +176,7 @@ Azimuthal integration (--mode azint)
|
||||
|
||||
Re-merging (--mode scale)
|
||||
rugnux --mode scale -o remerged -A myrun_process.h5
|
||||
rugnux --mode scale -o combined run1_process.h5 run2_process.h5 stills of several runs, as one
|
||||
Re-scales and merges the reflections of a _process.h5 written with --write-process-h5.
|
||||
|
||||
Basic options
|
||||
@@ -1800,10 +1801,11 @@ int RugnuxCli::Parse(int argc, char **argv) {
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
if (input_files.size() > 1) {
|
||||
// Several files are one stills dataset (JFJochMultiReader.h for NXmx masters); every way of
|
||||
// reading that is not a set of stored files is refused here, before anything is opened.
|
||||
if (mode != RugnuxMode::MX) {
|
||||
logger.Error("Several inputs are read together in --mode mx only");
|
||||
// Several files are one stills dataset (JFJochMultiReader.h for NXmx masters), or in --mode scale the
|
||||
// _process.h5 files of several stills runs, merged as one; every way of reading that is not a set of
|
||||
// stored files is refused here, before anything is opened.
|
||||
if (mode != RugnuxMode::MX && mode != RugnuxMode::Scale) {
|
||||
logger.Error("Several inputs are read together in --mode mx and --mode scale only");
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
if (!broker_url.empty() || simulate_live) {
|
||||
@@ -1978,7 +1980,8 @@ int RugnuxInputRun::PrepareRugnuxInput() {
|
||||
return JFJochXFELReader::CanReadPanelTable(f) || JFJochXFELReader::CanReadCXI(f) || JFJochXTCReader::CanRead(f);
|
||||
};
|
||||
input_is_crystfel = !crystfel_geometry.empty();
|
||||
if (input_files.size() > 1 && !input_is_crystfel && !is_foreign(input_file)) {
|
||||
// --mode scale reads the first of several _process.h5 files here, and the others with its reflections.
|
||||
if (input_files.size() > 1 && mode != RugnuxMode::Scale && !input_is_crystfel && !is_foreign(input_file)) {
|
||||
// One frame of a CBF, marCCD or SMV sweep names the whole sweep.
|
||||
if (JFJochCBFReader::CanRead(input_file) || JFJochMarCCDReader::CanRead(input_file)
|
||||
|| JFJochSMVReader::CanRead(input_file) || JFJochSFRMReader::CanRead(input_file)) {
|
||||
@@ -2372,7 +2375,69 @@ int RugnuxInputRun::PrepareRugnuxInput() {
|
||||
}
|
||||
const auto total_images = static_cast<int>(reader.GetNumberOfImages());
|
||||
const int last_image = (end_image < 0 || end_image >= total_images) ? total_images - 1 : end_image;
|
||||
// Several files: the stills of several runs merged as one, in the order given, each frame with the
|
||||
// geometry of its own file. The cell, the group, the given cell and the rocking curve merged in are
|
||||
// the first file's, unless -C / -S give them; the ice gate pools every file's frames.
|
||||
std::vector<float> ice_ring_score = dataset->ice_ring_score;
|
||||
std::vector<float> spot_count_ice_rings = dataset->spot_count_ice_rings;
|
||||
std::vector<float> spot_count_ice_control = dataset->spot_count_ice_control;
|
||||
std::vector<std::unique_ptr<JFJochHDF5Reader>> more; // the files after the first
|
||||
if (input_files.size() > 1) {
|
||||
if (start_image != 0 || end_image >= 0) {
|
||||
logger.Error("-s / -e select images of one file; with several _process.h5 files each is merged whole");
|
||||
throw RugnuxInputFailed{EXIT_FAILURE};
|
||||
}
|
||||
const auto describe = [this](const std::string &f, const JFJochReaderDataset &d) {
|
||||
const auto sg = d.experiment.GetGemmiSpaceGroup();
|
||||
return ScaleInput{f, d.experiment.GetGoniometer().has_value() && !force_still,
|
||||
d.experiment.GetUnitCell(),
|
||||
sg ? std::optional<int>(sg->number) : std::nullopt};
|
||||
};
|
||||
std::vector<ScaleInput> inputs = {describe(input_file, *dataset)};
|
||||
for (size_t k = 1; k < input_files.size(); ++k) {
|
||||
auto r = std::make_unique<JFJochHDF5Reader>();
|
||||
try {
|
||||
r->ReadFile(input_files[k]);
|
||||
} catch (const std::exception &e) {
|
||||
logger.Error("{} cannot be read: {}", input_files[k], e.what());
|
||||
throw RugnuxInputFailed{EXIT_FAILURE};
|
||||
}
|
||||
if (!r->GetDataset()) {
|
||||
logger.Error("No experiment dataset found in {}", input_files[k]);
|
||||
throw RugnuxInputFailed{EXIT_FAILURE};
|
||||
}
|
||||
inputs.push_back(describe(input_files[k], *r->GetDataset()));
|
||||
more.push_back(std::move(r));
|
||||
}
|
||||
if (const auto refusal = ScaleInputsRefusal(inputs, fixed_reference_unit_cell.has_value(),
|
||||
space_group != nullptr, IndexingSettings());
|
||||
!refusal.empty()) {
|
||||
logger.Error("{}", refusal);
|
||||
throw RugnuxInputFailed{EXIT_FAILURE};
|
||||
}
|
||||
}
|
||||
auto reflections = hdf5_reader.ReadReflections(start_image, last_image);
|
||||
std::vector<size_t> input_first_frame = {0}; // of each file, in `reflections`
|
||||
for (const auto &r : more) {
|
||||
input_first_frame.push_back(reflections.size());
|
||||
AppendScaleInput(reflections, r->ReadReflections());
|
||||
const auto &d = *r->GetDataset();
|
||||
ice_ring_score.insert(ice_ring_score.end(), d.ice_ring_score.begin(), d.ice_ring_score.end());
|
||||
spot_count_ice_rings.insert(spot_count_ice_rings.end(), d.spot_count_ice_rings.begin(),
|
||||
d.spot_count_ice_rings.end());
|
||||
spot_count_ice_control.insert(spot_count_ice_control.end(), d.spot_count_ice_control.begin(),
|
||||
d.spot_count_ice_control.end());
|
||||
}
|
||||
input_first_frame.push_back(reflections.size());
|
||||
std::vector<ScaleInputCount> input_counts;
|
||||
if (input_files.size() > 1)
|
||||
for (size_t k = 0; k < input_files.size(); ++k) {
|
||||
ScaleInputCount c{input_files[k], static_cast<int64_t>(input_first_frame[k + 1] - input_first_frame[k])};
|
||||
for (size_t i = input_first_frame[k]; i < input_first_frame[k + 1]; ++i)
|
||||
c.indexed += !reflections[i].reflections.empty();
|
||||
logger.Info("Input {}: {} images, {} indexed, from {}", k + 1, c.images, c.indexed, c.file);
|
||||
input_counts.push_back(c);
|
||||
}
|
||||
|
||||
DiffractionExperiment experiment(dataset->experiment);
|
||||
configure_offline_output(experiment, output_prefix);
|
||||
@@ -2547,7 +2612,7 @@ int RugnuxInputRun::PrepareRugnuxInput() {
|
||||
// --scale reaches the same verdict on the same data as the pipeline that produced it.
|
||||
double ice_sum = 0.0;
|
||||
size_t ice_n = 0;
|
||||
for (const float s : dataset->ice_ring_score)
|
||||
for (const float s : ice_ring_score)
|
||||
if (std::isfinite(s)) {
|
||||
ice_sum += s;
|
||||
++ice_n;
|
||||
@@ -2555,9 +2620,9 @@ int RugnuxInputRun::PrepareRugnuxInput() {
|
||||
const float ice_min_score = experiment.GetScalingSettings().GetIceMinScore();
|
||||
// ...and the spot channel, pooled over the run exactly as the full pipeline pools it.
|
||||
double ring_sum = 0.0, ctrl_sum = 0.0;
|
||||
for (const float v : dataset->spot_count_ice_rings)
|
||||
for (const float v : spot_count_ice_rings)
|
||||
if (std::isfinite(v)) ring_sum += v;
|
||||
for (const float v : dataset->spot_count_ice_control)
|
||||
for (const float v : spot_count_ice_control)
|
||||
if (std::isfinite(v)) ctrl_sum += v;
|
||||
// Empty control + spots on the rings = the strongest ice evidence, not its absence.
|
||||
const double ice_spot_ratio = ctrl_sum > 0.0 ? ring_sum / ctrl_sum : (ring_sum > 0.0 ? 1.0e3 : 0.0);
|
||||
@@ -2657,6 +2722,13 @@ int RugnuxInputRun::PrepareRugnuxInput() {
|
||||
dataset->stills_run_width, flag_ice, ice_width, live_checkpoints,
|
||||
live_final_full, nthreads, logger);
|
||||
experiment.ImagesPerTrigger(static_cast<int64_t>(so.images));
|
||||
for (size_t k = 0; k < input_counts.size(); ++k) {
|
||||
auto &c = input_counts[k];
|
||||
c.merged = static_cast<int64_t>(MergedCrystals(experiment, reflections, input_first_frame[k],
|
||||
input_first_frame[k + 1]));
|
||||
logger.Info("Input {}: {} of {} indexed crystals merged, {} left out, from {}", k + 1, c.merged,
|
||||
c.indexed, c.indexed - c.merged, c.file);
|
||||
}
|
||||
auto &sm = so.merge;
|
||||
merged_reflections = std::move(sm.merged);
|
||||
merged_statistics = std::move(sm.statistics);
|
||||
@@ -2869,7 +2941,8 @@ int RugnuxInputRun::PrepareRugnuxInput() {
|
||||
// --mode scale re-merges stored reflections, so it determines a merging result and gets the
|
||||
// same report. It has no ProcessResult of its own; fill the parts that exist here.
|
||||
ProcessResult scale_result;
|
||||
scale_result.images_processed = reader.GetNumberOfImages();
|
||||
scale_result.images_processed = input_files.size() > 1 ? reflections.size() : reader.GetNumberOfImages();
|
||||
scale_result.scale_inputs = input_counts;
|
||||
scale_result.consensus_cell = experiment.GetUnitCell();
|
||||
scale_result.space_group = experiment.GetGemmiSpaceGroup();
|
||||
scale_result.used_beam_x_pxl = experiment.GetBeamX_pxl();
|
||||
@@ -3984,6 +4057,14 @@ int RugnuxCli::Run() {
|
||||
|| JFJochXTCReader::CanRead(f);
|
||||
return d;
|
||||
};
|
||||
// --mode scale: several _process.h5 files are one stills merge, checked where they are read.
|
||||
if (mode == RugnuxMode::Scale) {
|
||||
try {
|
||||
return RugnuxInputRun(*this).Process();
|
||||
} catch (const RugnuxInputFailed &f) {
|
||||
return f.code;
|
||||
}
|
||||
}
|
||||
// --force-still: every input is stills, so several masters are one stills run, rotation masters
|
||||
// included. Frame files still go through the plan: their frames name one sweep.
|
||||
if (force_still && std::none_of(input_files.begin(), input_files.end(), [&](const std::string &f) {
|
||||
|
||||
@@ -128,6 +128,7 @@ ADD_EXECUTABLE(jfjoch_test
|
||||
LiveCollectionTest.cpp
|
||||
RugnuxLiveStatusTest.cpp
|
||||
RugnuxLiveTriageTest.cpp
|
||||
RugnuxScaleInputsTest.cpp
|
||||
StillsGPUAnalysisTest.cpp
|
||||
StillsGPUAnalysisFPGATest.cpp
|
||||
RugnuxStructureTest.cpp
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
||||
// SPDX-License-Identifier: GPL-3.0-only
|
||||
|
||||
#include <catch2/catch_all.hpp>
|
||||
|
||||
#include "../common/DiffractionExperiment.h"
|
||||
#include "../rugnux/ResultReport.h"
|
||||
#include "../rugnux/StillsLiveMerge.h"
|
||||
|
||||
namespace {
|
||||
const UnitCell orthorhombic_uc{.a = 40.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
|
||||
|
||||
ScaleInput Still(const std::string &file, const UnitCell &cell, int space_group = 19) {
|
||||
return ScaleInput{file, false, cell, space_group};
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("ScaleInputsRefusal_OneCrystal", "[RugnuxScale]") {
|
||||
const IndexingSettings settings;
|
||||
// The same cell, and the same lattice on permuted axes (the merge relabels those onto the first's).
|
||||
const UnitCell permuted{.a = 60.0f, .b = 90.0f, .c = 40.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
|
||||
const UnitCell refined{.a = 40.05f, .b = 59.95f, .c = 90.1f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
|
||||
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", refined), Still("c", permuted)}, false, false,
|
||||
settings).empty());
|
||||
}
|
||||
|
||||
TEST_CASE("ScaleInputsRefusal_CellsDisagree", "[RugnuxScale]") {
|
||||
const IndexingSettings settings;
|
||||
const UnitCell other{.a = 45.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
|
||||
// An axis doubled: a lattice that contains the first, but is not it.
|
||||
const UnitCell doubled{.a = 80.0f, .b = 60.0f, .c = 90.0f, .alpha = 90.0f, .beta = 90.0f, .gamma = 90.0f};
|
||||
const auto refusal = ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", other)}, false, false, settings);
|
||||
CHECK(refusal.find("not one lattice") != std::string::npos);
|
||||
CHECK(refusal.find("a=40.000 b=60.000 c=90.000") != std::string::npos);
|
||||
CHECK(refusal.find("a=45.000 b=60.000 c=90.000") != std::string::npos);
|
||||
CHECK(refusal.find("-C") != std::string::npos);
|
||||
CHECK(!ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", doubled)}, false, false, settings).empty());
|
||||
// A cell given on the command line is what they are merged in.
|
||||
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), Still("b", other)}, true, false, settings).empty());
|
||||
// A file without a cell, unless one is given.
|
||||
ScaleInput no_cell{"b", false, std::nullopt, 19};
|
||||
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), no_cell}, false, false, settings).find("no unit cell")
|
||||
!= std::string::npos);
|
||||
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc), no_cell}, true, false, settings).empty());
|
||||
}
|
||||
|
||||
TEST_CASE("ScaleInputsRefusal_SpaceGroupsDisagree", "[RugnuxScale]") {
|
||||
const IndexingSettings settings;
|
||||
const auto refusal = ScaleInputsRefusal({Still("a", orthorhombic_uc, 19), Still("b", orthorhombic_uc, 16)},
|
||||
false, false, settings);
|
||||
CHECK(refusal.find("space group 19") != std::string::npos);
|
||||
CHECK(refusal.find("-S") != std::string::npos);
|
||||
CHECK(ScaleInputsRefusal({Still("a", orthorhombic_uc, 19), Still("b", orthorhombic_uc, 16)}, false, true,
|
||||
settings).empty());
|
||||
}
|
||||
|
||||
TEST_CASE("ScaleInputsRefusal_Rotation", "[RugnuxScale]") {
|
||||
const IndexingSettings settings;
|
||||
const ScaleInput sweep{"r", true, orthorhombic_uc, 19};
|
||||
// Stills and a rotation sweep: refused, whatever is given.
|
||||
const auto mixed = ScaleInputsRefusal({Still("a", orthorhombic_uc), sweep}, true, true, settings);
|
||||
CHECK(mixed.find("r is a rotation sweep and a is stills") != std::string::npos);
|
||||
// Several sweeps: refused, pointing to --mode mx.
|
||||
const auto sweeps = ScaleInputsRefusal({sweep, sweep}, false, false, settings);
|
||||
CHECK(sweeps.find("--mode mx") != std::string::npos);
|
||||
}
|
||||
|
||||
TEST_CASE("AppendScaleInput_FramesStayDistinct", "[RugnuxScale]") {
|
||||
std::vector<IntegrationOutcome> all(3);
|
||||
all[1].reflections.push_back(Reflection{.image_number = 1.0f});
|
||||
std::vector<IntegrationOutcome> more(2);
|
||||
more[0].reflections.push_back(Reflection{.image_number = 0.0f});
|
||||
more[1].reflections.push_back(Reflection{.image_number = 1.0f});
|
||||
AppendScaleInput(all, std::move(more));
|
||||
REQUIRE(all.size() == 5);
|
||||
CHECK(all[1].reflections[0].image_number == 1.0f);
|
||||
CHECK(all[3].reflections[0].image_number == 3.0f);
|
||||
CHECK(all[4].reflections[0].image_number == 4.0f);
|
||||
}
|
||||
|
||||
TEST_CASE("ScaleInputs_Report", "[RugnuxScale]") {
|
||||
DiffractionExperiment x(DetJF(1));
|
||||
ProcessResult result;
|
||||
result.images_processed = 300;
|
||||
result.scale_inputs = {{"run1_process.h5", 100, 90, 85}, {"run2_process.h5", 200, 150, 140}};
|
||||
const std::string text = RenderResultReport("p", "run1_process.h5", x, result);
|
||||
CHECK(text.find("\nINPUT_01= 100 90 85 5 run1_process.h5\n") != std::string::npos);
|
||||
CHECK(text.find("\nINPUT_02= 200 150 140 10 run2_process.h5\n") != std::string::npos);
|
||||
// One file: no rows.
|
||||
result.scale_inputs.clear();
|
||||
CHECK(RenderResultReport("p", "in.h5", x, result).find("INPUT_01") == std::string::npos);
|
||||
}
|
||||
Reference in new issue
Block a user