mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 84 Wiki Activity

rugnux: the library returns its outputs in memory; the callers write the files

Browse Source 
The rugnux library wrote its results itself: _process.h5 through a
FileWriter, the merged/twin/_mr reflection files, the unmerged MTZs, the
maps and their MTZ, the placed models, _detector.jpg and _plot.txt, and
on a doubled cell it renamed files on disk. It now writes none of them.

- MtzTable (common/MtzTable.h): an MTZ file's header, columns and batch
  headers with no file behind it. It replaces MergedReflectionsMessage
  and MapCoefficientsMessage in the END message, with one CBOR encoding.
  WriteMtz (writer/WriteRugnuxFiles.h) is the one serialiser, used by
  rugnux and jfjoch_writer alike; it refuses a table whose columns differ
  in length, name a missing dataset, carry a malformed batch header or
  name a space group gemmi does not know. MergedMtzTable and
  UnmergedMtzTable build the tables the old writers built.
- ProcessResult::outputs (RugnuxOutputs) holds, for the output pass only:
  the answer's merge (MTZ table plus what the .cif/.hkl are written from),
  the twin hypothesis' and the model's choice's merges, the unmerged and
  unmerged-partials tables, the maps (coefficients always; the grids only
  with ProcessConfig::keep_map_grids), the placed models, the detector
  JPEG and the per-image plot table, and the start message of the
  no-lattice _process.h5. Map RMS is taken from the grid in memory, as the
  CCP4 header stated it, whether or not the map is kept.
- The per-image analysis goes to a caller-supplied RugnuxImageSink;
  rugnux and the viewer give a FileWriter-based ProcessFileSink. The
  geometry pre-pass no longer makes a _01_process.h5.
- The doubled cell keeps both output sets: the answer's in outputs, the
  other's in DoubledCell::other_outputs with its suffix; the report names
  files by suffix after the caller's prefix.
- The output prefix, write_process_h5 and the observation dump path are
  gone from ProcessConfig; --dump-observations is a callback.
- What used to be gated on having files to write (model validation and
  MR, the twin-hypothesis merges, the finalist ledger, the scaled fulls,
  the unmerged tables) is gated on the output pass instead, so a caller
  with no prefix gets the same results.
- RugnuxOutputFiles (new library JFJochRugnuxOutput, linked by rugnux,
  the viewer and the tests) writes every file from the result with the
  names and formats rugnux has always used. The viewer writes its job's
  files the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-17 16:51:20 +02:00
1 parent 6efc42360a
commit 5f731db81d
50 files changed
+1163 -965
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

image_analysis/structure_refinement/ModelValidation.cpp
+71 -92
View File
@@ -27,8 +27,6 @@
#include <gemmi/fourier.hpp> // get_size_for_hkl, get_f_phi_on_grid (the maps)
#include <gemmi/solmask.hpp> // SolventMasker
#include <gemmi/scaling.hpp> // Scaling (bulk solvent + anisotropic B)
#include <gemmi/ccp4.hpp> // Ccp4 map I/O
#include <gemmi/mtz.hpp> // Mtz (map-coefficient output)
#include <gemmi/stats.hpp> // calculate_data_statistics
#include "../../common/CorrelationCoefficient.h"
@@ -79,22 +77,9 @@ gemmi::Grid<float> map_from_coefficients(gemmi::AsuData<std::complex<float>> &co
return MapFromFPhi(gemmi::get_f_phi_on_grid<float>(coef, size, true));
}
// A file this run may already have written - the validation in the model's setting writes over the
// first one's maps - is removed before it is written again, not truncated: XFS (and ext4) flush a file
// truncated and rewritten when it is closed, and three 150 MB maps forced out to a disk that way cost
// seconds where a fresh file costs nothing.
void remove_before_rewriting(const std::string &path) {
std::remove(path.c_str());
}
// Write a map as CCP4; return its RMS (the sigma the map is read in).
double write_ccp4(const gemmi::Grid<float> &map, const std::string &path) {
gemmi::Ccp4<float> ccp4;
ccp4.grid = map;
ccp4.update_ccp4_header(2);
remove_before_rewriting(path);
ccp4.write_ccp4_map(path);
return ccp4.hstats.rms;
// The RMS of a map, the sigma it is read in: the one its CCP4 header states (Ccp4::update_ccp4_header).
double map_rms(const gemmi::Grid<float> &map) {
return gemmi::calculate_data_statistics(map.data).rms;
}
// Cubic, not the default linear, for reading a map at a point. The maps are sampled every d_min/3,
@@ -536,7 +521,6 @@ namespace {
ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
const UnitCell &cell,
const RugnuxModel &model,
const std::string &output_prefix,
Logger &logger,
const gemmi::SpaceGroup *data_space_group,
bool probe_indexing_ambiguity,
@@ -1711,23 +1695,27 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
mapfofc.v.push_back({terms[i].hkl, delfwt[i] * ph});
}
// A validation started before the one it follows had decided writes nothing until that one has,
// and nothing at all where it decided otherwise.
if (schedule.write_gate.valid() && !schedule.write_gate.get()) {
result.failure_reason = "superseded before its files were written";
// A validation started before the one it follows had decided makes no maps until that one has,
// and none at all where it decided otherwise.
if (schedule.maps_gate.valid() && !schedule.maps_gate.get()) {
result.failure_reason = "superseded before its maps were made";
return result;
}
// Where the validation in the model's setting, started beside this one, is going to be kept, it
// writes the maps and the map coefficients on the axes the files are written on, and this one's would
// be overwritten unread: they are not made at all.
// Where the validation in the model's setting, started beside this one, is going to be kept, its
// maps and map coefficients are on the axes the files are written on, and this one's would be
// thrown away unread: they are not made at all.
const bool maps_superseded = schedule.maps_superseded && schedule.maps_superseded(result);
std::shared_ptr<ModelMaps> maps;
if (!maps_superseded) {
// --- write the maps and score the 2mFo-DFc map at atom centres (a real map peaks there) ---
maps = std::make_shared<ModelMaps>();
// --- make the maps and score the 2mFo-DFc map at atom centres (a real map peaks there) ---
// The three maps - this one, the difference map and the anomalous map - share nothing but what
// they read, so the other two are made and written on threads of their own beside this one.
// they read, so the other two are made on threads of their own beside this one. Nothing here
// reads the difference map, so it is made only where it is kept.
std::future<void> fofc_map = std::async(std::launch::async, [&] {
write_ccp4(map_from_coefficients(mapfofc, sf_gpu), output_prefix + "_fofc.ccp4");
if (schedule.keep_map_grids)
maps->map_fofc = map_from_coefficients(mapfofc, sf_gpu);
});
// --- anomalous difference map, where the merge kept the Bijvoet split ---
@@ -1741,13 +1729,15 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
result.indexing_op, fmodel);
result.anomalous_pairs = static_cast<int>(mapanom.v.size());
if (!mapanom.v.empty()) {
const gemmi::Grid<float> grid = map_from_coefficients(mapanom, sf_gpu);
read_anomalous_map(grid, write_ccp4(grid, output_prefix + "_anom.ccp4"), st, result);
gemmi::Grid<float> grid = map_from_coefficients(mapanom, sf_gpu);
read_anomalous_map(grid, map_rms(grid), st, result);
if (schedule.keep_map_grids)
maps->map_anom = std::move(grid);
}
});
const gemmi::Grid<float> grid2fofc = map_from_coefficients(map2fofc, sf_gpu);
const double rms2 = write_ccp4(grid2fofc, output_prefix + "_2fofc.ccp4");
gemmi::Grid<float> grid2fofc = map_from_coefficients(map2fofc, sf_gpu);
const double rms2 = map_rms(grid2fofc);
{
double s = 0; int n = 0;
for (gemmi::Model &m : st.models)
@@ -1756,6 +1746,8 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
for (gemmi::Atom &a : r.atoms) { s += grid2fofc.interpolate_value(a.pos, MAP_INTERPOLATION_ORDER); ++n; }
result.mean_atom_density_sigma = (n > 0 && rms2 > 0) ? (s / n) / rms2 : 0;
}
if (schedule.keep_map_grids)
maps->map_2fofc = std::move(grid2fofc);
fofc_map.get();
anomalous_map.get();
}
@@ -1780,60 +1772,53 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
: "the model does not fit these data");
if (!maps_superseded) {
// --- MTZ of map coefficients so the maps can be re-opened / rebuilt in Coot etc. ---
try {
gemmi::Mtz mtz(true);
// The group the REFLECTIONS end up in, which is the model's only where its hand was adopted -
// AdoptModelFrame decides this the same way a few lines below. An enantiomorphic pair indexes
// identically, so the coefficients are the same numbers either way and only the label moves;
// but the two groups have different screw translations, so a reader that expands symmetry out
// of this file works in the wrong one if the label disagrees with the .mtz beside it.
mtz.spacegroup = result.adopted_model_enantiomorph
? sg
// No data group means the caller merged in P1, and P1 is what the
// reflection files beside this one carry - not the model's group.
: (data_space_group != nullptr ? data_space_group
: gemmi::find_spacegroup_by_number(1));
mtz.set_cell_for_all(ucell);
mtz.add_dataset("model_validation");
mtz.datasets.back().wavelength = wavelength_A;
mtz.add_column("FP", 'F', -1, -1, false);
mtz.add_column("FC", 'F', -1, -1, false);
mtz.add_column("PHIC", 'P', -1, -1, false);
mtz.add_column("FWT", 'F', -1, -1, false);
mtz.add_column("PHWT", 'P', -1, -1, false);
mtz.add_column("DELFWT", 'F', -1, -1, false);
mtz.add_column("PHDELWT", 'P', -1, -1, false);
// The figure of merit the coefficients carry, so the weighting can be read off - and undone -
// from the file rather than having to be taken on trust.
mtz.add_column("FOM", 'W', -1, -1, false);
mtz.add_column("FREE", 'I', -1, -1, false);
std::vector<float> data;
for (size_t i = 0; i < terms.size(); i++) {
// [0, 360), the convention the MTZ 'P' columns carried before sigma_A weighting: std::arg
// returns (-180, 180] and a P column is not supposed to.
const double phi_wrapped = terms[i].phi * 180.0 / PI;
const auto phi_deg = static_cast<float>(phi_wrapped < 0 ? phi_wrapped + 360.0 : phi_wrapped);
data.insert(data.end(), {static_cast<float>(terms[i].hkl[0]), static_cast<float>(terms[i].hkl[1]),
static_cast<float>(terms[i].hkl[2]),
static_cast<float>(terms[i].fo), static_cast<float>(terms[i].fc),
phi_deg,
fwt[i], phi_deg,
delfwt[i], phi_deg,
static_cast<float>(sigma_a.weight[i].m),
terms[i].free ? 0.0f : 1.0f});
}
mtz.nreflections = static_cast<int>(terms.size());
mtz.data = std::move(data);
remove_before_rewriting(output_prefix + "_maps.mtz");
mtz.write_to_file(output_prefix + "_maps.mtz");
} catch (const std::exception &e) {
logger.Warning("Model validation: could not write map MTZ: {}", e.what());
// --- the map coefficients, so the maps can be re-opened / rebuilt in Coot etc. ---
MtzTable &mtz = maps->coefficients;
// The group the REFLECTIONS end up in, which is the model's only where its hand was adopted -
// AdoptModelFrame decides this the same way a few lines below. An enantiomorphic pair indexes
// identically, so the coefficients are the same numbers either way and only the label moves;
// but the two groups have different screw translations, so a reader that expands symmetry out
// of this file works in the wrong one if the label disagrees with the .mtz beside it.
mtz.space_group_name = result.adopted_model_enantiomorph
? sg->xhm()
// No data group means the caller merged in P1, and P1 is what the
// reflection files beside this one carry - not the model's group.
: (data_space_group != nullptr ? data_space_group->xhm()
: gemmi::find_spacegroup_by_number(1)->xhm());
mtz.unit_cell = UnitCell{static_cast<float>(ucell.a), static_cast<float>(ucell.b),
static_cast<float>(ucell.c), static_cast<float>(ucell.alpha),
static_cast<float>(ucell.beta), static_cast<float>(ucell.gamma)};
mtz.datasets = {{"HKL_base", "HKL_base", "HKL_base", 0.0f},
{"model_validation", "model_validation", "model_validation",
static_cast<float>(wavelength_A)}};
// The figure of merit the coefficients carry, so the weighting can be read off - and undone -
// from the file rather than having to be taken on trust.
const std::pair<const char *, char> columns[] = {{"H", 'H'}, {"K", 'H'}, {"L", 'H'}, {"FP", 'F'},
{"FC", 'F'}, {"PHIC", 'P'}, {"FWT", 'F'}, {"PHWT", 'P'},
{"DELFWT", 'F'}, {"PHDELWT", 'P'}, {"FOM", 'W'},
{"FREE", 'I'}};
for (const auto &[label, type] : columns)
mtz.columns.push_back({label, type, type == 'H' ? 0 : 1, {}});
for (size_t i = 0; i < terms.size(); i++) {
// [0, 360), the convention the MTZ 'P' columns carried before sigma_A weighting: std::arg
// returns (-180, 180] and a P column is not supposed to.
const double phi_wrapped = terms[i].phi * 180.0 / PI;
const auto phi_deg = static_cast<float>(phi_wrapped < 0 ? phi_wrapped + 360.0 : phi_wrapped);
const float row[] = {static_cast<float>(terms[i].hkl[0]), static_cast<float>(terms[i].hkl[1]),
static_cast<float>(terms[i].hkl[2]),
static_cast<float>(terms[i].fo), static_cast<float>(terms[i].fc),
phi_deg,
fwt[i], phi_deg,
delfwt[i], phi_deg,
static_cast<float>(sigma_a.weight[i].m),
terms[i].free ? 0.0f : 1.0f};
for (size_t c = 0; c < mtz.columns.size(); c++)
mtz.columns[c].values.push_back(row[c]);
}
}
result.ok = true;
result.maps_prefix = output_prefix;
result.maps = std::move(maps);
// The placed coordinates, for the caller to write out beside the maps once the frame is settled.
result.placed_model = std::make_shared<gemmi::Structure>(st);
logger.Info("Model validation: R-work={:.4f} ({} refl) R-free={:.4f} ({} refl) "
@@ -1862,12 +1847,7 @@ ModelValidationResult Validate(const std::vector<MergedReflection> &merged,
"data were indexed in the wrong hand, and reindexing would hide which",
result.anomalous_deepest_site, result.anomalous_deepest_sigma);
if (maps_superseded)
logger.Info("Model validation: no maps made here - the validation in the model's setting writes them");
else
logger.Info("Model validation: wrote {}_2fofc.ccp4, {}_fofc.ccp4{}, {}_maps.mtz",
output_prefix, output_prefix,
result.anomalous_sites.empty() ? "" : ", " + output_prefix + "_anom.ccp4",
output_prefix);
logger.Info("Model validation: no maps made here - the validation in the model's setting makes them");
if (result.fit_tested && !result.model_fits)
logger.Info("Model validation: the model was rejected, so nothing downstream moved - the "
"reflection files are byte for byte the ones a run with no model would have "
@@ -1917,7 +1897,6 @@ ModelValidationGPUPlan PlanModelValidationGPU(const ModelValidationGPUNeed &need
ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &merged,
const UnitCell &cell,
const RugnuxModel &model,
const std::string &output_prefix,
Logger &logger,
const gemmi::SpaceGroup *data_space_group,
bool probe_indexing_ambiguity,
@@ -1932,7 +1911,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
// the validation reports why it did not finish, and a validation that did not finish decides nothing,
// so the reflection files are those of a run without a model.
try {
return Validate(merged, cell, model, output_prefix, logger, data_space_group,
return Validate(merged, cell, model, logger, data_space_group,
probe_indexing_ambiguity, nthreads, wavelength_A, report_shell_d_min, schedule, twin_law,
true, placed);
} catch (const RigidBodyGPUFailure &e) {
@@ -1945,7 +1924,7 @@ ModelValidationResult ValidateAgainstModel(const std::vector<MergedReflection> &
return failed;
}
#else
return Validate(merged, cell, model, output_prefix, logger, data_space_group, probe_indexing_ambiguity,
return Validate(merged, cell, model, logger, data_space_group, probe_indexing_ambiguity,
nthreads, wavelength_A, report_shell_d_min, schedule, twin_law, false, placed);
#endif
}