mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 84 Wiki Activity

rugnux: the library returns its outputs in memory; the callers write the files

Browse Source 
The rugnux library wrote its results itself: _process.h5 through a
FileWriter, the merged/twin/_mr reflection files, the unmerged MTZs, the
maps and their MTZ, the placed models, _detector.jpg and _plot.txt, and
on a doubled cell it renamed files on disk. It now writes none of them.

- MtzTable (common/MtzTable.h): an MTZ file's header, columns and batch
  headers with no file behind it. It replaces MergedReflectionsMessage
  and MapCoefficientsMessage in the END message, with one CBOR encoding.
  WriteMtz (writer/WriteRugnuxFiles.h) is the one serialiser, used by
  rugnux and jfjoch_writer alike; it refuses a table whose columns differ
  in length, name a missing dataset, carry a malformed batch header or
  name a space group gemmi does not know. MergedMtzTable and
  UnmergedMtzTable build the tables the old writers built.
- ProcessResult::outputs (RugnuxOutputs) holds, for the output pass only:
  the answer's merge (MTZ table plus what the .cif/.hkl are written from),
  the twin hypothesis' and the model's choice's merges, the unmerged and
  unmerged-partials tables, the maps (coefficients always; the grids only
  with ProcessConfig::keep_map_grids), the placed models, the detector
  JPEG and the per-image plot table, and the start message of the
  no-lattice _process.h5. Map RMS is taken from the grid in memory, as the
  CCP4 header stated it, whether or not the map is kept.
- The per-image analysis goes to a caller-supplied RugnuxImageSink;
  rugnux and the viewer give a FileWriter-based ProcessFileSink. The
  geometry pre-pass no longer makes a _01_process.h5.
- The doubled cell keeps both output sets: the answer's in outputs, the
  other's in DoubledCell::other_outputs with its suffix; the report names
  files by suffix after the caller's prefix.
- The output prefix, write_process_h5 and the observation dump path are
  gone from ProcessConfig; --dump-observations is a callback.
- What used to be gated on having files to write (model validation and
  MR, the twin-hypothesis merges, the finalist ledger, the scaled fulls,
  the unmerged tables) is gated on the output pass instead, so a caller
  with no prefix gets the same results.
- RugnuxOutputFiles (new library JFJochRugnuxOutput, linked by rugnux,
  the viewer and the tests) writes every file from the result with the
  names and formats rugnux has always used. The viewer writes its job's
  files the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-17 16:51:20 +02:00
1 parent 6efc42360a
commit 5f731db81d
50 files changed
+1163 -965
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/ModelValidationTest.cpp
+15 -26
View File
@@ -292,7 +292,7 @@ TEST_CASE("ModelValidation_UnusableModelGivesAReason", "[ModelValidation]") {
no_cell.structure.models.emplace_back(1);
const auto result = ValidateAgainstModel({}, UnitCell{.a = 40, .b = 50, .c = 60,
.alpha = 90, .beta = 90, .gamma = 90},
no_cell, "", logger);
no_cell, logger);
CHECK_FALSE(result.ok);
CHECK_FALSE(result.failure_reason.empty());
CHECK_THAT(result.failure_reason, Catch::Matchers::ContainsSubstring("no_cell.pdb"));
@@ -500,7 +500,6 @@ TEST_CASE("ModelValidation_FindsTheDatasDescriptionOfTheLattice", "[ModelValidat
}
const UnitCell data_cell{.a = 30, .b = 34, .c = 38, .alpha = 90, .beta = 90, .gamma = 90};
const gemmi::SpaceGroup *data_sg = gemmi::find_spacegroup_by_name("P 21 21 21");
const std::string prefix = (std::filesystem::temp_directory_path() / "cob_test").string();
// The same structure as a deposition on permuted axes, a' = b, b' = c, c' = a. The cell is
// orthogonal, so the Cartesian coordinates permute with the axes.
@@ -523,9 +522,13 @@ TEST_CASE("ModelValidation_FindsTheDatasDescriptionOfTheLattice", "[ModelValidat
}
const auto other_setting = WriteTemp("cob_model_test.pdb", permuted.c_str());
const auto moved = ValidateAgainstModel(obs, data_cell, ReadModel(other_setting), prefix, logger, data_sg,
const auto moved = ValidateAgainstModel(obs, data_cell, ReadModel(other_setting), logger, data_sg,
/*probe_indexing_ambiguity=*/false, 1, 1.0);
REQUIRE(moved.ok);
// The maps come back in memory: their coefficients always, the maps themselves only when asked for.
REQUIRE(moved.maps);
CHECK(moved.maps->coefficients.columns.size() == 12);
CHECK_FALSE(moved.maps->map_2fofc);
logger.Info("Change of basis test: took {} out of {}, R-work {:.4f}",
moved.change_of_basis_op.triplet(), moved.setting_as_read, moved.r_work);
CHECK_FALSE(moved.change_of_basis_op == gemmi::Op::identity());
@@ -534,15 +537,13 @@ TEST_CASE("ModelValidation_FindsTheDatasDescriptionOfTheLattice", "[ModelValidat
// The isomorphous case, which is what nearly every run is: the model already describes the
// lattice the way the data do, the only operators that map its cell are its own symmetry, and
// nothing is scored or moved.
const auto same = ValidateAgainstModel(obs, data_cell, ReadModel(data_model), prefix, logger, data_sg,
const auto same = ValidateAgainstModel(obs, data_cell, ReadModel(data_model), logger, data_sg,
false, 1, 1.0);
REQUIRE(same.ok);
CHECK(same.change_of_basis_op == gemmi::Op::identity());
CHECK(same.setting_as_read.empty());
CHECK(same.r_work < 0.15);
for (const char *suffix : {"_2fofc.ccp4", "_fofc.ccp4", "_anom.ccp4", "_maps.mtz"})
std::filesystem::remove(prefix + suffix);
std::filesystem::remove(data_model);
std::filesystem::remove(other_setting);
}
@@ -566,12 +567,11 @@ TEST_CASE("ModelValidation_ReindexesTheDataIntoTheModelsIndexing", "[ModelValida
const gemmi::SpaceGroup *p4 = gemmi::find_spacegroup_by_name("P 4");
const auto laws = ReindexAmbiguityOperators(cell, *p4);
REQUIRE(laws.size() == 1);
const std::string prefix = (std::filesystem::temp_directory_path() / "reidx_test").string();
// The data as a run that picked the other indexing would have merged them.
const auto obs = ReindexReflections(own, laws.front());
const auto to_model = ValidateAgainstModel(obs, cell, ReadModel(model), prefix, logger, p4,
const auto to_model = ValidateAgainstModel(obs, cell, ReadModel(model), logger, p4,
/*probe_indexing_ambiguity=*/true, 1, 1.0);
REQUIRE(to_model.ok);
CHECK(to_model.change_of_basis_op == gemmi::Op::identity());
@@ -580,7 +580,7 @@ TEST_CASE("ModelValidation_ReindexesTheDataIntoTheModelsIndexing", "[ModelValida
CHECK(to_model.r_work < 0.15);
// A reference already fixed the data's indexing: the data stay, and the model is moved.
const auto fixed = ValidateAgainstModel(obs, cell, ReadModel(model), prefix, logger, p4, false, 1, 1.0);
const auto fixed = ValidateAgainstModel(obs, cell, ReadModel(model), logger, p4, false, 1, 1.0);
REQUIRE(fixed.ok);
CHECK_FALSE(fixed.change_of_basis_op == gemmi::Op::identity());
CHECK(fixed.indexing_op == gemmi::Op::identity());
@@ -604,7 +604,7 @@ TEST_CASE("ModelValidation_ReindexesTheDataIntoTheModelsIndexing", "[ModelValida
r.F = std::sqrt(std::max(0.0f, r.I));
}
}
const auto twin = ValidateAgainstModel(twinned, cell, ReadModel(model), prefix, logger, p4, true, 1, 1.0);
const auto twin = ValidateAgainstModel(twinned, cell, ReadModel(model), logger, p4, true, 1, 1.0);
REQUIRE(twin.ok);
CHECK_FALSE(twin.indexing_decided);
CHECK(twin.indexing_op == gemmi::Op::identity());
@@ -612,7 +612,7 @@ TEST_CASE("ModelValidation_ReindexesTheDataIntoTheModelsIndexing", "[ModelValida
// Data already in the model's indexing: nothing moves. The null is run all the same, as on every
// validation that judges its model (ModelValidationSchedule::fit_test), and the model fits.
const auto same = ValidateAgainstModel(own, cell, ReadModel(model), prefix, logger, p4, true, 1, 1.0);
const auto same = ValidateAgainstModel(own, cell, ReadModel(model), logger, p4, true, 1, 1.0);
REQUIRE(same.ok);
CHECK(same.change_of_basis_op == gemmi::Op::identity());
CHECK(same.indexing_op == gemmi::Op::identity());
@@ -620,8 +620,6 @@ TEST_CASE("ModelValidation_ReindexesTheDataIntoTheModelsIndexing", "[ModelValida
CHECK(same.model_fits);
CHECK(same.r_work < 0.15);
for (const char *suffix : {"_2fofc.ccp4", "_fofc.ccp4", "_anom.ccp4", "_maps.mtz"})
std::filesystem::remove(prefix + suffix);
std::filesystem::remove(model);
}
@@ -778,11 +776,10 @@ TEST_CASE("ModelValidation_CCModelFollowsTheSignalByShell", "[ModelValidation]")
obs[i].rfree_flag = (i % 20) == 0;
}
const std::string prefix = (std::filesystem::temp_directory_path() / "cc_model_test").string();
const auto result =
ValidateAgainstModel(obs, UnitCell{.a = 30, .b = 34, .c = 38,
.alpha = 90, .beta = 90, .gamma = 90},
ReadModel(path), prefix, logger, gemmi::find_spacegroup_by_name("P 21 21 21"),
ReadModel(path), logger, gemmi::find_spacegroup_by_name("P 21 21 21"),
/*probe_indexing_ambiguity=*/false, 1, 1.0, shells);
REQUIRE(result.ok);
REQUIRE(result.cc_model_shells.size() == shells.size());
@@ -810,13 +807,11 @@ TEST_CASE("ModelValidation_CCModelFollowsTheSignalByShell", "[ModelValidation]")
const auto no_shells =
ValidateAgainstModel(obs, UnitCell{.a = 30, .b = 34, .c = 38,
.alpha = 90, .beta = 90, .gamma = 90},
ReadModel(path), prefix, logger, gemmi::find_spacegroup_by_name("P 21 21 21"),
ReadModel(path), logger, gemmi::find_spacegroup_by_name("P 21 21 21"),
false, 1, 1.0);
CHECK(no_shells.ok);
CHECK(no_shells.cc_model_shells.empty());
for (const char *suffix : {"_2fofc.ccp4", "_fofc.ccp4", "_anom.ccp4", "_maps.mtz"})
std::filesystem::remove(prefix + suffix);
std::filesystem::remove(path);
}
@@ -839,15 +834,12 @@ TEST_CASE("ModelValidation_WrongModelInAGroupOffTheDataMetric", "[ModelValidatio
obs[i].F = std::sqrt(std::max(0.0f, obs[i].I));
obs[i].rfree_flag = (i % 20) == 0;
}
const std::string prefix = (std::filesystem::temp_directory_path() / "off_metric_test").string();
const auto result =
ValidateAgainstModel(obs, UnitCell{.a = 30, .b = 34, .c = 38, .alpha = 90, .beta = 90, .gamma = 90},
ReadModel(wrong_path), prefix, logger, gemmi::find_spacegroup_by_name("P 21 21 21"));
ReadModel(wrong_path), logger, gemmi::find_spacegroup_by_name("P 21 21 21"));
CHECK(result.ok);
CHECK(!result.adopted_model_enantiomorph);
for (const char *suffix : {"_2fofc.ccp4", "_fofc.ccp4", "_anom.ccp4", "_maps.mtz"})
std::filesystem::remove(prefix + suffix);
std::filesystem::remove(data_path);
std::filesystem::remove(wrong_path);
}
@@ -872,10 +864,9 @@ TEST_CASE("ModelValidation_CCModelUnrelatedModelConfirmsNothing", "[ModelValidat
obs[i].rfree_flag = (i % 20) == 0;
}
const std::vector<float> shells{4.0f, 3.0f, 2.5f, 2.2f, 2.0f};
const std::string prefix = (std::filesystem::temp_directory_path() / "cc_null_test").string();
const auto result =
ValidateAgainstModel(obs, UnitCell{.a = 30, .b = 34, .c = 38, .alpha = 90, .beta = 90, .gamma = 90},
ReadModel(wrong_path), prefix, logger, gemmi::find_spacegroup_by_name("P 21 21 21"),
ReadModel(wrong_path), logger, gemmi::find_spacegroup_by_name("P 21 21 21"),
/*probe_indexing_ambiguity=*/false, 1, 1.0, shells);
REQUIRE(result.ok);
REQUIRE(result.cc_model_shells.size() == shells.size());
@@ -890,8 +881,6 @@ TEST_CASE("ModelValidation_CCModelUnrelatedModelConfirmsNothing", "[ModelValidat
// The trap is there: on independent reflections some shell would have read as significant.
CHECK(fisher_max > 3.0);
for (const char *suffix : {"_2fofc.ccp4", "_fofc.ccp4", "_anom.ccp4", "_maps.mtz"})
std::filesystem::remove(prefix + suffix);
std::filesystem::remove(data_path);
std::filesystem::remove(wrong_path);
}