mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 6 Packages Projects Releases 71 Wiki Activity

_process.h5: record the change of basis, and the space group

Browse Source 
A _process.h5 was internally inconsistent whenever the space group re-seated
the lattice. The per-image reflections and lattices go to file as each image
is processed, but the conventional setting is only chosen afterwards, so the
file kept pre-reindex indices beside a post-reindex cell. Measured on an
affected dataset, max|d_file - d(hkl, stored cell)| was 18.6 A. --mode scale
then compared each frame's lattice against the stored cell, found none of
1800 acceptable, rejected every observation and died in the merge with
"resolution calculation failed".

/entry/MX/reindexMatrix now carries M with hkl_cell = M . hkl_written, and
the reader applies it, so everything it hands out is in the setting of
/entry/sample/unit_cell. Absent means the identity, so a file written before
this reads exactly as before. On the affected dataset M comes out
[[1,1,0],[0,1,1],[1,0,1]], det 2 - the primitive-to-body-centred basis its
volume ratio implied - and the same measure falls from 18.6 A to 1.8e-5.

Writing the reflections in the final setting instead was rejected: the
per-image writer is shared with the broker, which streams and cannot buffer a
run; and h,k,l, predicted_x/y and the per-image lattice are one consistent
statement about one image, which retro-editing the indices would silently
break.

Two things turned up while fixing it. There are three re-seat sites, not one
- the space-group search's own centred-lattice test re-seats too, and logged
nothing - so the matrix composes over all of them. And the space group itself
was never written: it was set only on the arm that searches, while a two-pass
rotation run reuses pass 1's group and takes the other arm, so the canonical
file carried a cell but no group and --mode scale merged in P1.

--mode scale now reproduces --mode mx on the affected dataset: same space
group and cell, 39329 unique reflections both, 100% of reflections common,
CC 0.99985, sum|dI|/sum|I| = 0.0081. The residual is three reflections in
11.09 M crossing an ice-band edge, because mx carries the integrator's d and
scale recomputes it from the cell. An unaffected dataset is byte-identical in
.mtz, .hkl and .cif.

An older affected file still cannot merge - M is not recoverable from it -
but now says so in 51 s, naming both cells and the -S/-C override to use,
instead of failing inside the merge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P
This commit is contained in:
leonarski_f
2026-08-25 21:21:55 +02:00
co-authored by Claude Opus 5
parent 8fa1ab7b05
commit db908c6ec1
10 changed files with 266 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
reader/HDF5MetadataSource.cpp
+44 -9
View File
@@ -132,9 +132,22 @@ std::string dataset_name(const std::string &path) {
return file;
}
// Per-image reflections and lattices are written in the setting the images were INDEXED in; the
// unit cell, the run lattice and the space group beside them are in the setting the merge settled
// on, which the space-group search can re-seat to. /entry/MX/reindexMatrix is the integral change of
// basis between the two, so applying it here is what makes the file read as one consistent dataset.
// No matrix means the two settings are the same one.
CrystalLattice ApplyReindex(const CrystalLattice &latt, const std::optional<std::array<int32_t, 9>> &m) {
if (!m)
return latt;
const auto &v = *m;
return latt.Multiply(gemmi::Mat33(v[0], v[1], v[2], v[3], v[4], v[5], v[6], v[7], v[8]));
}
bool ReadReflectionsFromGroup(HDF5Object &file,
const std::string &image_group_name,
std::vector<Reflection> &reflections) {
std::vector<Reflection> &reflections,
const std::optional<std::array<int32_t, 9>> &reindex) {
if (!file.Exists("/entry/reflections") || !file.Exists(image_group_name))
return false;
@@ -165,6 +178,15 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
throw JFJochException(JFJochExceptionCategory::HDF5, "Wrong size of reflections dataset");
for (size_t i = 0; i < h.size(); i++) {
int32_t hh = h.at(i), kk = k.at(i), ll = l.at(i);
if (reindex) {
const auto &m = *reindex;
const int32_t h0 = hh, k0 = kk, l0 = ll;
hh = m[0] * h0 + m[1] * k0 + m[2] * l0;
kk = m[3] * h0 + m[4] * k0 + m[5] * l0;
ll = m[6] * h0 + m[7] * k0 + m[8] * l0;
}
float lp_val = 0.0;
if (lp.size() > i && lp[i] != 0.0f)
lp_val = 1.0f / lp[i];
@@ -201,9 +223,9 @@ bool ReadReflectionsFromGroup(HDF5Object &file,
}
Reflection r{
.h = h.at(i),
.k = k.at(i),
.l = l.at(i),
.h = hh,
.k = kk,
.l = ll,
.image_number = image_number.at(i),
.delta_phi_deg = delta_phi_val,
.predicted_x = predicted_x.at(i),
@@ -687,6 +709,13 @@ HDF5MetadataSource::OpenResult HDF5MetadataSource::Open(const std::string &filen
.gamma = tmp[5]
});
dataset->experiment.SpaceGroupNumber(master_file->GetOptInt("/entry/sample/space_group_number"));
// The setting the cell and space group just read are in, relative to the setting the per-image
// reflections and lattices were written in. Absent on every file written before the offline
// analysis started recording it, and on every run that never re-seated its lattice - both mean
// the identity, and both are read as such.
if (const auto m = master_file->ReadOptVector<int32_t>("/entry/MX/reindexMatrix"); m.size() == 9)
dataset->reindex_matrix = std::array<int32_t, 9>{m[0], m[1], m[2], m[3], m[4],
m[5], m[6], m[7], m[8]};
dataset->experiment.SampleName(master_file->GetString("/entry/sample/name"));
@@ -1070,7 +1099,7 @@ void HDF5MetadataSource::FillPerImage(DataMessage &message, int64_t requested_im
);
if (tmp.size() == 9)
message.indexing_lattice = CrystalLattice(tmp);
message.indexing_lattice = ApplyReindex(CrystalLattice(tmp), dataset->reindex_matrix);
std::optional<std::string> lattice;
if (master_file->Exists("/entry/MX/bravaisLattice"))
@@ -1102,8 +1131,10 @@ void HDF5MetadataSource::FillPerImage(DataMessage &message, int64_t requested_im
const std::string master_reflection_group_name = fmt::format("/entry/reflections/image_{:06d}", image_number);
const std::string source_reflection_group_name = fmt::format("/entry/reflections/image_{:06d}", image_id);
if (!ReadReflectionsFromGroup(*master_file, master_reflection_group_name, message.reflections))
ReadReflectionsFromGroup(*source_file, source_reflection_group_name, message.reflections);
if (!ReadReflectionsFromGroup(*master_file, master_reflection_group_name, message.reflections,
dataset->reindex_matrix))
ReadReflectionsFromGroup(*source_file, source_reflection_group_name, message.reflections,
dataset->reindex_matrix);
if (!message.reflections.empty()) {
CalcISigma(message);
CalcWilsonBFactor(message, !message.b_factor.has_value());
@@ -1214,6 +1245,10 @@ std::vector<IntegrationOutcome> HDF5MetadataSource::ReadReflections(size_t start
// reflections lazily from the source data files instead.
const bool master_reflections_authoritative = master_file->Exists("/entry/reflections");
// Everything below comes out in the setting of the dataset's unit cell and space group, not in the
// setting it was written in (see ApplyReindex).
const auto &reindex = dataset_->reindex_matrix;
for (size_t img = start_image; img <= end_image_val; img++) {
IntegrationOutcome outcome;
@@ -1235,7 +1270,7 @@ std::vector<IntegrationOutcome> HDF5MetadataSource::ReadReflections(size_t start
}
// ── reflections ──────────────────────────────────────────────────────
ReadReflectionsFromGroup(*meta_file, refl_group, outcome.reflections);
ReadReflectionsFromGroup(*meta_file, refl_group, outcome.reflections, reindex);
// ── per-image mosaicity ───────────────────────────────────────────────
if (meta_file->Exists("/entry/MX/mosaicity")) {
@@ -1252,7 +1287,7 @@ std::vector<IntegrationOutcome> HDF5MetadataSource::ReadReflections(size_t start
auto lattice_vec = meta_file->ReadOptVector<float>(
"/entry/MX/latticeIndexed", {meta_image_id, 0}, {1, 9});
if (lattice_vec.size() == 9)
outcome.latt = CrystalLattice(lattice_vec);
outcome.latt = ApplyReindex(CrystalLattice(lattice_vec), reindex);
} catch (...) {
}
}