mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases 71 Wiki Activity

rugnux: export the integrated observations as an unmerged MTZ

Browse Source 
Adds --export-unmerged <file.mtz>, in --mode mx and --mode scale alike.
rugnux could only write merged reflections, which closed it off from the
programs that consume unmerged observations - aimless, pointless, careless
and iotbx.merging_statistics all read this file.

The column layout follows what pointless itself writes (H K L M/ISYM BATCH
I SIGI FRACTIONCALC XDET YDET ROT LP FLAG), with DELPHI, ZETA, BGMEAN and
BGVAR added because we have them and a scale model can use them.

Three details decide whether the file is usable:

Lorentz-polarization is applied and recorded in LP. It is per-observation
geometry spanning a factor of ~120 across a sweep and no reading program
can recover it; merging with no scale model at all gives R_meas 0.372 /
0.346 / 0.324 in the low shells without it against 0.183 / 0.218 / 0.257
with it. The partiality is not divided out - it stays in FRACTIONCALC -
and neither is the per-image scale, since these programs fit their own.

The observations are partials and are flagged as such, packed as
256*M + ISYM with LDTYPE=1. Without the flag pointless reads every part as
a whole observation and mis-assigns the symmetry; with it, a tetragonal
case assembles 8.29M parts into 1.15M observations and comes back as its
own space group at confidence 0.92.

The scan axis is written as the negation of the stored goniometer axis,
which is the Cambridge convention: against pointless's own orientation
matrix that agrees to 0.6 degrees, where the axis as stored disagrees by 40.

Merging the exported file with aimless reproduces rugnux's own merge to
CC 0.9997 on a tetragonal case and 0.9978 on a cubic one, and the
anomalous signal survives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P
This commit is contained in:
leonarski_f
2026-08-25 15:21:57 +02:00
co-authored by Claude Opus 5
parent 61c603b274
commit 5248b1bc00
6 changed files with 214 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
image_analysis/WriteReflections.cpp
+163
View File
@@ -11,6 +11,7 @@
#include <cmath>
#include <cstdio>
#include <map>
#include <set>
#include <tuple>
#include <fstream>
#include <iomanip>
@@ -537,6 +538,168 @@ void WriteShelxHklReflections(const std::vector<MergedReflection> &reflections,
out.close();
}
void WriteUnmergedMtzReflections(const std::vector<IntegrationOutcome> &outcomes,
const UnitCell &unitCell,
const DiffractionExperiment &experiment,
const std::string &filename) {
gemmi::Mtz mtz;
mtz.spacegroup = gemmi::find_spacegroup_by_number(experiment.GetSpaceGroupNumber().value_or(1));
mtz.set_cell_for_all(unitCell);
mtz.title = "Rugnux unmerged observations";
mtz.history.push_back("From Rugnux " + jfjoch_version() + ", data reduction");
mtz.add_base(); // the HKL_base dataset and the H K L columns
gemmi::Mtz::Dataset &ds = mtz.add_dataset("unmerged");
ds.crystal_name = experiment.GetSampleName();
ds.wavelength = experiment.GetWavelength_A();
// Every data column belongs to HKL_base and only the batches carry a dataset id, as in a
// POINTLESS file. Down to FLAG this is POINTLESS's own column set; the four after it are what
// rugnux measures beyond it - the offset of the reflection from the centre of its rocking curve,
// the Lorentz geometry of that curve, and the background that was subtracted.
mtz.add_column("M/ISYM", 'Y', 0, -1, false);
mtz.add_column("BATCH", 'B', 0, -1, false);
mtz.add_column("I", 'J', 0, -1, false);
mtz.add_column("SIGI", 'Q', 0, -1, false);
mtz.add_column("FRACTIONCALC", 'R', 0, -1, false);
mtz.add_column("XDET", 'R', 0, -1, false);
mtz.add_column("YDET", 'R', 0, -1, false);
mtz.add_column("ROT", 'R', 0, -1, false);
mtz.add_column("LP", 'R', 0, -1, false);
mtz.add_column("FLAG", 'I', 0, -1, false);
mtz.add_column("DELPHI", 'R', 0, -1, false);
mtz.add_column("ZETA", 'R', 0, -1, false);
mtz.add_column("BGMEAN", 'R', 0, -1, false);
mtz.add_column("BGVAR", 'R', 0, -1, false);
const auto gon = experiment.GetGoniometer();
// A scanning goniometer means rugnux integrated a rocking curve image by image, so every
// observation here is one PART of a reflection; on a still each is a whole measurement. The
// difference has to be declared in two places, because that is where POINTLESS and AIMLESS look
// for it: the batch header's data type, and the M flag that M/ISYM packs as 256*M + ISYM. Told
// neither, they take each partial for a whole reflection and neither the symmetry determination
// nor the scaling survives it.
const bool partials = gon && gon->IsScanning();
const float wedge_deg = gon ? gon->GetWedge_deg() : 0.0f;
const auto phi_start_deg = [&](float image_number) {
return gon ? gon->GetAngle_deg(image_number) : 0.0f;
};
// H K L are the ASU indices and M/ISYM says which symmetry operation (and which Friedel hand) got
// them there, so the index the reflection was actually measured at is recoverable - that is the
// crystal-frame information careless scales on, and what makes the file unmerged rather than a
// merge waiting to happen.
// I and SIGI are the integrated intensity with the Lorentz-polarization factor applied and
// nothing else, which is what IOBS means in every unmerged format (LP records the factor, so the
// raw counts are I/LP). LP is geometry, not a scale, and a program that reads this file has no
// way to recover it. The partiality is NOT divided out - that is a scale, FRACTIONCALC carries
// it, and every program this file is for wants to handle it its own way.
gemmi::UnmergedHklMover hkl_mover(mtz.spacegroup);
std::set<int> batch_numbers;
for (const auto &outcome : outcomes) {
for (const auto &r : outcome.reflections) {
std::array<int, 3> hkl{r.h, r.k, r.l};
const int isym = hkl_mover.move_to_asu(hkl);
const int batch = 1 + static_cast<int>(std::lround(r.image_number));
batch_numbers.insert(batch);
mtz.data.push_back(static_cast<float>(hkl[0]));
mtz.data.push_back(static_cast<float>(hkl[1]));
mtz.data.push_back(static_cast<float>(hkl[2]));
mtz.data.push_back(static_cast<float>((partials ? 256 : 0) + isym));
mtz.data.push_back(static_cast<float>(batch));
mtz.data.push_back(r.I * r.rlp);
mtz.data.push_back(r.sigma * r.rlp);
mtz.data.push_back(r.partiality);
// A reflection too weak to have a measured centroid still has a predicted position, and a
// detector coordinate is what the scale models downstream read this column for.
mtz.data.push_back(std::isfinite(r.observed_x) ? r.observed_x : r.predicted_x);
mtz.data.push_back(std::isfinite(r.observed_y) ? r.observed_y : r.predicted_y);
mtz.data.push_back(phi_start_deg(r.image_number) + wedge_deg / 2.0f);
mtz.data.push_back(r.rlp);
mtz.data.push_back(0.0f); // FLAG: nothing here is a rejected observation
mtz.data.push_back(r.delta_phi_deg);
mtz.data.push_back(r.zeta);
mtz.data.push_back(r.bkg);
mtz.data.push_back(r.var_bkg);
}
}
mtz.nreflections = static_cast<int>(mtz.data.size() / mtz.columns.size());
// The batch header's orientation matrix is the crystal at rotation angle zero - each batch's own
// PHISTT is applied on top of it - but the lattice stored with an outcome is the crystal as it
// stood on that image. Turn the first indexed one back by its own angle to get the orientation of
// the sweep, which is the one matrix POINTLESS also writes into every batch.
std::optional<CrystalLattice> lattice_at_zero;
std::optional<float> mosaicity_deg;
for (const auto &outcome : outcomes) {
if (outcome.reflections.empty() || outcome.latt.CalcVolume() <= 1.0f)
continue;
const float mid_deg = phi_start_deg(outcome.reflections.front().image_number) + wedge_deg / 2.0f;
lattice_at_zero = gon ? outcome.latt.Multiply(gon->GetTransformationAngle(mid_deg)) : outcome.latt;
mosaicity_deg = outcome.mosaicity_deg;
break;
}
// The batch header is written in the "Cambridge" frame - z along the principal rotation axis, x
// along the beam - while the jfjoch lab frame has the beam along +z, so the two are related by a
// rotation. These three lab-frame vectors are the Cambridge axes; a lab vector's components in
// that frame are its dot products with them. A still has no rotation axis, and any axis across
// the beam then defines the frame just as consistently.
// The axis is NEGATED: rugnux turns an observation made at angle phi back to phi = 0 by rotating
// it by +phi about the goniometer axis, so the crystal itself turns by -phi about it, while the
// MTZ batch header's scan axis is the one a batch's own increasing PHI turns the crystal about.
const Coord beam = experiment.GetDiffractionGeometry().GetScatteringVector().Normalize();
const Coord z_cam = gon ? -gon->GetAxis().Normalize() : Coord(0, 1, 0);
const Coord y_cam = (z_cam % beam).Normalize();
const Coord x_cam = (y_cam % z_cam).Normalize();
gemmi::Mtz::Batch batch;
batch.title = "Rugnux";
batch.axes.emplace_back("PHI");
batch.set_dataset_id(ds.id);
batch.ints[12] = 1; // ncryst
batch.ints[14] = partials ? 1 : 2; // ldtype: oscillation data (2D spots) / area detector (3D)
batch.ints[15] = 1; // jsaxs: the goniostat scan axis
batch.ints[17] = 1; // ngonax
batch.ints[19] = 1; // ndet
batch.set_cell(unitCell);
if (lattice_at_zero) {
// Orientation matrix U, built from the reciprocal axes and stored column by column in
// Cambridge components, as gemmi's XDS_ASCII converter builds it (gemmi/xds2mtz.hpp).
const Coord ar = lattice_at_zero->Astar().Normalize();
const Coord cr = (ar % lattice_at_zero->Bstar()).Normalize();
const Coord u[3] = {ar, cr % ar, cr};
for (int i = 0; i < 3; i++) {
batch.floats[6 + 3 * i] = u[i] * x_cam;
batch.floats[7 + 3 * i] = u[i] * y_cam;
batch.floats[8 + 3 * i] = u[i] * z_cam;
}
}
batch.floats[21] = mosaicity_deg.value_or(0.0f); // crydat(0), the reflecting range
batch.floats[40] = 1.0f; // scanax = [0, 0, 1]: the rotation axis IS z in the Cambridge frame
batch.floats[47] = wedge_deg;
batch.floats[61] = 1.0f; // e1 = scanax, the only goniostat axis
batch.floats[80] = -1.0f; // idealised source vector, antiparallel to the beam
batch.floats[83] = -(beam * x_cam); // s0, the source vector of the geometry as it really stands
batch.floats[84] = -(beam * y_cam);
batch.floats[85] = -(beam * z_cam);
batch.set_wavelength(experiment.GetWavelength_A());
batch.floats[111] = experiment.GetDetectorDistance_mm();
batch.floats[113] = 1.0f; // detector limits, in pixels
batch.floats[114] = static_cast<float>(experiment.GetXPixelsNum());
batch.floats[115] = 1.0f;
batch.floats[116] = static_cast<float>(experiment.GetYPixelsNum());
for (const int number : batch_numbers) {
batch.number = number;
batch.floats[36] = phi_start_deg(static_cast<float>(number - 1)); // phistt
batch.floats[37] = batch.floats[36] + wedge_deg; // phiend
mtz.batches.push_back(batch);
}
mtz.sort(5); // by H K L M/ISYM BATCH, the order POINTLESS leaves an unmerged file in
mtz.write_to_file(filename);
}
void WriteReflections(const std::vector<MergedReflection> &reflections,
const UnitCell &unitCell,
const DiffractionExperiment &experiment,