The rc178 branch continued after #88 was squash-merged; this is its content fromb9ada7914(the tree #88 squashed) to094eee31a, as one change on main. The individual commits stay on the rc178 branch: - still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes - AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets - DetectorModules: panel look-up by id, plane -> panel, plane bounds and a per-pixel walk; mask size checked - tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout - Dense per-pixel maps through the panel table (panel layout stage 1) - Panel id on spots and the panel table in the start message - Viewer and preview show a panel detector as it lies on the detector plane - Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster - Panel layout stage 2: prediction and Bragg integration through the panel table - XTC reader: frames handed up as stored, streams indexed in parallel - XFEL reader: integer frames stay integer, one gain, placement by panel - EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery" - still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed - float32_to_int16.py: a non-finite pixel becomes -32768, the int16 error marker, not a zero count - CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them - Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table - Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane - still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good - still battery: take the deposited model as mmCIF where the RCSB has no PDB-format file - still battery: dimple gets PDB-format coordinates converted from an mmCIF-only deposition - stills fixed cost: geometry refinement and pre-scan do the same work in less wall time - stills geometry sample and trial spot finding run as many workers as the image pass - Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame - StillsPartialityRefine: the partiality width is the run's, measured on the crystals, not the prediction window's - MergeOnTheFly: capture term and multiplicative outlier band for the stills merge (both opt-in) - Stills given a cell: the written cell is the mean over the frames that are merged - Stills: the radial curvature correction decided per run on the same empty apertures - StillsPartialityRefine: measure the run width only on shells with signal - Stills pre-scan: report the paired z of the radial-correction comparison - module placement: the five concurrent fits each keep every thread - still battery: score R-free against the deposited data through the same dimple protocol - still battery: the R-free target is the lower of the deposition's dimple and published values - Stills: background clip follows the radial curve; signal disk from the lattice spacing - Stills keep an overloaded observation as a lower bound instead of dropping it - Stills: drop the lattice-spacing signal disk; the correction decision back on the plain clip - battery: R-free against the deposition without refinement only, one protocol for rotation and stills - still battery: EuXFEL sets run on a 2,000-image subset for now - stills geometry refinement: fit the detector distance and the cell together - stills geometry refinement: a distance given with --detector-distance is held in the joint fit - Metal integration engine: back to its panel-s5 kernels; Mac stills keep the old clip, no saturation count - Metal integration test: the CPU comparison runs on a rotation experiment - Stills: kept overloads go back to the drop where they are not concentrated at low resolution - stills distance search: an absolute 30-3000 mm scan, a sample found without d-limits, the fit iterated - battery results: XFEL still battery 20261022-1358 on rc1780cf62452c- 12 sets, cold, R_free unrefined - battery results: still summary gains the resolution of our merge and of the deposition - Pre-scan masks a pixel at the saturation limit on most frames, as one holding the error value - Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed - XFEL reader: a per-frame mask is read the way the frames are, its chunk inflated outside the HDF5 lock - stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes - rugnux usage: --beam-center-check describes the absolute 30-3000 mm stills distance scan, not the old 60-160 % grid - stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree - CLAUDE.md: the changelog is written once, just before the merge into main - condensed one-liners of what users get - broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4) - frontend: Mol* viewer of the placed model and its density on the Rugnux live page - writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does - rugnux live maps: one unit cell centred on the model; the viewer opens on all of it - Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer - Mol* viewer: open with the maps drawn within 10 A of the centre of view - Mol* viewer: aaredb's structure-viewer setup, on a black stage - rugnux: the gain of a stills file that states none is read off its single-photon peak where the background resolves photons - Mol* viewer: in line with AareDB's StructureView at its current head (8ece2c2) - ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI) - stills tilt decision: the modules stay where the run placed them - still battery: no distance pins and no per-set tuning flags; gain from the data except where a file's baseline hides it - battery results: rc178 6f6f69 - full still battery and full rotation battery with REFMAC - Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons - BackgroundGain: the gain of a frame that resolves photons is the spacing of its photon peaks, so a baseline left in the frame no longer hides it - ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization - BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain - ShadowFinder: measure the modulation robustly under a wide one-sided shadow - ShadowFinderTest: a one-sided shadow is not fitted as part of the background's modulation - ShadowFinder: back to the predicted polarization factor, in the plane the background shows - MXAnalysisAfterFPGA: the Bragg engine's edge mask in the image's layout, so raw / untransformed FPGA runs start again - battery results: still battery on rc178 e54f55 replaces the earlier rc178 still tables - XFELReader: per-frame mask and photon conversion without a library call or a branch per pixel - stills geometry trial: the hypotheses a search asks are indexed side by side - stills tilt decision: the fits start beside the pre-scan and the geometry trial - XFELReader: a deflated frame is inflated and converted a piece at a time, while it is in the cache - XFELReader: a byte-shuffled frame is unshuffled a piece at a time, a whole pixel per step - GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block - GeometryRefiner: a spot's residual differentiates its observed and predicted halves apart - docs: EXTERNAL_TEST_DATA lists every open-arm battery set, with corrected values and counts - battery: doc_check.py cross-checks the manifests against the dataset pages - stills module placement: the placement fit gets half the threads, each held-out fit an eighth - still battery: the EuXFEL sets run on all their frames, as the published still runs did - battery results: still battery on rc178 a0ce6c, cold on a quiet machine, replaces the earlier rc178 table - macOS build: explicit size_t in the XFEL reader's row bounds; parentheses in the CPU preprocessor's select Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
200 lines
9.5 KiB
C++
200 lines
9.5 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include "WriteRugnuxFiles.h"
|
|
|
|
#include <chrono>
|
|
#include <stdexcept>
|
|
#include <filesystem>
|
|
|
|
#include <gemmi/mtz.hpp>
|
|
#include <gemmi/read_cif.hpp> // cif::read_string
|
|
#include <gemmi/mmcif.hpp> // make_structure_from_block
|
|
|
|
#include "WriteModel.h"
|
|
#include "../common/Logger.h"
|
|
|
|
void WriteMtz(const MtzTable &table, const std::string &filename) {
|
|
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name(table.space_group_name);
|
|
if (sg == nullptr)
|
|
throw std::runtime_error(filename + ": unknown space group '" + table.space_group_name + "'");
|
|
const size_t rows = table.columns.empty() ? 0 : table.columns[0].values.size();
|
|
for (const auto &c : table.columns) {
|
|
if (c.values.size() != rows)
|
|
throw std::runtime_error(fmt::format("{}: column {} has {} rows, column {} {}", filename, c.label,
|
|
c.values.size(), table.columns[0].label, rows));
|
|
if (c.dataset_id < 0 || c.dataset_id >= static_cast<int32_t>(table.datasets.size()))
|
|
throw std::runtime_error(fmt::format("{}: column {} is in dataset {}, of {}", filename, c.label,
|
|
c.dataset_id, table.datasets.size()));
|
|
}
|
|
for (const auto &b : table.batches)
|
|
if (b.ints.size() != 29 || b.floats.size() != 156
|
|
|| b.ints[20] < 0 || b.ints[20] >= static_cast<int32_t>(table.datasets.size()))
|
|
throw std::runtime_error(fmt::format("{}: batch {} has no header of a dataset in the table", filename,
|
|
b.number));
|
|
|
|
gemmi::Mtz mtz;
|
|
mtz.title = table.title;
|
|
mtz.history = table.history;
|
|
mtz.spacegroup = sg;
|
|
mtz.set_cell_for_all(table.unit_cell);
|
|
mtz.sort_order = table.sort_order;
|
|
for (size_t i = 0; i < table.datasets.size(); i++) {
|
|
const MtzTable::Dataset &d = table.datasets[i];
|
|
mtz.datasets.push_back({static_cast<int>(i), d.project_name, d.crystal_name, d.dataset_name, mtz.cell,
|
|
d.wavelength_A});
|
|
}
|
|
for (const auto &c : table.columns)
|
|
mtz.add_column(c.label, c.type, c.dataset_id, -1, false);
|
|
const size_t w = table.columns.size();
|
|
mtz.nreflections = static_cast<int>(rows);
|
|
mtz.data.resize(rows * w);
|
|
for (size_t c = 0; c < w; c++)
|
|
for (size_t r = 0; r < rows; r++)
|
|
mtz.data[r * w + c] = table.columns[c].values[r];
|
|
for (const auto &b : table.batches) {
|
|
gemmi::Mtz::Batch &batch = mtz.batches.emplace_back();
|
|
batch.number = b.number;
|
|
batch.title = b.title;
|
|
batch.ints.assign(b.ints.begin(), b.ints.end());
|
|
batch.floats = b.floats;
|
|
batch.axes = b.axes;
|
|
}
|
|
mtz.write_to_file(filename);
|
|
}
|
|
|
|
gemmi::Structure PlacedModelStructure(const PlacedModelMessage &model) {
|
|
const size_t n = model.atom_name.size();
|
|
for (const size_t size : {model.model_number.size(), model.chain.size(), model.residue_name.size(),
|
|
model.seq_num.size(), model.icode.size(), model.het_flag.size(), model.element.size(),
|
|
model.altloc.size(), model.charge.size(), model.x.size(), model.y.size(),
|
|
model.z.size(), model.b_iso.size(), model.occupancy.size()})
|
|
if (size != n)
|
|
throw std::runtime_error(fmt::format("placed model: a per-atom field has {} entries for {} atoms",
|
|
size, n));
|
|
for (const size_t size : {model.subchain.size(), model.entity_id.size(), model.label_seq.size()})
|
|
if (size != 0 && size != n)
|
|
throw std::runtime_error(fmt::format("placed model: a per-atom label field has {} entries for {} atoms",
|
|
size, n));
|
|
if (!model.aniso.empty() && model.aniso.size() != 6 * n)
|
|
throw std::runtime_error(fmt::format("placed model: {} anisotropic values for {} atoms",
|
|
model.aniso.size(), n));
|
|
gemmi::Structure st;
|
|
st.name = model.name;
|
|
st.cell = model.unit_cell;
|
|
st.spacegroup_hm = model.model_space_group_name;
|
|
const bool has_aniso = !model.aniso.empty();
|
|
gemmi::Model *m = nullptr;
|
|
gemmi::Chain *ch = nullptr;
|
|
gemmi::Residue *res = nullptr;
|
|
for (size_t i = 0; i < model.atom_name.size(); i++) {
|
|
if (m == nullptr || m->num != model.model_number[i]) {
|
|
m = &st.models.emplace_back(model.model_number[i]);
|
|
ch = nullptr;
|
|
}
|
|
if (ch == nullptr || ch->name != model.chain[i]) {
|
|
ch = &m->chains.emplace_back(model.chain[i]);
|
|
res = nullptr;
|
|
}
|
|
const gemmi::SeqId seqid(model.seq_num[i], static_cast<char>(model.icode[i]));
|
|
if (res == nullptr || res->name != model.residue_name[i] || res->seqid != seqid
|
|
|| res->het_flag != static_cast<char>(model.het_flag[i])) {
|
|
res = &ch->residues.emplace_back(gemmi::ResidueId{seqid, {}, model.residue_name[i]});
|
|
res->het_flag = static_cast<char>(model.het_flag[i]);
|
|
if (!model.subchain.empty())
|
|
res->subchain = model.subchain[i];
|
|
if (!model.entity_id.empty())
|
|
res->entity_id = model.entity_id[i];
|
|
if (!model.label_seq.empty())
|
|
res->label_seq.value = model.label_seq[i];
|
|
}
|
|
gemmi::Atom &a = res->atoms.emplace_back();
|
|
a.name = model.atom_name[i];
|
|
a.element = gemmi::Element(model.element[i]);
|
|
a.altloc = static_cast<char>(model.altloc[i]);
|
|
a.charge = static_cast<signed char>(model.charge[i]);
|
|
a.pos = gemmi::Position(model.x[i], model.y[i], model.z[i]);
|
|
a.b_iso = model.b_iso[i];
|
|
a.occ = model.occupancy[i];
|
|
if (has_aniso)
|
|
a.aniso = {model.aniso[6 * i], model.aniso[6 * i + 1], model.aniso[6 * i + 2],
|
|
model.aniso[6 * i + 3], model.aniso[6 * i + 4], model.aniso[6 * i + 5]};
|
|
}
|
|
if (!model.metadata_mmcif.empty()) {
|
|
// The input model's metadata, from the mmCIF categories it came as: parsed by gemmi, never copied.
|
|
gemmi::cif::Document doc = gemmi::cif::read_string(model.metadata_mmcif);
|
|
if (doc.blocks.empty())
|
|
throw std::runtime_error("placed model: the metadata block is empty");
|
|
gemmi::Structure meta = gemmi::make_structure_from_block(doc.blocks[0]);
|
|
st.entities = std::move(meta.entities);
|
|
st.connections = std::move(meta.connections);
|
|
st.cispeps = std::move(meta.cispeps);
|
|
st.mod_residues = std::move(meta.mod_residues);
|
|
st.helices = std::move(meta.helices);
|
|
st.sheets = std::move(meta.sheets);
|
|
st.assemblies = std::move(meta.assemblies);
|
|
st.ncs = std::move(meta.ncs);
|
|
st.meta = std::move(meta.meta);
|
|
st.info = std::move(meta.info);
|
|
st.resolution = meta.resolution;
|
|
}
|
|
st.setup_cell_images();
|
|
return st;
|
|
}
|
|
|
|
namespace {
|
|
|
|
double WallClockSeconds() {
|
|
return std::chrono::duration<double>(std::chrono::system_clock::now().time_since_epoch()).count();
|
|
}
|
|
|
|
// Rename tmp onto final, unless final is there already and may not be overwritten.
|
|
void RenameIntoPlace(const std::string &tmp, const std::string &final_name, bool overwrite, Logger &logger) {
|
|
if (!overwrite && std::filesystem::exists(final_name)) {
|
|
std::filesystem::remove(tmp);
|
|
logger.Error("{} already exists and overwrite is off - not written", final_name);
|
|
return;
|
|
}
|
|
std::filesystem::rename(tmp, final_name);
|
|
logger.Info("{} written, renamed into place at wall clock {:.3f} s", final_name, WallClockSeconds());
|
|
}
|
|
|
|
} // namespace
|
|
|
|
void WriteRugnuxResultFiles(const EndMessage &end, const std::string &file_prefix, bool overwrite,
|
|
Logger &logger) {
|
|
// A file that cannot be written costs only itself: the master is already in place.
|
|
try {
|
|
if (end.merged) {
|
|
WriteMtz(*end.merged, file_prefix + ".mtz.tmp");
|
|
RenameIntoPlace(file_prefix + ".mtz.tmp", file_prefix + ".mtz", overwrite, logger);
|
|
}
|
|
} catch (const std::exception &e) {
|
|
logger.Error("Cannot write {}.mtz: {}", file_prefix, e.what());
|
|
}
|
|
try {
|
|
if (end.map_coefficients) {
|
|
WriteMtz(*end.map_coefficients, file_prefix + "_maps.mtz.tmp");
|
|
RenameIntoPlace(file_prefix + "_maps.mtz.tmp", file_prefix + "_maps.mtz", overwrite, logger);
|
|
}
|
|
} catch (const std::exception &e) {
|
|
logger.Error("Cannot write {}_maps.mtz: {}", file_prefix, e.what());
|
|
}
|
|
try {
|
|
if (end.placed_model) {
|
|
const PlacedModelMessage &model = *end.placed_model;
|
|
const gemmi::SpaceGroup *sg = gemmi::find_spacegroup_by_name(model.space_group_name);
|
|
if (sg == nullptr)
|
|
throw std::runtime_error("unknown space group '" + model.space_group_name + "'");
|
|
// WritePlacedModel appends _model.cif and _model.pdb to the prefix it is given.
|
|
const std::string tmp_prefix = file_prefix + ".tmp";
|
|
WritePlacedModel(PlacedModelStructure(model), model.unit_cell, *sg, tmp_prefix, logger);
|
|
for (const std::string suffix : {"_model.cif", "_model.pdb"})
|
|
if (std::filesystem::exists(tmp_prefix + suffix))
|
|
RenameIntoPlace(tmp_prefix + suffix, file_prefix + suffix, overwrite, logger);
|
|
}
|
|
} catch (const std::exception &e) {
|
|
logger.Error("Cannot write {}_model.cif: {}", file_prefix, e.what());
|
|
}
|
|
}
|