Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 16m43s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 28m46s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 32m5s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 39m44s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m7s
Build Packages / Create release (push) Successful in 8s
Build Packages / Build documentation (push) Successful in 49s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m30s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m54s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 11m29s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m51s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m7s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 29m38s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m6s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 29m44s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 33m22s
Build Packages / Generate python client (push) Successful in 47s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 34m6s
Build Packages / Unit tests (push) Successful in 1h2m47s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 35m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 34m8s
Build Packages / Validation Battery (push) Successful in 38m31s
* Rugnux: Refinement several times faster, one protocol on the CPU and the GPU, parts of it on the Apple GPU; its random draws are the same on every platform. * Rugnux: Refinement follows common practice: the free set decides only the restraint weights and the displacement model (isotropic, TLS or anisotropic, by Hamilton's test), everything else is decided on the working set; lower R-free for placed and predicted models. * Rugnux: Restraint dictionaries via `--restraints` (or among the inputs) and more built-in ligands; a model whose space group is a subgroup of the data's is refined in its own group; reflections plus a model select `--mode refine` on their own. * Rugnux: `-C` fixes the crystal system on the axes as typed; `-S` alone fixes the space group only, the cell is still found and reduced; a space group or cell stored in the input file is ignored, with a line in the log. * Rugnux: More reliable multi-sweep merging; `--mode scale` merges several stills runs as one; `_plot.txt` and the report name each input; a second crystal form is reported for stills given a cell. * jfjoch_broker (experimental): Live Rugnux on FPGA detectors (JUNGFRAU) takes the same path as a DECTRIS stream, rotation and serial alike, model step included, with analysis threads per GPU; frames waiting for the analysis can be viewed; no host stage drops a frame - a source faster than the analysis is held at the card. * jfjoch_broker: Summation of more than 256 frames per image is split between the card and the host (`summation_mode` is then `hybrid`), and live Rugnux runs on such collections; pedestal runs take a receiver of their own. * jfjoch_broker (experimental): `stills_downsample` analyses 1 of every N serial images (chosen by a hash; the first 2000 all; the others written without analysis; reset to 1 by `/initialize`); `max_spot_count` left out means 250 for per-image indexing and up to 1000 for live stills; the live status gives the merge before the model work, the fitted resolution and the anomalous peak height of each unexplained blob. * jfjoch_broker: Without a `pcie` entry the broker runs on a mock of the card instead of the HLS simulation; its spot, ROI and azimuthal results are zero then. * Web frontend: Redesigned Rugnux live page; Vite 8, TypeScript 6, Mol* 5.13 and dependency security updates. * Tools: `jfjoch_fpga_stills_test` measures live stills on the cards with a stored dataset; `jfjoch_fpga_test` loses the HLS-simulation option. * Other: FFTW no longer fetched or linked (the CPU FFT indexer keeps the name `FFTW`); PCIe driver now GPL-2.0-or-later and third-party notices complete; several crash fixes. Reviewed-on: #98 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
418 lines
16 KiB
C++
418 lines
16 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include "RugnuxLiveStatus.h"
|
|
|
|
#include <algorithm>
|
|
#include <cctype>
|
|
#include <cmath>
|
|
|
|
#include "../rugnux/Rugnux.h"
|
|
#include "../rugnux/ReportDocument.h"
|
|
|
|
namespace {
|
|
// A score the status carries only where it is a number: JSON has no NaN or infinity (nlohmann writes null)
|
|
void SetIfFinite(std::optional<double> &out, double v) {
|
|
if (std::isfinite(v))
|
|
out = v;
|
|
}
|
|
|
|
// The refined model's R-work and R-free: after the restrained stage, else after the rigid body and grouped B
|
|
double RefinedRWork(const RefinementResult &rf) {
|
|
return rf.restrained ? rf.r_work : rf.r_work_placement;
|
|
}
|
|
|
|
double RefinedRFree(const RefinementResult &rf) {
|
|
return rf.restrained ? rf.r_free : rf.r_free_placement;
|
|
}
|
|
|
|
// The validation's shells are the merge's, labelled by the same d_min (report_shell_d_min); its overall R-free
|
|
// is the overall row's - unless the model was refined, when the overall row is the refined model's
|
|
void AddShellRFree(std::vector<RugnuxLiveShell> &shells, std::optional<RugnuxLiveShell> &overall,
|
|
const ModelValidationResult &validation, const RefinementResult *refinement) {
|
|
for (const auto &m : validation.cc_model_shells)
|
|
for (auto &sh : shells)
|
|
if (static_cast<float>(sh.d_min) == m.d_min && m.n_free > 0)
|
|
SetIfFinite(sh.r_free, m.r_free);
|
|
if (overall && validation.ok)
|
|
SetIfFinite(overall->r_free, validation.r_free);
|
|
if (overall && refinement && refinement->ok)
|
|
SetIfFinite(overall->r_free, RefinedRFree(*refinement));
|
|
}
|
|
|
|
// A row of the report's shell table - a shell, or the overall row
|
|
RugnuxLiveShell LiveShell(const MergeStatisticsShell &in, bool stills) {
|
|
RugnuxLiveShell sh;
|
|
sh.d_min = in.d_min;
|
|
sh.d_max = in.d_max;
|
|
sh.n_obs = in.total_observations;
|
|
sh.n_unique = in.unique_reflections;
|
|
if (in.possible_unique_reflections > 0)
|
|
sh.completeness = static_cast<double>(in.unique_reflections) / in.possible_unique_reflections;
|
|
if (in.unique_reflections > 0)
|
|
sh.multiplicity = static_cast<double>(in.total_observations) / in.unique_reflections;
|
|
SetIfFinite(sh.i_over_sigma, in.mean_i_over_sigma);
|
|
if (!stills)
|
|
SetIfFinite(sh.r_meas, in.r_meas);
|
|
SetIfFinite(sh.cc_half, in.cc_half);
|
|
SetIfFinite(sh.cc_ref, in.cc_ref);
|
|
if (in.n_split_pairs > 0)
|
|
SetIfFinite(sh.r_split, in.r_split);
|
|
return sh;
|
|
}
|
|
}
|
|
|
|
double RugnuxLiveStatus::Since(std::chrono::steady_clock::time_point t) const {
|
|
return std::chrono::duration<double>(t - start).count();
|
|
}
|
|
|
|
double RugnuxLiveStatus::Now() const {
|
|
return Since(std::chrono::steady_clock::now());
|
|
}
|
|
|
|
// A step of the run, "..., N images", covers no more images than came: after a cancel the run goes on over those.
|
|
void RugnuxLiveStatus::CapImages(std::string &title) const {
|
|
const std::string suffix = " images";
|
|
const size_t pos = title.rfind(", ");
|
|
if (snapshot.cut != "cancel" || pos == std::string::npos || !title.ends_with(suffix))
|
|
return;
|
|
const std::string count = title.substr(pos + 2, title.size() - pos - 2 - suffix.size());
|
|
if (count.empty() || !std::all_of(count.begin(), count.end(), ::isdigit))
|
|
return;
|
|
if (std::stoll(count) > snapshot.images_received)
|
|
title = title.substr(0, pos + 2) + std::to_string(snapshot.images_received) + suffix;
|
|
}
|
|
|
|
void RugnuxLiveStatus::CloseStep() {
|
|
if (!snapshot.steps.empty() && !snapshot.steps.back().duration_s)
|
|
snapshot.steps.back().duration_s = Now() - snapshot.steps.back().started_s;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Start(int64_t images_expected, std::optional<double> expected_collection_s,
|
|
std::chrono::steady_clock::time_point start_received) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot = RugnuxLiveSnapshot{};
|
|
snapshot.state = RugnuxLiveState::Running;
|
|
snapshot.images_expected = images_expected;
|
|
snapshot.expected_collection_s = expected_collection_s;
|
|
start = start_received;
|
|
last_image.reset();
|
|
lattice = false;
|
|
}
|
|
|
|
void RugnuxLiveStatus::ImageReceived(std::chrono::steady_clock::time_point received) {
|
|
std::unique_lock ul(status_mutex);
|
|
++snapshot.images_received;
|
|
// Several forward threads report, so the images need not come here in the order they arrived.
|
|
const double t = Since(received);
|
|
if (!snapshot.start_to_first_image_s || t < snapshot.start_to_first_image_s.value())
|
|
snapshot.start_to_first_image_s = t;
|
|
if (!last_image || received > last_image.value())
|
|
last_image = received;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Step(const std::string &title) {
|
|
std::unique_lock ul(status_mutex);
|
|
CloseStep();
|
|
snapshot.steps.push_back(RugnuxLiveStep{.title = title, .started_s = Now()});
|
|
CapImages(snapshot.steps.back().title);
|
|
snapshot.step_progress.reset();
|
|
}
|
|
|
|
void RugnuxLiveStatus::Note(const std::string &text) {
|
|
std::unique_lock ul(status_mutex);
|
|
if (snapshot.steps.empty())
|
|
snapshot.steps.push_back(RugnuxLiveStep{.title = "", .started_s = Now()});
|
|
snapshot.steps.back().notes.push_back(text);
|
|
}
|
|
|
|
void RugnuxLiveStatus::Progress(uint64_t done, uint64_t total) {
|
|
std::unique_lock ul(status_mutex);
|
|
if (total > 0)
|
|
snapshot.step_progress = static_cast<float>(done) / static_cast<float>(total);
|
|
}
|
|
|
|
void RugnuxLiveStatus::Finish(RugnuxLiveState state, const std::string &message) {
|
|
std::unique_lock ul(status_mutex);
|
|
CloseStep();
|
|
snapshot.elapsed_s = Now();
|
|
snapshot.state = state;
|
|
snapshot.message = message;
|
|
snapshot.step_progress.reset();
|
|
}
|
|
|
|
void RugnuxLiveStatus::Result(const std::string &verdict, const std::string &verdict_text,
|
|
const std::vector<std::pair<std::string, std::string>> &summary,
|
|
const std::vector<std::string> &warnings, bool anomalous_signal) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.verdict = verdict;
|
|
snapshot.verdict_text = verdict_text;
|
|
snapshot.summary = summary;
|
|
snapshot.warnings = warnings;
|
|
snapshot.anomalous_signal = anomalous_signal;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Triage(const std::vector<RugnuxLiveTriage> &triage) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.triage = triage;
|
|
}
|
|
|
|
std::vector<RugnuxLiveTriage> LiveTriage(const ReportDocument &doc) {
|
|
std::vector<RugnuxLiveTriage> ret;
|
|
for (const auto &c: doc.checks) {
|
|
RugnuxLiveTriage t{.label = c.label, .codes = c.codes, .text = c.text, .detail = c.detail};
|
|
switch (c.status) {
|
|
case ReportCheck::Status::Absent:
|
|
t.severity = RugnuxLiveTriageSeverity::Ok;
|
|
break;
|
|
case ReportCheck::Status::Present:
|
|
t.severity = std::ranges::any_of(c.codes, PathologyIsSerious) ? RugnuxLiveTriageSeverity::Serious
|
|
: RugnuxLiveTriageSeverity::Attention;
|
|
break;
|
|
case ReportCheck::Status::Undetermined:
|
|
t.severity = RugnuxLiveTriageSeverity::NotMeasured;
|
|
break;
|
|
}
|
|
ret.push_back(t);
|
|
}
|
|
return ret;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Model(const RugnuxLiveModel &model) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.model = model;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Headline(const std::string &space_group, std::optional<double> d_min,
|
|
std::optional<double> isa, std::optional<double> resolution_fit) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.space_group = space_group;
|
|
snapshot.d_min = d_min;
|
|
snapshot.resolution_fit = resolution_fit;
|
|
snapshot.isa = isa;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Stills(const RugnuxLiveStills &stills) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.stills = stills;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Shells(const std::string &merge, const std::vector<RugnuxLiveShell> &shells,
|
|
const std::optional<RugnuxLiveShell> &overall) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.shells_merge = merge;
|
|
snapshot.shells = shells;
|
|
snapshot.overall = overall;
|
|
}
|
|
|
|
void RugnuxLiveStatus::ShellRFree(const std::string &merge, const ModelValidationResult &validation,
|
|
const RefinementResult *refinement) {
|
|
std::unique_lock ul(status_mutex);
|
|
if (snapshot.shells_merge == merge)
|
|
AddShellRFree(snapshot.shells, snapshot.overall, validation, refinement);
|
|
}
|
|
|
|
void RugnuxLiveStatus::Frames(const std::vector<float> &scales, const std::vector<RugnuxLiveFrameRange> &ranges) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.frame_scales = scales;
|
|
snapshot.frame_ranges = ranges;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Latency(double last_image_to_written_s) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.last_image_to_written_s = last_image_to_written_s;
|
|
}
|
|
|
|
std::optional<std::chrono::steady_clock::time_point> RugnuxLiveStatus::LastImage() {
|
|
std::unique_lock ul(status_mutex);
|
|
if (last_image)
|
|
snapshot.last_image_s = Since(last_image.value());
|
|
return last_image;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Cut(const std::string &reason) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.cut = reason;
|
|
snapshot.cut_s = Now();
|
|
for (auto &step: snapshot.steps)
|
|
CapImages(step.title);
|
|
}
|
|
|
|
void RugnuxLiveStatus::Probe(const std::string &text) {
|
|
std::unique_lock ul(status_mutex);
|
|
if (lattice)
|
|
return;
|
|
snapshot.probe = text;
|
|
snapshot.probe_s = Now();
|
|
}
|
|
|
|
void RugnuxLiveStatus::Lattice(const std::string &text) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.probe = text;
|
|
snapshot.probe_s = Now();
|
|
lattice = true;
|
|
}
|
|
|
|
void RugnuxLiveStatus::Warning(const std::string &text) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.warnings.push_back(text);
|
|
}
|
|
|
|
void RugnuxLiveStatus::Structure(std::shared_ptr<const RugnuxStructure> structure,
|
|
std::shared_ptr<const RugnuxStructure> refined_structure) {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot.structure = std::move(structure);
|
|
snapshot.refined_structure = std::move(refined_structure);
|
|
}
|
|
|
|
void RugnuxLiveStatus::Clear() {
|
|
std::unique_lock ul(status_mutex);
|
|
snapshot = RugnuxLiveSnapshot{};
|
|
last_image.reset();
|
|
lattice = false;
|
|
}
|
|
|
|
RugnuxLiveSnapshot RugnuxLiveStatus::Get() const {
|
|
std::unique_lock ul(status_mutex);
|
|
RugnuxLiveSnapshot ret = snapshot;
|
|
if (snapshot.state == RugnuxLiveState::Running)
|
|
ret.elapsed_s = Now();
|
|
|
|
// Held since START, reported since the first image - or, before it has come, since now.
|
|
const double zero = snapshot.start_to_first_image_s.value_or(snapshot.state == RugnuxLiveState::Running ? ret.elapsed_s : 0.0);
|
|
ret.elapsed_s -= zero;
|
|
for (auto &step: ret.steps)
|
|
step.started_s -= zero;
|
|
if (ret.probe_s)
|
|
*ret.probe_s -= zero;
|
|
if (ret.last_image_s)
|
|
*ret.last_image_s -= zero;
|
|
if (ret.cut_s)
|
|
*ret.cut_s -= zero;
|
|
return ret;
|
|
}
|
|
|
|
std::vector<RugnuxLiveShell> LiveShells(const MergeStatistics &statistics, const ModelValidationResult *validation) {
|
|
std::vector<RugnuxLiveShell> ret;
|
|
// The rows of the report's table: a shell with no reflection is not printed
|
|
for (const auto &in : statistics.shells)
|
|
if (in.unique_reflections > 0)
|
|
ret.push_back(LiveShell(in, statistics.stills));
|
|
std::optional<RugnuxLiveShell> no_overall;
|
|
if (validation)
|
|
AddShellRFree(ret, no_overall, *validation, nullptr);
|
|
return ret;
|
|
}
|
|
|
|
RugnuxLiveShell LiveOverall(const MergeStatistics &statistics, const ModelValidationResult *validation,
|
|
const RefinementResult *refinement) {
|
|
std::vector<RugnuxLiveShell> no_shells;
|
|
std::optional<RugnuxLiveShell> ret = LiveShell(statistics.overall, statistics.stills);
|
|
if (validation)
|
|
AddShellRFree(no_shells, ret, *validation, refinement);
|
|
return *ret;
|
|
}
|
|
|
|
std::optional<RugnuxLiveModel> LiveModel(const ProcessResult &result) {
|
|
if (!result.model_validation && !result.model_prior)
|
|
return std::nullopt;
|
|
RugnuxLiveModel m;
|
|
m.placement = result.model_placement;
|
|
if (result.model_prior) {
|
|
const MolecularReplacementResult &mr = *result.model_prior;
|
|
m.model_name = mr.model_name;
|
|
m.predicted = mr.predicted_model;
|
|
m.mr_decision = mr.decision;
|
|
if (!mr.ok)
|
|
m.reason = mr.failure_reason;
|
|
if (mr.chosen >= 0) {
|
|
const MRCandidate &c = mr.candidates[mr.chosen];
|
|
m.mr_space_group = c.space_group.xhm();
|
|
m.copies = c.copies;
|
|
SetIfFinite(m.tf_z, c.tf_z);
|
|
SetIfFinite(m.rf_z, c.rf_z);
|
|
SetIfFinite(m.llg_z, c.llg_z);
|
|
SetIfFinite(m.fit_sigma, c.null_sigma);
|
|
SetIfFinite(m.mr_d_min, mr.d_min_search);
|
|
}
|
|
}
|
|
if (result.model_validation) {
|
|
const ModelValidationResult &v = *result.model_validation;
|
|
if (!v.model_name.empty())
|
|
m.model_name = v.model_name;
|
|
if (!v.ok) {
|
|
if (m.reason.empty())
|
|
m.reason = v.failure_reason;
|
|
} else {
|
|
SetIfFinite(m.r_work, v.r_work);
|
|
SetIfFinite(m.r_free, v.r_free);
|
|
SetIfFinite(m.cc_model, v.cc_model_overall);
|
|
if (v.fit_tested)
|
|
SetIfFinite(m.fit_sigma, v.r_work_sigma);
|
|
SetIfFinite(m.map_atom_sigma, v.mean_atom_density_sigma);
|
|
if (v.rigid_body_applied) {
|
|
SetIfFinite(m.rigid_body_angle_deg, v.rigid_body_angle_deg);
|
|
SetIfFinite(m.rigid_body_shift_A, v.rigid_body_shift_A);
|
|
}
|
|
if (!v.anomalous_sites.empty()) {
|
|
m.anomalous_site = v.anomalous_sites.front().label;
|
|
SetIfFinite(m.anomalous_site_sigma, v.anomalous_sites.front().sigma);
|
|
}
|
|
}
|
|
}
|
|
if (result.refinement) {
|
|
const RefinementResult &rf = *result.refinement;
|
|
RugnuxLiveRefinement r;
|
|
r.performed = rf.ok;
|
|
r.cancelled = rf.cancelled;
|
|
r.reason = rf.failure_reason;
|
|
r.on_gpu = rf.on_gpu;
|
|
r.seconds = rf.seconds;
|
|
if (rf.ok) {
|
|
SetIfFinite(r.r_work, RefinedRWork(rf));
|
|
SetIfFinite(r.r_free, RefinedRFree(rf));
|
|
if (rf.restrained) {
|
|
SetIfFinite(r.rmsd_bonds_A, rf.geometry.rmsd_bonds);
|
|
SetIfFinite(r.rmsd_angles_deg, rf.geometry.rmsd_angles);
|
|
}
|
|
if (rf.solvent_build.ran) {
|
|
r.waters_added = rf.solvent_build.added;
|
|
r.waters_removed = rf.solvent_build.removed;
|
|
}
|
|
}
|
|
r.blob_count = static_cast<int64_t>(rf.blobs.size());
|
|
for (size_t i = 0; i < std::min(rf.blobs.size(), RugnuxLiveRefinement::MAX_BLOBS); i++) {
|
|
const DensityBlob &b = rf.blobs[i];
|
|
RugnuxLiveBlob blob{DensityBlobKindName(b.kind), b.centroid.x, b.centroid.y, b.centroid.z, b.peak_sigma,
|
|
b.volume, b.nearest_atom, std::nullopt, std::nullopt};
|
|
SetIfFinite(blob.nearest_atom_distance_A, b.nearest_atom_distance); // infinite: no atom near
|
|
SetIfFinite(blob.anomalous_sigma, b.anomalous_sigma); // NaN: no anomalous map
|
|
r.blobs.push_back(blob);
|
|
}
|
|
m.refinement = std::move(r);
|
|
}
|
|
return m;
|
|
}
|
|
|
|
std::vector<RugnuxLiveFrameRange> LiveFrameRanges(const SweepQuality &sweep) {
|
|
std::vector<RugnuxLiveFrameRange> ret;
|
|
const auto &d = sweep.frame_disposition;
|
|
for (size_t i = 0; i < d.size();) {
|
|
size_t j = i + 1;
|
|
while (j < d.size() && d[j] == d[i])
|
|
j++;
|
|
if (d[i] != static_cast<uint8_t>(FrameDisposition::Merged)) {
|
|
RugnuxLiveFrameRange r{.first_image = static_cast<int64_t>(i), .last_image = static_cast<int64_t>(j - 1),
|
|
.disposition = FrameDispositionCode(static_cast<FrameDisposition>(d[i]))};
|
|
for (const auto &range: sweep.ranges)
|
|
if (range.first_image <= r.last_image && range.last_image >= r.first_image) {
|
|
r.reason = SweepQualityReasonText(range.reason);
|
|
break;
|
|
}
|
|
ret.push_back(r);
|
|
}
|
|
i = j;
|
|
}
|
|
return ret;
|
|
}
|