6 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	78c2a69267	Merge broker-rugnux-live into rc176: rugnux in jfjoch_broker as a live thread during a DECTRIS rotation collection (off by default), the rugnux library without file I/O (results in ProcessResult::outputs, the CLI writes them through JFJochRugnuxOutput), END carrying the merged table, map coefficients and placed model ... Branch reviewed at e16db07ff (review: owner's decisions accepted - _maps.mtz instead of .ccp4 in the viewer, -o "" computes everything incl. MR/validation, Ctrl+C/error paths may leave fewer files, the writer links gemmi/zlib) and merged at 55184eb09 with its review fixes (END size guard, frame drop as in Lite, outputs released before re-runs, pool priority, docs, frontend gating). Conflicts: rugnux/Rugnux.h (kept rc176's host_memory_budget_bytes, output_prefix removed as on the branch), rugnux/RugnuxPipeline.h (MtzTable future + rc176's GPUMerge pointer), MolecularReplacement.h/.cpp (rc176's GPU cost model, packed transform and split functions with the branch's RugnuxModel signature; search_model with rc176's other_budgets, result.model_name). tests/RugnuxLargeTest.cpp: rc176's host-budget test writes its files through WriteRugnuxOutputs now that the library writes none. Generated code (broker/gen, frontend client, Python client docs, Redoc) regenerated from the merged jfjoch_api.yaml with update_version.sh (version unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 21:01:38 +02:00
leonarski_fandClaude Opus 5.5	5f731db81d	rugnux: the library returns its outputs in memory; the callers write the files ... The rugnux library wrote its results itself: _process.h5 through a FileWriter, the merged/twin/_mr reflection files, the unmerged MTZs, the maps and their MTZ, the placed models, _detector.jpg and _plot.txt, and on a doubled cell it renamed files on disk. It now writes none of them. - MtzTable (common/MtzTable.h): an MTZ file's header, columns and batch headers with no file behind it. It replaces MergedReflectionsMessage and MapCoefficientsMessage in the END message, with one CBOR encoding. WriteMtz (writer/WriteRugnuxFiles.h) is the one serialiser, used by rugnux and jfjoch_writer alike; it refuses a table whose columns differ in length, name a missing dataset, carry a malformed batch header or name a space group gemmi does not know. MergedMtzTable and UnmergedMtzTable build the tables the old writers built. - ProcessResult::outputs (RugnuxOutputs) holds, for the output pass only: the answer's merge (MTZ table plus what the .cif/.hkl are written from), the twin hypothesis' and the model's choice's merges, the unmerged and unmerged-partials tables, the maps (coefficients always; the grids only with ProcessConfig::keep_map_grids), the placed models, the detector JPEG and the per-image plot table, and the start message of the no-lattice _process.h5. Map RMS is taken from the grid in memory, as the CCP4 header stated it, whether or not the map is kept. - The per-image analysis goes to a caller-supplied RugnuxImageSink; rugnux and the viewer give a FileWriter-based ProcessFileSink. The geometry pre-pass no longer makes a _01_process.h5. - The doubled cell keeps both output sets: the answer's in outputs, the other's in DoubledCell::other_outputs with its suffix; the report names files by suffix after the caller's prefix. - The output prefix, write_process_h5 and the observation dump path are gone from ProcessConfig; --dump-observations is a callback. - What used to be gated on having files to write (model validation and MR, the twin-hypothesis merges, the finalist ledger, the scaled fulls, the unmerged tables) is gated on the output pass instead, so a caller with no prefix gets the same results. - RugnuxOutputFiles (new library JFJochRugnuxOutput, linked by rugnux, the viewer and the tests) writes every file from the result with the names and formats rugnux has always used. The viewer writes its job's files the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 16:51:20 +02:00
leonarski_fandClaude Opus 5.5	4bf41983ed	rugnux: the model is parsed by the caller and handed in as RugnuxModel ... The analysis read the --model file itself, four to five times per run (the model kind, every validation, molecular replacement, the stills reference intensities) and located the PAE sidecar from the file name. The library is to do no file I/O, so the model now arrives in memory: - RugnuxModel (image_analysis/structure_refinement/RugnuxModel.h): the gemmi structure as read, whether it is ModelCIF, the PAE matrix where a prediction came with one, and the name the report gives it. - ParseRugnuxModel / ReadRugnuxModelFile (writer/RugnuxModelFile.h, in JFJochRugnuxFiles): bytes in, gzip by magic, PDB/mmCIF by content, _ma_qa_metric detection as before; the file reader takes the PAE JSON from the same place PaePathFor named. - ProcessConfig::model replaces model_path; PrepareSearchModel, RunMolecularReplacement, ValidateAgainstModel and ModelReferenceIntensities take the model; result fields name it by model_name. - rugnux reads the model before the run starts and stops with an error where the file holds no model (it used to run and report the model as not validated). The viewer reads it when the job is set up. - The broker parses an uploaded model at once and answers 400 where it is not one; it keeps the parsed model in memory, with no temporary directory or file. It takes no PAE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 16:11:58 +02:00
leonarski_fandClaude Opus 5.5	3fa17b9385	Model validation: the rigid-body engines on every card in turn, even where this card has room for all ... The engines past this card's first now go one card after the other, from the next card and this one last, each within half of its own budget, as MR's do. Before, this card took every extra engine it had room for and the other cards only the rest, so a small cell on four cards ran all its fits on one. Where this card has no room for a second engine (a large cell on 15.6 GB cards) the plan is unchanged: 1/3/3/3. This card's scale fits take the room its engines leave. Engines are interchangeable and deterministic, so only where fits run changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 10:34:05 +02:00
leonarski_fandClaude Opus 5.5	bfc6ca8487	MR and model validation: use every card's memory, split MR's functions over the cards ... On a multi-GPU host the structure stage ran essentially on one card: the rotation function's launches all went to the search's card, and the rigid-body engines of model validation and MR were admitted only within half of that one card's budget, although RigidBodyGPUPool already placed engines on other cards. On a 4x T4 server 4nwv's MR rotation function took 75.8 s on one card, validation queued its 10 fits on one engine and MR's rescoring and null got one engine. - PlanModelValidationGPU / PlanMolecularReplacementGPU: this card is planned exactly as on a machine with it alone (GPU-vs-CPU placement unchanged: 80% of the total, less what is resident). The rigid-body engines it has no room for go on the other cards in turn, each against its own budget (80% of its total, less the resident bytes, counted on every card) within half of it. MR also puts a copy of the molecular transform on each other card within half of its budget. - RigidBodyGPUPool::Reserve takes per-card engine counts. - Validation forwards the most it holds on any card as gpu_bytes to a validation beside, and may run one beside only where twice that fits every card's budget. - MR splits every rotation and translation function into contiguous slices of the orientation list over the cards holding a transform copy (and room for the call's buffers), one thread per card, gathered back in order. Each orientation is scored by itself (one block / one map), nothing is summed across orientations, so N cards give the bytes of one. - ScoreEngines keeps a room per card; score engines always go on the search's own card; function buffers are held on the card they run on. Byte-identical at N=1 (workstation, one RTX 5080): timing benchmark (myob, cytc, 4nwv) and 4nwv, 8tyy, 6oel, 3inp with --model give the same p.hkl, MTZ data, p_model.pdb, maps and report against rc176. New tests: per-card plans on four 15.6 GB cards (validation 1/3/3/3 engines, MR T+1/T+2/T+2/T+2), and the split driven over 1-4 virtual cards on one device, bitwise equal to one card. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-16 17:16:28 +02:00
leonarski_fandFilip Leonarski	19b135186d	v1.0.0-rc.175 (#85) ... * Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`). * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS). * Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead. * Rugnux: On macOS, Bragg integration runs on the Apple GPU. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. * Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`. * Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. * Rugnux: The beam-stop shadow is found correctly on tilted detectors. * Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement. * jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps. * jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images. * Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. --------- Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home> Reviewed-on: #85 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-16 15:34:43 +02:00