12 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	011b98838e	Molecular replacement: one GPU plan per merge on model validation's budget, the transform cropped, output unchanged ... What molecular replacement put on its GPU was decided part by part on a quarter of the card each: every group's rigid-body pool (up to three engines), every null's (up to nine), the score engines' cache, the --mr hand check's structure factors. The groups and the nulls run side by side, so the quarters added up: on 4nwv on a 16 GB card the search held 4 rigid-body engines of 1.88 GB, two score engines of 1.64 GB, the 1.82 GB molecular transform, two translation functions' 1.2 GB and the 2.4 GB the allocation pool kept from the validation before - 13.8 GB at its peak, 83% of the card. On an 8.55 GB card that pages. Now each merge's search is planned once, before it starts, on the budget model validation uses (ModelGPUBudget: 80% of the card's total, less what a merge engine still at work holds): the transform, one rigid-body engine and the largest score engine take their place in that order where they still fit, more rigid-body engines only within half of the budget, and what is left is a room the score engines (least recently used given up first, as before) and the rotation and translation functions' working buffers share, each waiting for the other to finish rather than spilling past it (PlanMolecularReplacementGPU). A part that does not fit runs on the CPU, decided on its size alone. - One rigid-body pool per merge serves every fit of the merge - the groups' first and further copies, the nulls, the tNCS shifts, the carried copies - sized for the largest (the most copies Matthews allows, in any group) and reserved once, instead of a pool made and freed for each. RigidBodyGPUPool::Cover widens a planned pool to another model or group; its zones are kept by group, atom count and resolution. RigidBodyGPUPool::Create (the quarter rule) is gone. - The sizes are worked out on the CPU while the transform is computed. - The molecular transform keeps only the indices it is read at (|k|, l <= box / d_min + 1): 0.84 GB instead of 1.82 on 4nwv, host and device. Its rows keep their full length: cut short too, the rotation function read them 15% slower. - The translation function's prepared setup is kept on the host and uploaded by each call, so the device holds it only while a function runs; the cross-term offsets are uploaded only where there are cross terms. - The allocation pool is trimmed before the search, after the transform's GPU FFT, before a score engine is made and before the hand check, whose structure factors are now judged against the budget. - rugnux passes the merge engine's bytes, where the finalist ledger keeps it, to the search, the hand check and the validation of the placed model. Measured on the RTX 5080 (16.6 GB), timing benchmark: peak device memory during MR 13.8 -> 8.7 GB on 4nwv (start 3.0 -> 0.66 GB after the trim), 1.67 -> 1.14 GB on cytc, 6.3 -> 0.76 GB on myob. On 4nwv the plan holds the transform 0.84 GB, one rigid-body engine of 2.10 GB and leaves 10.35 GB of a 13.29 GB budget to the score engines and the functions; a second engine would pass half of the budget (one engine against two: +1.2 s on the rescoring and the null). MR 31.7 -> 31.6 s on 4nwv. Reflection files, maps, placed models, reports and the MR candidate lines byte-identical to rc175-metal-merge on myob, cytc and 4nwv; [MolecularReplacement], [ModelValidation], RigidBody*, ModelScale*, [CUDAMemHelpers] pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 13:33:39 +02:00
leonarski_fandClaude Opus 5.5	3373106ae1	Model validation: budget counts only what is really held; fewer extra engines ... On an 8.55 GB card (WDDM) the validation planned 6.81 GB of a 6.84 GB budget, but the card already held more than the 20% headroom allowed, and the driver paged GPU memory to system RAM: every rigid-body fit ran ~5x slower. Measured on 4nwv on a 16 GB card, at the start of model validation the device held 1.1 GB beyond a fresh context, and on 6oel 2.7 GB: - the stream-ordered allocation pool's free-but-reserved memory, left by the image passes and the scaling's pooled buffers (1.07 GB on 4nwv, 2.1 GB on 6oel); the validation's engines are allocated outside the pool and cannot reuse it; - the image pass's FFT indexers (IndexerThreadPool), alive for the whole run (0.8 GB on 4nwv, 0.5 GB on 6oel), although nothing indexes after the merge. Both are now released: the validation trims the pool (cuda_trim_allocation_pool) before it plans, and rugnux drops the indexer pool once the leftover-lattice census, its last user, has ended, before the model is validated or searched for. What is left beside the validation is 0.06 GB over a fresh context on both sets (streams, idle cuFFT plans). Extra engines are now added only within half of the budget. They change no number and bought little where measured (4nwv, 16 GB card: 13.6 s with one rigid-body engine, 13.1 s with two, 12.6 s with five), so the card keeps headroom instead. On 16 GB 4nwv now plans 2 rigid-body and 2 scale-fit engines, 6.56 GB (measured peak 7.3 GB with the context); on an 8.55 GB card one of each, 4.42 GB. Reflection files, maps and models byte-identical to rc175 HEAD on 4nwv, 8tyy, 6oel and 3inp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 09:56:15 +02:00
leonarski_fandClaude Opus 5.5	0d25d765cf	Model validation: one GPU budget for all its parts, 80% of the card's total ... What a validation puts on its GPU was decided part by part on fixed fractions of the card: the structure factors to d_min on a quarter, the null's structure factors on whatever of that quarter the first left, the rigid-body engines on another quarter, the scale fits on an eighth. On an 8 GB card that sent the null's structure factors of a 4.4 GB validation to the CPU (and held the rigid body to one engine) although everything the validation holds fits the card with room to spare. Now PlanModelValidationGPU decides all of it at once, before anything is reserved, from the card's total memory: every engine is reserved up front and kept to the end, so their sum is held against one budget, 80% of the card less what a merge engine still at work holds there (the finalist ledger's P1 merge, RotationScaleMerge::DeviceBytes). Each part takes one engine where it still fits, in the order structure factors to d_min, rigid body, the null's structure factors, scale fits; a part that does not fit runs on the CPU. What is left goes to more rigid-body engines (up to one per null replicate plus the real fit) and then more scale-fit engines, which only change how many fits run at once. A second validation may run beside where twice this one's whole GPU plan fits the cards' budgets; it gets this one's bytes as gpu_beside_bytes, which limit its engine counts and never its placement, so its numbers are those of the validation it stands in for. One log line states the placement, the bytes held and the budget. - RigidBodyGPUPool::Plan sizes the pool without engines (EngineBytes), Reserve(n) makes them; Create (Plan + engines on a quarter of the card) stays for molecular replacement, whose own rules, and the --mr hand check's structure factors (ReadModelAnomalousMap, a quarter), are unchanged. - The null's resolution is worked out before the plan, so its engine is decided and reserved with the others instead of after the indexing probe. Placement changes only on smaller cards: with the logged footprints of the open-arm sets, 4nwv's null structure factors move to the GPU below ~9 GB (and its rigid body at 6 GB), 8tyy's structure factors to d_min below ~8.7 GB; on 16 GB and larger nothing moves. On the RTX 5080 (16 GB), 4nwv, 8tyy, 6oel and 3inp give byte-identical reflection files, maps and models to rc175 HEAD, with more rigid-body engines where they fit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 08:05:50 +02:00
leonarski_fandClaude Opus 5.5	d0d225cb99	rugnux --model: molecular replacement where the rigid body does not place the model ... One --model now does the right thing by itself: - A prediction (AlphaFold/ModelCIF, read from the file) goes straight to molecular replacement. - An experimental structure is placed by the rigid body as before. The null now runs on every --model run (no NOT_TESTED): "placed" = the fit beats its null (4.51 sd) in the answer's merge or in the twin hypothesis' merge. Placed: behaviour unchanged (hand and reindex may go into p.mtz); a subgroup the model fits and prefers is written as <prefix>_mr. - Not placed (or no crystal cell): molecular replacement with the experimental-model path. It may relabel the hand into p.mtz past the anomalous veto, never reindexes; another group or the twin subgroup goes only to <prefix>_mr. Maps, R-free and CC come from validating the MR placement with every decision switched off; MODEL_FIT stays the rigid body's verdict. - The twin hypothesis' merge is made in memory with a model where a metric re-ask promoted the group or the model's group has fewer rotations than the adopted one; written and reported only where the model chooses it. - --mr is kept as "--model, searched for by MR straight away". - Report: MODEL_PLACEMENT= RIGID_BODY|MR|NOT_PLACED, MODEL_DECISIONS_TAKEN gains MR_ENANTIOMORPH, MODEL_CHOICE_FILE_PREFIX/SPACE_GROUP; the MR keys move into 5. MODEL VALIDATION; MODEL_FIT absent where no rigid body ran. MR NOT_PLACED is never a warning. - Battery wrong-model step: check (c) also fails a wrong model placed by MR. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 22:20:36 +02:00
leonarski_fandClaude Opus 5.5	30c7bee15f	rugnux: --af-model becomes --mr, for predictions and experimental structures alike ... - The option is --mr <file>. What kind of model it is is read from the file: a prediction where it is ModelCIF (_ma_qa_metric) or has no cell and one B value per residue in 0-100 (AlphaFold PDB/mmCIF); an experimental structure otherwise. A prediction is handled as before (pLDDT >= 70 trim, B from pLDDT, PAE domains where the AlphaFold DB's PAE file sits beside it). An experimental structure is taken as its polymer - no waters, ligands, hydrogens or alternative conformers, occupancy 1, isotropic B as refined - with a coordinate error of 0.67 A (a prediction's at pLDDT 90) for sigma_A; a B column above 100 no longer stops the search. - The report section is MOLECULAR REPLACEMENT (EXPERIMENTAL), its keys MR_* (MR, MR_MODEL_FILE, MR_MODEL_KIND, MR_DECISION, MR_SPACE_GROUP, MR_COPIES, MR_HAND_CHECK, MR_SECONDS, ...); the <prefix>_mr outputs are unchanged. - The twin hypothesis' merge that is made only for the model to test (a metric re-ask promoted the indexed lattice's group) is written and reported only where the model chooses the subgroup. Raw deposited coordinates of nine control sets, unconverted: every one placed and confirmed against the deposition (all chains as one body), the groups the same as with the prediction or the prepared model. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 19:30:22 +02:00
leonarski_fandClaude Opus 5.5	ee350f1f74	rugnux --model: a wrong model confirms no shell, fits no hand, and fits the GPU reserve ... Three defects the battery's wrong-model null found (5 sets x 5 deposited models of unrelated proteins): CC(model, data) significance. The per-shell sigma was Fisher's transform on independent reflections, but the data and a scaled model share the falloff across a shell, the anisotropic B fitted to these very data, the bulk solvent and the symmetry enhancement epsilon - whatever the model is. 21 of 24 wrong models read CC_MODEL_CONFIRMED_TO_D_MIN to the data's d_min (10-15 sigma in most shells). The correlation is now formed on I/epsilon and |F_model|^2/epsilon, and its null is empirical: the data at h against the model at h + d for the 26 nearest lattice offsets, which keeps everything smooth in reciprocal space and loses what the model knows of h. SIGMA = (z - mean) / max(sd, 1/sqrt(N-3)) in Fisher z over those 26, a new CC_NULL column shows the null, and a shell confirms only beside a coarser shell that also clears 3 sigma (one shell alone did so once in 216 wrong-model shells). Prototyped on the p_maps.mtz of a full open-arm run and the null runs: wrong models confirm 0/24 (was 21/24), the deposited models confirm 197/200 (was 198/200 in the prototype's reading; the one lost has CC_MODEL 0.05-0.13 in every shell). The report no longer says a model cannot agree by accident. MODEL_FIT gate. Accepted at 3.0 sample sds of a nine-draw null, as if the sd were known; an unrelated P1 helical model on a P3(1) helical crystal was ACCEPTED at +3.05. Against 48 draws it reads +2.09 and the best random placement fits as well as it does, and drawing the null about the cell axes instead changes nothing (+2.15): the cause is the nine-draw sd, not the orientation. The gate is now 3 sigma allowing for that: t(8) * sqrt(1 + 1/9) = 4.51. The deposited models of the open arm read +7.0 and up. The indexing lead stays at 3 sigma (INDEXING_MARGIN_SIGMA): it is read only behind the fit gate, and a deposited model settles one at +4.15. Rigid-body GPU reserve. The finest zone's observations were counted by the data's d, but the zones count them by the d of the index in the model's ASU, and a model's group need not keep the data's metric (an I2(1)3 model on I222 data: 2320 in the zone against 1712 reserved), so the validation died with "zone over the engine's reserve". Counted the zones' way, under every indexing probed. Tests: ModelValidation_CCModelUnrelatedModelConfirmsNothing (Fisher alone reads +4.7 on an unrelated model with an anisotropic falloff; the new sigma stays under 1), ModelValidation_WrongModelInAGroupOffTheDataMetric (fails with the old count), ResultReport_ModelValidationSection (CC_NULL column, lone-shell rule). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 21:30:34 +02:00
leonarski_fandClaude Opus 5.5	50e3b59fb7	rugnux --af-model: hand check reads only the anomalous map, on MR's placement and solvent ... The ENANTIOMORPH hand check ran a whole ValidateAgainstModel on the placed model: model read and frame probe, a rigid body, a 9-replicate null (~0.5 s of the ~0.6-0.9 s), sigma_A, three maps written. The decision needs only --model's rule: the model fits its null (the MR choice has already required its own null of the same construction) and the anomalous map at the model's atoms is not inverted. ReadModelAnomalousMap (ModelValidation) computes F_calc and F_mask to the data's resolution (GPU engine where there is one), carries them through the bulk solvent MR scored the placement with (the overall scale and anisotropic B do not move phases), and reads the anomalous map with the same code as the validation: its coefficient construction and its reading at the atoms are factored out of Validate (anomalous_coefficients, read_anomalous_map) and used by both, so plain --model is unchanged. Hand check 0.6/0.6/0.9 s -> 0.027/0.028/0.040 s on 11if/7ris/9zmu; same ADOPTED/VETOED on all seven cases tried (the three demos, two anomalous lysozyme sets and their mirrored-detector runs), with the scatterer means and deepest troughs within 0.1 sigma of the full validation's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 18:11:01 +02:00
leonarski_fandClaude Opus 5.5	8442d55f6f	rugnux --model: score the twin hypothesis too, and take the maps from the merge the model prefers ... Where a twin hypothesis (<prefix>_twin_<SG>) was written, the model is also validated against that merge: as merged, and as twinned by its law - |F_twin(h)|^2 = (1-a)|F_model(h)|^2 + a|F_model(Th)|^2 from the placed, scaled model, one overall scale refitted per a, a scanned 0..0.5 on the working set and R-free read at it (ValidateAgainstModel's new twin_law argument; GPU structure factors as before). The maps and the placed model are then written from whichever merge has the lower R-free; the report gives MAPS_FROM and the subgroup's plain and twinned R-work/R-free and fraction. The subgroup file follows its own validation's indexing, as the answer follows its own. The model chooses the maps only: the space group, <prefix>.mtz and R_FREE stay the answer's. Documented in RUGNUX_REPORT.md, RUGNUX_ADVANCED.md and CPU_DATA_ANALYSIS_DECISIONS.md (14.9); Yeates (1997) credited. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:40:22 +02:00
leonarski_fandClaude Opus 5.5	9a1f729993	rugnux: do not make the first validation's maps where the forecast one is kept ... Where the validation in the model's setting, started beside the first on the forecast (b5980c9ba), is going to be kept, it writes the maps, the map coefficients and the placed model on the written axes, and the first validation's p_2fofc/p_fofc/p_anom.ccp4 and p_maps.mtz were overwritten unread. ModelValidationSchedule::maps_superseded is asked once the verdict, the setting and the indexing are decided, before any map is made; the first validation then makes none (the map coefficients and the MTZ are not built, the three map transforms not run, nothing written). The answer is the same rule that keeps the forecast validation afterwards (forecast_kept, now one lambda for both). The kept validation also brings back the reflections it relabelled, free set included, so relabel_output does not reindex and redraw them a second time. Where the forecast is discarded or was never started, everything runs as before, and modelpar's delete-before-rewrite stays for those cases. If the kept validation then fails, a warning says no maps were written. Log lines that disappear (first validation only, kept-forecast runs): "mean 2mFo-DFc density at atom centres = ...", "anomalous difference map from ... Bijvoet pairs; strongest density ...", "wrote p_2fofc.ccp4, ..."; in their place: "no maps made here - the validation in the model's setting writes them". (The "anomalous density ... inverted" warning would also go; it cannot be raised where a forecast is kept.) Verified on 8sa8, 8xtg, 9ea5 (forecast kept), 8tyy, 7qis, myob_x10sa (no forecast): p.mtz, p.hkl, p.cif, the three maps, p_maps.mtz and p_model.cif/pdb md5-identical to the merge commit, repeated and at -N 8; the discard path (8sa8 with a rotated model) identical files and log. Time saved is small with the maps already on the GPU: 8sa8 7.6 -> 7.4 s, 8xtg 5.1 -> 5.0 s, 9ea5 within noise. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:34:46 +02:00
leonarski_fandClaude Opus 5.5	b5980c9ba0	rugnux: start the validation in the model's setting on a forecast, beside the first ... Where a model that fits is written on other axes than the data, the files take its setting and the validation is made a second time on the relabelled data. That second validation depends on the first only through the setting and the indexing the first settles, and both are known long before the first has finished: the change of basis right after the frame scoring, the indexing the probe prefers before the null. So ValidateAgainstModel now reports them (ModelFrameForecast, through ModelValidationSchedule:: on_forecast), and the second validation is started there, on the relabelling AdoptModelFrame and relabel_output would make, applied to a copy of the merge - beside the first one's null, real fit and maps. It is kept only where the first decides exactly what was forecast (the model fits, same change of basis, same indexing; a model asserting the other enantiomorph is not forecast, as the label is decided last, on the anomalous map). Until then its log is held (Logger::Buffered, replayed where the serial run logged it) and its files wait on a gate (ModelValidationSchedule::write_gate) placed before the first map is written; otherwise it is released with false and returns unwritten, and the serial validation runs as before. GPU memory: two validations at once take twice the rigid-body engines. The parallel start is decided up front from sizes, never from what is free: allowed where twice what the first pool asked for (bytes per engine times the engines wanted) fits a quarter of the card's TOTAL memory, the share one validation may take. Threads: both validations submit to the one ParallelFor pool from threads outside it (the second runs on a std::async thread, as the null's replicates do), so no pass runs inline on a pool worker and the pool's size bounds the workers. Measured on the loaded 16-core workstation, TIMING model validation: 8sa8 30.6 -> 19.7 s, 8xtg 19.9 -> 13.9 s, 9ea5 22.1 -> 16.4 s (this and the previous commit together). p.mtz, p.hkl, p.cif, the three maps, p_maps.mtz and p_model.cif/pdb md5-identical to the base on 8sa8, 8xtg, 9ea5, 7qis and myob_x10sa; the validation's log lines identical as a set (the null's replicate lines were already in completion order). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 12:10:46 +02:00
leonarski_fandClaude Opus 5.5	aabac1541a	ModelValidation: score the candidate changes of basis at the same time ... Each candidate setting of the model was scored one after another on the coarse shell (a density, an FFT and an isotropic scale fit, 0.2-0.5 s each), and a run with a model in another setting scores them in both of its validations. They share nothing but what they read, so they now run on ParallelFor, each on its own copy of the model, and are read back in their own order: the log lines, the ranking and the tie-break (first of equal R wins) are those of the serial loop. Measured (16-core workstation, loaded): 8sa8 3 candidates 0.64 s -> 0.22 s; 9ea5 3 candidates 1.47 s -> 0.50 s; 8xtg 1.45 s -> 0.40 s. p.mtz, the maps, the map MTZ and the placed model md5-identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 12:08:39 +02:00
leonarski_fandClaude Opus 5.5	9ad92b6bfe	Move the atomic-model code to image_analysis/structure_refinement/ and WriteModel to writer/ ... A pure move. ModelValidation, RigidBodyRefine, RigidBodyGPU, ModelFFT, ModelGrid, ModelScaling, ModelMaskGPU, ModelScaleGPU and SigmaA - everything that works on an atomic model - become the JFJochStructureRefinement library, linked by JFJochImageAnalysis. WriteModel (the placed-model mmCIF/PDB writer) goes to writer/ as its own small JFJochModelWriter target, so JFJochWriter, which a writer-only build compiles, does not gain a gemmi dependency. Only include paths and CMake lists change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 14:05:37 +02:00