v1.0.0-rc.180 #91

Merged

leonarski_f merged 122 commits from rc180 into main 2026-10-27 13:18:43 +01:00

Conversation 0 Commits 122 Files Changed 512

+67489 -2412

122 Commits

Author	SHA1	Message	Date
leonarski_fandClaude Fable 5.1	7782e304ed	RESULT-CHANGING: Refinement: atoms gridded out to a density of 1e-4 e/A^3, not 1e-5 ... The refinement's density calculators (the CPU engine's, the GPU engine's and the map gradient's) cut every atom's density at an absolute 1e-4 e/A^3 of its blurred density instead of gemmi's default 1e-5. The radius, the gather's box and the gradient's box all follow from it, so the structure factors, the gradients and with them the refined model change in the last digits. Molecular replacement's and the validation's own calculators keep gemmi's default. Where this sits against other programs' defaults, which are relative to the atom's own peak density: cctbx resolves phenix.refine's wing_cutoff to 1e-3 of the peak below 5 A (1e-4 above), X-PLOR's ELIM 7 is e^-7 of the peak and SFALL's VDWR 2.5 about 2e-3 of it. For carbon at B + blur of 20-120 A^2 an absolute 1e-4 is 6e-5 to 6e-4 of the peak - still stricter than any of them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-27 11:04:46 +01:00
leonarski_fandClaude Opus 5.5	a645d0a9fd	docs: front page names the reimplemented algorithms and generative AI in the licence section, and says the licences' sharing condition plainly ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37	2026-10-27 11:04:46 +01:00
leonarski_fandClaude Opus 5.5	3006f2bb33	docs: front page footer says the software reimplements existing algorithms, moved to the GPU and made faster, and was developed with the help of generative AI ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37	2026-10-27 10:55:57 +01:00
leonarski_fandClaude Opus 5.5	4fd2bf231d	docs: front page says "structures" rather than "refined structures", names the programs' user operation at the SLS 2.0 MX beamlines, and carries the PSI logo ... The logo is the viewer's and frontend's psi_01.png, linked to www.psi.ch in the header corner and inverted for the dark theme as the viewer does it; the Sphinx extension copies it beside the page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37	2026-10-27 09:23:47 +01:00
leonarski_fandClaude Fable 5.1	546566cff4	Refinement: the gather's pairs built in fixed-shape buffers, and each evaluation replayed as a CUDA graph ... ModelDensityGPU::Compute() no longer reads the pair count back to the host between its launches: the atoms are counted per brick with integer atomics, the counts scanned into the bricks' segments, the segments filled through atomics and then each brick put into model order by rank (one block per brick), which leaves the pairs exactly as the stable radix sort of the atom-major fill left them - the gather sums in the same order, bit for bit. The pair bound of the atoms set is checked on the host at SetAtoms() instead of on the device at every evaluation, and the buffers shrink from four to three ints per pair. With nothing coming back mid-evaluation, RefinementGPUEngine records the X-ray evaluation and the restraint evaluation each as a CUDA graph on its stream (cudaStreamBeginCapture, thread-local mode) and replays it, re-recording only where the shape changes (the atom count, a gradient asked or not, a restraint set re-uploaded). Two graphs, not one, so that the host still fills the restraints' positions while the X-ray half runs, as before. Where a stream cannot be captured the launches are queued as they were. Outputs byte-identical to 037c32d9f on the in-house cytochrome C live run (MTZ arrays, HKL, models, blobs, report). Workstation (RTX 5080): refinement 1.64 -> 1.50 s on that run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-27 05:57:57 +01:00
leonarski_fandClaude Fable 5.1	d7c15c629e	Rotation tail: the final merge made ahead on a spare card, in the groups the run is expected to adopt ... The final merge in the adopted group was the last long pole of the scaling tail after the space-group decision - on a four-card server about two of the step's four and a half seconds on a 16M protein set - and it waited for a decision that, on nearly every run, lands on a group the run could have named much earlier. The canonical pass now clones its engine after the ingest (as the all-observation arm does) and merges ahead, beside the P1 merges, the search and the decisions, in the group a search of the geometry pre-pass's merge predicts; and where the first arm's search names another group, in that group too, on a second card. The prediction is searched on a P1 merge of ALL the pre-pass's observations, made on a copy of its engine on the spare card beside the second pass: the pre-pass's own merge is Lorentz-filtered, and a screw axis is read on the weak reflections the filter discards - searched on that merge, a 4(1) crystal was predicted as its symmorphic group every time. Each merge ahead is taken only where the decision lands on exactly its group (Hall symbol) on the same ingest - every change of basis and re-ingest of the reflections clears that - and otherwise dropped, left to end on its own off the critical path, so the outputs are those of the run without it: no merge depends on the merges before it on an engine (RunArm restores the ingest's corr and G = 1), the per-frame scales are written back from the copy and its partial-scaling choice handed to the engine. The centre-of-symmetry statistics are made in the same thread, and the pre-promotion tNCS/twinning analysis, which the final merge used to hide, starts on a copy of the all-observation merge beside the searches when a merge ahead may be made, taken on the same cell and centring. Spare cards only, decided from the card count before anything starts: card 3 from four cards up, then card 2 (the all-observation arm's, whose merge is done when the second is asked for); on two or three cards the last card, beside the engine there, where one more engine fits its total memory; and only where one more merge fits the host budget. On one card nothing is made ahead, the pre-pass's search is not run, and the run is what it was, byte for byte and in time. Same code live and in the CLI. The search's resolution cut on a P1 merge moves into SearchCut() (a pure move) so the pre-pass's search is set up as the canonical one is. Workstation (one card): outputs and timings identical to the base. Server (4x T4): see the report of the A/B. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-27 05:57:57 +01:00
leonarski_fandClaude Fable 5.1	037c32d9fb	Error model: ISa is reported only where b is the solution of a fit - not beside a clamped a, not when b was held at zero, and on stills under the same rule ... The reported ISa is 1/b of the merge error model var = a*s2 + b^2*<I>^2. Three outputs printed a number that was not the result of any fit: * A bin whose median normalised deviation lies far below the others' - observations that repeat their symmetry mates to the bit, as an input read twice gives - takes the whole fit through its 1/bd2^2 relative weight and pulls both terms towards zero. a stops on its 0.25 bound, b^2 does not, and the remainder's 1/b reads as an I/sigma in the hundreds to millions; the standard-error test let it through on about half of the synthetic pools. Beside a clamped a the b^2 is not the solution of any fit, so it is now left unresolved. * When b had no leverage and was held at zero, b_resolved kept its default and the report printed ISA 0.00 where the log said ISa was not reported. Unresolved now. * The stills fit (MergeOnTheFly::RefineErrorModel) had no resolution test at all, so any tiny positive b^2 went out as ISa; on pools with no systematic error one seed in twelve printed ISa ~1000. It now carries the same test and rugnux reports it the same way. * The --report-resolution refit's ISa (REFRES_ISA) was written without the gate. No a or b changes: the merge weights, outlier band and every value except the resolved flag are untouched, so a merge whose a sits inside its bounds reports exactly what it did. The flag also feeds the per-frame-scale arm choice (PartialScalingWins), which is the one decision that can move - only on a merge whose a is on a bound. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 23:09:38 +01:00
leonarski_fandClaude Fable 5.1	d122118d73	Refinement: the per-evaluation host chain on the threads and in place ... Byte-identical: every pass is element by element in the serial loop's order, or a copy that is no longer made. - The engine's gradients are read in place from its pinned buffers and widened to double on the threads (RefinementGPUEngine::HostGradient/HostRestraintGradient); the positions go up from a pinned vector of the engine's host side; the model's atom pointers are kept between calls (the model is the context's own). - Context: the riders moved and folded on the threads - a parent takes its riders in index order through a CSR list (SetRiderParents), as the serial walk added them. - AtomTarget: the parameters set and the gradient assembled on the threads; the per-atom gradient vectors kept between evaluations (7 MB each on a large model: made anew, their pages fault in every time). - RefinementMinimiser: the sweeps over the parameters (x + alpha d, s, y) on the threads it is given, the dot products the serial chains they were; Direction()'s working vector and a let-go pair's vectors reused. A/B against the frozen combined2 binary, one lock hold each, outputs identical but provenance lines: cytc refinement 1.600 -> 1.548 s (live), 1.724 -> 1.668 (CLI); thau 1.906 -> 1.885, 2.031 -> 2.027; 4nwv (with the commit before) 39.9 -> 35.5 s, the ladder 14.6 -> 12.7, each step 8.3 -> 7.2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 22:48:28 +01:00
leonarski_fandClaude Fable 5.1	b16873036c	Refinement: density records sent from pinned memory, and the L-BFGS direction in fused sweeps ... Two byte-identical cuts to the per-evaluation host chain, which on a large model is twice the GPU's work per evaluation (34 ms against 16 ms of kernels per weight-trial slot, measured in nsys) and so bounds the refinement on one card with three slots, and the whole of it on a card with one slot. - The gather's per-atom density records (38 MB on a 242k-atom model, sent on every B-factor evaluation) live in a PinnedVector and go up by DMA; the boxes go through a pinned staging buffer of the density object. ModelDensityGPU::SetAtoms and RefinementGPUEngine::SetDensities take a pointer and a count. - RefinementMinimiser::Direction takes each pair's dot product in the same sweep that applies the previous pair's update to q: the dot products are serial chains of n additions, and the update's memory traffic now hides under them (20 -> 13 ms per call at 365k parameters). The same operations per element in the same order; the loop-2 coefficient stays written as (a[i] - b) with b on its own, because a[i] - rho * dot contracts into a fused multiply-add and changes the bits (checked bitwise in a standalone harness). - A debug line with the GPU engine's bytes and the budget. 4nwv (open arm, 242k atoms), live flags, one lock hold, loaded box: refinement 42.35 -> 38.40 s; refine.gpu.upload 8.14 -> 3.94 s. All outputs identical but provenance lines. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 22:48:28 +01:00
leonarski_fandClaude Fable 5.1	3189f71e8f	Refinement: the engine polls for its evaluations and the pair count comes back pinned; the slots beyond one per card go to the cards the ladder's long trials leave free ... The GPU engine's waits on its stream (every evaluation, the refits' transforms) and the model density's wait for the (brick, atom) pair count were blocking event and stream synchronisations; where a weight trial has a card to itself, as on a box with a card per trial, the wake-up of the sleeping thread is on the critical path of every evaluation. Both now poll, and the pair count and overflow flag come back into pinned memory instead of a pageable int (a pageable copy back is staged and blocks the host for the whole stream). OwnSlotDevices places the slots beyond one per card on the cards the ladder's long trials (those refining B) leave free, so that on four cards the three B rungs have a card each and the three B-held rungs share the fourth, instead of two cards each carrying a B rung beside a B-held one. The outcome does not depend on the placement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 22:41:43 +01:00
leonarski_fandClaude Opus 5.5	68e567bd02	Tail: the molecular replacement of a prediction or --mr starts beside the analyses; a pass's integrated observations are handed back on a thread of their own ... The search (or the carry of the early search) reads of the merge only its French-Wilson amplitudes, which the analyses make last from a tensor fitted to the merge alone, so - as the first validation of an experimental model already does - it starts before the analyses on a copy given the same amplitudes, where no twin hypothesis' merge is searched beside it and no reference supplies the free set. The pass's integrated observations (ReflectionArena mappings, gigabytes) are freed on a thread of their own as the pass returns, joined when the run goes, instead of before the outputs are written and before the broker publishes the result. Output unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 22:37:44 +01:00
leonarski_fandClaude Opus 5.5	b0c04d0070	broker: rugnux live publishes its result before the run's teardown, which goes on beside the end message ... The rugnux thread put its result in place only after destroying the run and releasing the pinned host memory the engines kept between them, so the end message and the writer waited for that teardown. The result is now moved out first and a promise tells the measurement thread it can send the end message; the run is destroyed and the pinned pool released on the rugnux thread (the same thread, so the same CUDA device as before) beside it, and the measurement thread joins the rugnux thread at the end of the collection, before it reports itself finished. The end message reads only rugnux_result, a value moved out of the run (it already outlived the run), and nothing in the teardown touches it. A run that fails still releases the waiting measurement thread. Measured on the workstation (CLI harness doing the broker's teardown after Run): the run's destruction 0.03-0.04 s and the pinned pool release (440-540 MB) 0.08-0.10 s, so ~0.12 s off last image -> end message on a cytochrome-C-size rotation set. Processing and outputs are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 21:13:17 +01:00
leonarski_fandClaude Fable 5.1	604b270114	ModelDensityGPU: the gather stages each atom's box as the brick sees it, and only what the brick needs of the atom ... The staged record of an atom is now its isotropic Gaussians, occupancy and radius (an anisotropic atom's matrices are read from global memory on that rarer path) and its box as bit masks over the brick's eight points along each axis, with the base and wrap that turn a point into gemmi's unwrapped box index. The inner loop tests a point with three shifts instead of three box offsets and six shared reads, and the shared record is a quarter of its size, so a fourth block fits on a multiprocessor. The density arithmetic, the order over the atoms and the summation are unchanged, so the grid is bit for bit the CPU walk's as before. Outputs are byte-identical; refinement and molecular replacement each gain a few percent on the demo-sized sets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 20:58:10 +01:00
leonarski_fandClaude Fable 5.1	f724bd1117	ModelScaleGPU: the host polls for a scale fit's launches instead of sleeping on them ... A Levenberg-Marquardt step is a few short launches the host then reads the sums of, a hundred or so steps per fit, and the process has every synchronisation sleep until the device raises the wake-up (set_gpu_blocking_sync): for these launches the wake-up costs more than the work. The fit's waits poll the stream instead. Outputs are byte-identical; the refinement's scale refit counter drops 20-27% and the molecular replacement's scale fits gain a little on the demo-sized sets. Summing the block partials on the host instead of in a kernel was tried alongside and dropped: the coarse solvent grid of 88 fits made it a megabyte of read-back per step, slower than the kernel. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 20:58:10 +01:00
leonarski_fandClaude Fable 5.1	128e3bb9ae	RefinementGPU: the symmetry phases from a table instead of sincospi in the compose and scatter kernels ... exp(2 pi i h.t) depends only on the integer h.t (in units of 1/den), so the engine tabulates it once over the range the rows and operators make, with the same op_phase() at the same double argument - the same bits. The compose kernel reads the table at h.t; the scatter terms carry their table entry, and their row directly, so neither hot kernel divides: the rotations go to the device in whole units (gemmi's are den times an integer matrix), the term's row no longer has to be split from its operator. The table is a few thousand entries, the scatter terms one int wider. Outputs are byte-identical; the X-ray evaluation counter drops 7-10% on the demo-sized sets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 20:58:10 +01:00
leonarski_fandClaude Fable 5.1	dfb1536fe2	Sweep ends, overloads and ice flags: one walk over the reflections instead of two ... Before the merge, the integrated reflections were walked twice - once per frame for what the blank sweep ends are decided on, once more for the overload counts and the ice flags - and each walk reads the whole reflection set from memory. The frame's sweep-end tally is now a function of its own (TallySweepEnd), and the merge step makes it in the same per-frame walk that counts the overloads and flags the ice, then adds the counts up over the frames still in, so the trimmed and left-out frames count nothing, as before. Per frame the sums are the same sums in the same order; the totals are integers. TrimBlankSweepEnds keeps its one-argument form for the callers that walk the frames for nothing else. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 20:58:10 +01:00
leonarski_fandClaude Fable 5.1	093ac1612c	Space-group search and frame disposition: three host-side waits shortened ... The first arm's space-group search now runs beside the wait for the all-observation merge and the search started on it, instead of after both. Classifying a reflection for a candidate group asks gemmi twice which operators fix it (once for the absence, once for the epsilon factor); one pass over the operators answers both, with the absence read off a fixing operator's own translation, which is what gemmi's test reduces to once the centring has been checked. The delta-CC1/2 frame disposition builds its per-reflection totals on all threads, each taking a range of reflections and walking every observation so the sums keep their order, and measures up to sixteen candidate chains side by side with a worker's scratch made on first use. Same numbers, same bits; only the waiting changed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 20:58:10 +01:00
leonarski_fandClaude Fable 5.1	bbcae8ae7f	RotationScaleMerge: the resident ingest uploads the reflections once ... The device ingest staged every reflection to the card twice - once for what the sort and the cut read, once again for the observations the cut kept - and the second staging was a full copy of the reflections into pinned memory and over the bus. Where the whole set of reflections and the ingest's own peak beside it fit a quarter of the card (decided on the card's total memory, so the answer depends on the card and not on the moment), the first upload now goes into one resident copy, and the observations are taken from it in a single extract; the second staging stays as the fallback where the copy does not fit, and where the card reads host memory in place. The extract kernel also hands over the predicted positions, which the host had gathered itself from a downloaded index array, and the two pinned staging buffers are kept across the ingest instead of being allocated per staging. Same observations, same order, same bits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 20:58:10 +01:00
leonarski_fandClaude Fable 5.1	00f8c8ba4b	Refinement: the final ladder takes the rungs the step from the best state tried already; the longest trials to the slots first ... The trajectory ends where a step gains nothing, and that step ran from the best state at the weights the final ladder then tries from the same state: on a positional step every rung, on a B step the middle one. A trial of the same weights from the same model is the same computation (RunTrials: the outcome is the same on any slot), so the final takes those outcomes from the step and runs only the rungs it has not got. The maps' Refit() then starts from the fit the main slot would hold had it run its rung (Context::Restore of that rung's fit) - the solvent grid and the likelihood's D it starts from otherwise depend on which trial the main slot ran last, and that showed as last-bit differences in the map coefficients. RunTrials gives the trials that refine B, the longest, to the slots first: on a machine whose cards hold a slot each the ladder's two B rungs no longer share one card while a positional rung has one of its own. Byte-identical (p.mtz, p.hkl, p.cif, p_model, p_refined_model, p_refined_maps.mtz, p_blobs.json, report minus timing) on three rotation sets run live-path, and on two of them in --mode refine. Refinement wall on one RTX 5080: 1.86 -> 1.71 s, 2.42 -> 2.17 s, 50.0 -> 40.8 s (one stage of the trajectory each). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 19:26:48 +01:00
leonarski_fandClaude Opus 5.5	e4d64e67c9	rugnux: French-Wilson takes the anisotropy tensor from the merge alone, the rest of the analysis is joined after molecular replacement ... AnisotropyPriorHKL is the same tensor, bit for bit, as AnisotropyTensorHKL(AnalyzeAnisotropy(...)), so the amplitudes no longer wait for the unmerged-observation part of the analysis (ScaledObservations): the model work starts at once and the analysis, its text and the merge statistics are taken before the rigid-body validation or after molecular replacement. The analysis takes its own copy of the space group, which the enantiomorph relabelling changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 18:23:43 +01:00
leonarski_fandClaude Opus 5.5	5338e1b535	Centre-of-symmetry statistics and anisotropy: hashed lookups and the simulation and cell assignment on all threads, output unchanged ... AnalyzeCentreOfSymmetry ran on one thread between the adopted group's merge and the analyses: its orbit fold and L-test partner lookup were std::map (now hashed - the orbits are sorted by a total order right after, and the lookups only test membership), and its 20 simulated populations took their log/cos per reflection serially. The generator's draws are now taken in the same order on one thread and turned into intensities on all of them (the Box-Muller operands in the order GCC evaluated them; bit-exact against the old function on five groups at 1/4/32 threads). AnalyzeAnisotropy's three BuildCells and the observation cells of the systematic floor find each copy's nearest of 60 directions on all threads and fill the cells in the old order; the observation sort is a ParallelSort (index-unique pairs, a total order). AnisotropyRunInfo carries the worker count. Live path, workstation, 3 A/B pairs, outputs identical (mtz/hkl/cif/model/maps/report): thau centre stats 0.18 -> 0.06 s, analyses 0.57 -> 0.43 s; cytc 0.04 -> 0.016 s, 0.37 -> 0.32 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 18:23:27 +01:00
leonarski_fandClaude Opus 5.5	7c49fefb54	Tail: the cell e.s.d.s and the unmerged MTZ table beside the refinement; the refined difference and anomalous maps side by side ... The cell e.s.d.s and the unmerged MTZ read only the integrated observations and the group, both final once the model has been read, and the refinement reads neither: both now start before the refinement instead of after it. In MakeMaps the difference map is made on its own thread while the anomalous map is made. Outputs identical (MTZ data, CIF/HKL/INS, refined model, maps, blobs, report minus provenance) on the thaumatin and cytochrome C live runs. Live path without the unmerged MTZ (as the broker runs it), step Refinement: thaumatin 2.76 -> 2.40 s, cytochrome C 1.99 -> 1.93 s. With the unmerged MTZ (CLI), Refinement + Writing files: 3.21 -> 2.67 s and 3.33 -> 2.80 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 18:23:27 +01:00
leonarski_f	067f3a7d7b	docs landing: Rugnux card names synchrotron rotation and XFEL serial data and links the public test-data lists; footer motto removed ... The Rugnux card says in passing that it takes a synchrotron rotation sweep or XFEL serial stills, and ends with one sentence pointing at the two pages that list the public datasets it is validated on (External test data: over 200 rotation sets, 207 in the open battery manifest; External test data: serial crystallography: eleven CXIDB entries). The "two mountains, two railways" line at the foot of the page is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 03e4431836a2f4ba816e6d3cdd57cc6ada87b95d)	2026-10-26 16:59:43 +01:00
leonarski_f	2736f4c85e	Docs landing page: the hero is a live Rugnux run - a frame, the placed model in its density, the timeline ... The front page's hero image is now the cytochrome C collection at SLS X10SA as the broker's Rugnux live page ran it: a detector frame on the page's indigo beside the AlphaFold model placed in its electron density, over the two-lane timeline of the live analysis (the 18 s collection, the image processing alongside it, scaling, merging and molecular replacement 4.9 s after the last image). One WebP per theme (hero_live_light.webp, hero_live_dark.webp), shown by the same rules that pick the theme's colour tokens, so the toggle and the system preference both switch it; the hero's art column widens to 560 px on desktop (360 px on a phone). The lysozyme frame and its generator go; the Sphinx extension copies the two new files into the site. Made by docs/landing/hero_live/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 5f2592390d9ec39428198df8cd8e18ad60167a79)	2026-10-26 16:54:39 +01:00
leonarski_f	2e296e6bed	jfjoch_writer: the live refinement's model, maps and blobs are written beside the master file ... In broker live mode the refined model, its map coefficients and the difference-map blobs stayed in the broker for the viewer routes and were lost at the next run. They now go to the writer in the END message as the placed model and maps already do - EndMessage::refined_model (PlacedModelMessage), refined_map_coefficients (MtzTable) and blobs (DensityBlobsMessage, one vector per field) - and the writer writes <prefix>_refined_model.cif (+ .pdb), <prefix>_refined_maps.mtz and <prefix>_blobs.json, the names the CLI gives them. The blobs' JSON is made by one function (DensityBlobsJson) that the CLI now writes through as well, so the two files are byte-identical. Optional CBOR fields, no version bump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c65220d96a96d2b1897f3c6632c94f9379b670a3)	2026-10-26 15:50:40 +01:00
leonarski_f	2461d67e14	NXmx: the unit cell's e.s.d.s are saved as /entry/sample/unit_cell_esd ... Rugnux measured the cell's e.s.d.s (CellEsd) but they only reached the mmCIF's _cell.*_esd, the .ins's ZERR and the report. They now travel in the END message (EndMessage::unit_cell_esd, CBOR key unit_cell_esd, same layout as unit_cell), are written beside the cell in NXsample as unit_cell_esd (float[6]: a b c alpha beta gamma, A and degrees) and are read back into JFJochReaderDataset::unit_cell_esd. rugnux's _process.h5 and, through AddToEndMessage, the broker's live master file get them. Rotation only; absent on stills and where nothing was measured. Optional CBOR field, no version bump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6902b5637aa92c27565f22bcf3fe11b5d2536038)	2026-10-26 15:50:40 +01:00
leonarski_fandClaude Opus 5.5	d50b3d7034	CI: the documentation goes to rugnux/gitea-pages only when a release is created ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 15:46:17 +01:00
leonarski_fandClaude Opus 5.5	9830fe9dd0	broker: redoc regenerated from the merged API (blob distance optional, live refinement on the CPU too) ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 15:39:21 +01:00
leonarski_f	825b88c19d	CHANGELOG: 1.0.0-rc.180 entry ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 339c6cbaa4177964e5eb06d0878efb83d1da2dbc)	2026-10-26 15:39:07 +01:00
leonarski_f	13bd9db55c	Refinement: a B factor is held at its bound by the whole target's gradient, restraints included ... The held test read the X-ray gradient alone. At the floor, an atom whose X-ray term pushed outward but whose restraints pulled it back in more strongly was held fixed against the total target, and one whose restraints pushed it out was handed a gradient along the direction Set() clamps away - a flat direction for L-BFGS, failed line searches and s.y <= 0 pairs. On the 14 parity sets (--mode refine, GPU) the refined models are md5-identical; on the four 5f6m MD starts R-free 0.1707/0.1676/0.1720/0.1669 -> 0.1707/0.1690/0.1719/0.1668 (median 0.1692 -> 0.1699). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 2f4b0b5ea54fdac54de7ee6c5a1476d2fc019fd8)	2026-10-26 15:38:59 +01:00
leonarski_f	2be88e11e3	Refinement restraints: contacts with lattice translates are kept, and an atom's own symmetry contact once ... Contacts() dropped every pair found in image 0 with a nonzero lattice shift ("the identity term covers the contact"), but that pair is a contact with a translated copy of j, not with j itself: in P1 no intermolecular repulsion or B-contact restraint was made at all, and in every group the contacts along a, b and c were missing (MeasureGeometry's clash count with them). Such a pair is now kept like a symmetry mate's, from the lower atom of the two. An atom against its own image x -> T x is also met through T^-1 at the same distance (another operator, or the same one at another lattice shift for a screw), and both were kept: atoms near a 3-, 4- or 6-fold or a screw axis were repelled twice. For i == j the image first in fractional coordinates of T x and T^-1 x is kept; an involution is kept once as before. Test: in P1 two waters 2.4 A apart across a cell face get one repulsion and one B contact (none before); a water 1.4 A from a 3-fold axis gets one repulsion with its images (two before). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0d6e4d0491e655a49d59b709e1398843506d162a)	2026-10-26 15:38:59 +01:00
leonarski_f	470fc4e479	ffbidx (vendored): INDEXER_GPU_DEVICE is not read, the indexer stays on the thread's pinned GPU ... The environment variable overrode the current device that ScopedGpuPin sets, so a stray value moved an indexing thread to another card. Recorded in the file header as a local modification, together with the deterministic tie-break of c982ba4f6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 89acdeef939ee82cf15ee9c5fb52a016fb1441f8)	2026-10-26 15:38:59 +01:00
leonarski_f	257bf2fe2b	Refinement: trial slots counted with their scale pool and both restraint sets; on the CPU no more than -N ... GPU: a slot was packed onto a card by its engine's bytes alone. The per-slot estimate now adds the slot's ModelScaleGPUPool with the robust fit's buffers (ModelScaleGPUPool::DeviceBytes(n, 1, true)), and RefinementGPUEngine::DeviceBytes counts the two restraint sets SetRestraints() holds, not one. The 80%-of-card budget is unchanged, so where these do not fit fewer slots are packed instead of an allocation failing inside a running trial. CPU: OwnSlotDevices made a slot per wanted trial whatever the thread count, so -N 1 ran six trial threads with six models, topologies and engines in memory. The main slot and those beside it are now at most -N. Side by side equals sequential, so no result changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3c04c01158266c267f8e5ffaaaa98d6d8fd2a747)	2026-10-26 15:38:58 +01:00
leonarski_f	ac58921445	rugnux live: a score that is not finite is left out of the status, not sent as null ... tf_z, rf_z, mr_d_min, the placed model's R-factors, CC, map and anomalous sigmas, the rigid body's move, the refinement's R-factors and rmsds, and the headline d_min are set only where finite (llg_z and null_sigma already were). nlohmann writes NaN/inf as null, which the page's "!== undefined" checks let through to .toFixed(), and which the zod schema rejects for the required blob distance - a blob with no atom within 12 A has an infinite one. That distance is now optional (jfjoch_api.yaml; C++ model and frontend client regenerated), absent then. The model cards test for != null, so a null that arrives anyway does not take the page down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 62ff026aa507e5bda1daa5aa079f32fb5c21f16e)	2026-10-26 15:38:58 +01:00
leonarski_f	3d382ce0a0	Refinement: an exception in a trial slot or in RefineModel no longer ends the run ... A slot builder thread that throws (gemmi's topology or hydrogen placement, bad_alloc) would call std::terminate - rugnux or the broker gone. The slot is now left out as one the card cannot take is, with a warning, and MakeOwnSlot's GPU engine catches std::exception, not only JFJochException. Both callers of RefineModel (after the merge, and --mode refine) catch std::exception: the refinement is reported NOT_PERFORMED with "refinement failed: <what>" and the run writes everything else (cell e.s.d.s, unmerged MTZ, report), as the refinement is processing-inert. Test: --mode refine on a cell too large for the bulk-solvent grid comes back with REFINE= NOT_PERFORMED and the reason, not an exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dc4e99fe4b3da2a9a3fa4290b5f70bfcc8c7bdb8)	2026-10-26 15:38:58 +01:00
leonarski_f	0143d57eb0	CellEsdTest: synthetic cells in place of cells that looked measured ... The monoclinic cell of the .ins/mmCIF and fit tests is now 10/7/12, beta 100, and the orthorhombic, hexagonal and tetragonal cells of the sampling tests are rounded synthetic ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6f964bc0de0ca842f374e6aed83c8848e410173b)	2026-10-26 15:38:58 +01:00
leonarski_f	48bf9ebe3a	CellEsd: PI from JFJochMath.h instead of M_PI, and the Herbstein (2000) credit at the block e.s.d.s ... M_PI is not defined by MSVC without _USE_MATH_DEFINES, and CellEsd.cpp is reachable from the viewer build. The citation, so far only in CellEsd.h, now also sits at the algorithm it credits. The other files new in rc180 under common/, image_analysis/, reader/, writer/, rugnux/ and viewer/ use no M_* constants or POSIX-only calls (grep of the rc180 diff). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a019bc5fd2084907af9379068af3265ffe6c9ef5)	2026-10-26 15:38:58 +01:00
leonarski_f	1b48f3873f	docs: front page and quick start - Rugnux goes on to a refined structure with a model ... Landing page (title, hero, lede, Rugnux card), the documentation home's Rugnux card and the opening of RUGNUX.md now say what the program does: from diffraction images to merged data and, with a model, on to a refined structure and maps. RUGNUX.md Quick start is rebuilt around how a run is started: rotation (master file, or the first frame of a CBF/marCCD/SMV sweep), serial stills (several masters or all of a dataset's files by glob), serial stills with -C/-S and --crystfel-geometry (wavelength/distance flags only where the files state neither), and with a model (positional or --model): placement by rigid body or MR, refinement by default (--no-refine), maps and blobs, and that the model changes no processing decision except the indexing choice on serial stills with -C/-S. It says plainly that the defaults are meant to be the answer, and lists the output files with and without a model. The tutorial's stills section drops -X ffbidx, -z and the resolution limit from its first example and shows the same three stills commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit bfca846332037688364b9d2d1c4250315f09410d)	2026-10-26 15:23:07 +01:00
leonarski_f	cd0b994027	API: rugnux_live_refinement no longer says live refinement needs a GPU ... Live refinement also runs on a broker without a GPU, on the CPU and slower, and the run's deadline or cancel can cut it. Description only; regenerated broker/gen (Rugnux_live_refinement.h), the frontend client (types.gen.ts, zod.gen.ts), broker/redoc-static.html and the python client doc page - each differs from before only in this description. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 53d06c6e221fa1cf776822965560fb990cc69c73)	2026-10-26 15:23:07 +01:00
leonarski_fandClaude Opus 5.5	4b9809d9fa	docs: BATTERY_REPORT.md removed - a stale battery report in no toctree ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 15:16:27 +01:00
leonarski_f	66cc0bd563	docs: acknowledge Herbstein (2000) for the unit-cell e.s.d.s ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 795c98c9d6b5db0cde4d6e0b8c7b0dad9c99775f)	2026-10-26 14:50:42 +01:00
leonarski_f	af28cf430b	docs: technical pages caught up with the stills rocking curve, beam-shadow integration and refinement ... INTEGRATION: the stills prediction window and partiality from the run's measured rocking curve (a d* table, kept in _process.h5 for --mode scale), the model-error term in the per-crystal fits, --min-partiality defaults, the adaptive radius on stills (background-ring reach). IMAGE: reflections reaching into the beam-stop shadow are not integrated; no-data pixels masked as dead, not as shadow. INDEXING: the rotation-scale walk's refusal and refit; the stills geometry refinement on the given cell's crystal form only. DECISIONS: the model is refined after validation by default. CPU_DATA_ANALYSIS: titles in the references that lacked them. GPU: the still prediction kernel follows the rocking-curve window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 5ae198b566b17de259fd0f0d6c8a0f8ea18eaa05)	2026-10-26 14:50:42 +01:00
leonarski_f	7b1bc3a3ea	docs: report headline keys, model section for every way a model is given, still battery ... RUGNUX_REPORT: a table of the headline keys (checked against ResultReport.cpp); section 5 appears for --model, --mr or a model among the inputs and holds the MR and REFINE_ keys; a --mode refine report has one section; stills ambiguity resolved by the crystals' own cells first; the developer-only CELL_ESD_OBSERVATIONS / CELL_DSPACING_FIT; the "nothing here is refined" paragraph points at REFINE_*. BENCHMARKING: the still battery, the refined R-free beside the placement one, the Refinement and Doubled-cell steps in the timing table. EXTERNAL_TEST_DATA_STILLS: the gain is measured, not passed; how the battery runs these sets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit e9f5e3b63fb6208d6bd5984c1ecd0eb0da7c29ab)	2026-10-26 14:50:42 +01:00
leonarski_f	07c8b2ddd4	docs: the Rugnux card and the naming table mention refinement and live collections ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f8d07397288137dd5e3a0e5c101f2036d51228bd)	2026-10-26 14:50:42 +01:00
leonarski_f	c3c5a32a0d	docs: rugnux getting-started pages - refinement, the full output list, a troubleshooting note ... RUGNUX: a model run also refines the model it placed; the model can be given as an input; only the unmerged MTZ is written by default; the _process.h5 is not a default output; a short "when a run fails" note. TUTORIAL: the .ins, the refined model/maps/blobs, the log and _modules.h5 in the output list; the report is not written with -o ""; stills: -X Auto picks ffbidx with a cell, a run over several files. FORMATS: the XFEL stills formats and model/reference inputs told by content. CALIBRATION: -C and -S as the calibrant. INSTALL: the CPU path runs every indexer and the refinement. MODEL_AND_REFERENCE: the refined model is written as .pdb too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0062d72b95e996438ddc65dc343b526c84daa7fb)	2026-10-26 14:50:42 +01:00
leonarski_f	8edd0e5f3e	docs: rugnux options, model and export pages brought in line with the usage text ... RUGNUX_ADVANCED: --simulate-live/--fast-live rows, -e exclusive, stills behaviour of --beam-center-check, -S, --min-partiality (0.5), --capture-uncertainty, --adaptive-integration-radius (on for stills too), --detector-distance/--rot1/--rot2/--no-refine-tilt; --background-radial auto as it decides now; --refine-modules syntax; _cell.*_esd. MODEL_AND_REFERENCE: the default refinement of a placed model in the summary, the decision table and the step list; --mode scale does not refine. INTEGRATION: the .ins written beside every .hkl (ZERR on rotation). VIEWER: rugnux writes _process.h5 only with --no-merge/--write-process-h5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 51d2564356ba46bda8730c855ff8d3ee7c23d605)	2026-10-26 14:50:42 +01:00
leonarski_f	d8c14046b2	rugnux: unit-cell e.s.d.s - ZERR in a new SHELX .ins beside the .hkl, _cell.*_esd in the mmCIF, CELL_ESD_* in the report ... Rotation runs now estimate the standard uncertainties of the cell. Every strong integrated observation (I > 3 sigma, unclipped, unsaturated, off ice rings) gives |q| = 1/d at its observed centroid, which depends on the detector geometry only; q^2 = h^T G* h is linear in the reciprocal metric, so the components of G* the point group leaves free (projected from the group's rotation parts, any setting) are fitted by weighted linear least squares with 4-sigma rejection, and the GoF-scaled covariance is carried analytically onto a, b, c, alpha, beta, gamma (ties give equal e.s.d.s, fixed angles zero). Least-squares cell e.s.d.s are known to be too small (Herbstein 2000), so the same fit is made on 8 blocks of consecutive frames and their scatter / sqrt(8) is the second estimate; the written e.s.d. is the larger of the two per parameter. Stills get none. Outputs: <prefix>.ins next to every .hkl (TITL, CELL at the wavelength, ZERR with Z = order of the group, LATT/SYMM from the space group - positive LATT only with -1 at the origin - a placeholder organic SFAC/UNIT at 18 A^3 per non-H atom, HKLF 4, END); the mmCIF gains _cell.*_esd ("." where fixed) and _cell_measurement.reflns_used/theta_min/theta_max, and writes the cell to 4/3 decimals; report keys CELL_ESD, CELL_ESD_LSQ, CELL_ESD_BLOCKS, CELL_ESD_BLOCK_COUNT, CELL_ESD_REFLECTIONS, CELL_ESD_THETA_RANGE_DEG (+ dev CELL_ESD_OBSERVATIONS, CELL_DSPACING_FIT). On 13 small-molecule sets the block scatter is 10-40x the least-squares e.s.d. and is what gets written; it lands within ~0.3-5x of XDS's CORRECT.LP e.s.d.s. SHELXT solves from the generated .ins and SHELXL carries ZERR into its CIF su. Costs 12 ms on a small molecule, 0.16 s on 690k lysozyme observations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit e74a3be76261fe1b6c0ff7b80c21afc4768c46cd)	2026-10-26 14:33:38 +01:00
leonarski_f	22debb1be6	docs: the stills rocking-curve datasets in the _process.h5 master ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 1195b9be6e0ca66774774f6de4867180b7bd0a47)	2026-10-26 14:09:32 +01:00
leonarski_f	e6b550fe42	Tests: the stills rocking width - the window at a reflection's resolution, _process.h5 round trip ... StillsPartialityRefine_MeasureRunWidth_WindowAtReflectionResolution, JFJochReader_StillsRunWidth. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dafb8d1bf996d1000741eecf019aee780dd91ebe)	2026-10-26 14:09:32 +01:00
leonarski_f	8a123e794d	Stills: the run's rocking curve goes to _process.h5 and --mode scale merges with it; MeasureRunWidth takes the window at each reflection's resolution ... --mode scale re-measured the rocking width on the stored reflections, so on any file whose frames were predicted through the run's curve it merged with a different curve than the mx run did. The curve the post-refinement used (formula, table, radial scale) is now carried in the EndMessage (StillsRunWidth) and written to the master: /entry/MX/stillsRockingWidthFormula [gamma0 1/A, gamma_e rad, bandwidth sigma], /entry/MX/stillsRadialScale, /entry/MX/stillsRockingWidthTableDstar and /entry/MX/stillsRockingWidthTable (1/A). The reader returns it, and --mode scale passes it to the post-refinement. A file without it is re-measured as before. MeasureRunWidth grouped each frame's reflections by the frame's window, taken as the largest excitation error it predicted. With a window that follows sigma(d*) that is the high-resolution window, applied to the frame's low-resolution reflections too. Given the curve the frames were predicted through, the window is now that curve at the reflection's own resolution; without one, the frame's largest excitation error as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 992343d0206489991880bb927efc79131134a1d9)	2026-10-26 14:09:32 +01:00
leonarski_f	00662c4110	Refinement: common cofactors and modified RNA nucleotides in the embedded dictionary; heme iron restrained from LINK records ... The embedded CCP4 Monomer Library subset grows from 62 to 84 compounds: - the modified tRNA/rRNA nucleotides PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, M2G, 7MG, 2MG, 5MU (RNA group, joined by the library's backbone link "p"); - the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM, SAH, and dioxygen (OXY). CMO is left out: its dictionary has C-O at 1.25 A, where bound CO is near 1.15 A. links_and_mods.cif gains the links these compounds use: HEC-CYS1/HEC-CYS2 (heme C thioethers), CYS-HEMB/CYS-HEMC, CYS-FAD and HIS-FAD1/2, with their modifications. AddMetalLinks now takes a metal atom of any residue, not only a one-atom ion, so a heme iron's LINK records to His NE2 / Met SD / water become bonds at the library's metal-site distances; the site's coordination counts the metal's bonds in its own dictionary (a heme's four pyrrole N) as well as the records. A metal within a ligand is also a metal for the repulsions (1.9 A to N, O, S) and its dictionary bonds are kept with the metal bonds. Note HEC.cif's own Fe-N(pyrrole) ideal is 1.90 A (HEM: 2.04 A), so a cytochrome c reports a metal-bond rmsd near 0.1 A. Test RefinementRestraints_deposited_ligands builds the topology of eleven deposited models cut down to the ligand and its surroundings (tests/test_data/ligands, 66 KB, from PDB 1EHZ, 8BKH, 6K9I, 2W3V, 5VML, 6F3P, 8CON, 3TK0, 7TCD, 6W4H, 5EPE): every ligand restrained, bond rmsd < 0.03 A, the RNA backbone bonded to each modified nucleotide, the heme Fe-His bonds, and the HEC thioethers. rugnux --mode refine on the full entries against their deposited structure factors: R-free within +-0.003 of the build where these were held fixed, ligand bond rmsd vs the dictionary from 0.004-0.058 A down to 0.002-0.042 A. rugnux grows by 425 KiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f4d407c8de6cb462320edb21a70665350e436d28)	2026-10-26 13:56:29 +01:00
leonarski_fandClaude Opus 5.5	d90bc68085	CI: the documentation also pushed to the gitea-pages branch of rugnux/gitea-pages, on every push for now (token test) ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 11:39:10 +01:00
leonarski_fandClaude Opus 5.5	771e669f75	rugnux usage: -S on serial stills also settles the indexing ambiguity from the data ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 11:16:34 +01:00
leonarski_f	6491cc5a7e	docs: serial-stills indexing ambiguity described the same way everywhere ... Two mechanisms, as the code has them: with -S and no reference or model, the crystals made consistent from the data (Brehm & Diederichs objective, cell-metric anchors); with -z or --model (-C/-S), per image against the reference, crystals their own cell decides keeping that indexing, the data-only result as a check. Removes the statements that the ambiguity needs a reference or model (tutorial, quick start, overview, §10.9), adds the anchoring to the reference/model rows, the "no -S" case (mixed merge, can pass for higher symmetry, warning), and the report keys in the reference case (AMBIGUITY_DATA_DISAGREES in place of AMBIGUITY_REINDEXED, AMBIGUITY_TIME_S, none in --mode scale). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8db1d3ecb97dba90471c4db238aff632d11bc3e8)	2026-10-26 11:15:54 +01:00
leonarski_f	0cc3129650	docs: refinement's GPU engine in GPU data analysis, refinement in the pipeline figures ... GPU_DATA_ANALYSIS.md: the refinement section at the level of the other engines - density gather, cuFFT r2c/c2r, compose + likelihood target, CSR adjoint, warp-per-atom gradient, restraints and slot gather, fine mask and robust scale fit (on-device median), reductions and determinism, stream priority and blocking-sync waits, trial slots over cards, DeviceBytes budget, CPU fallback, no Metal; indicative RTX 5080 timings. Pipeline figures: a dashed Refinement box after Structure & map on both, the model branch labelled as deciding nothing; the stills row gains the indexing-ambiguity step before the first merge. RUGNUX_OVERVIEW.md: captions, a refinement paragraph, refined outputs under Write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8645fb2aca1abbbddec29cdfa99056471f8f38a5)	2026-10-26 11:15:54 +01:00
leonarski_f	e406eb17ea	docs: refinement chapter (§15), REFINE_ report keys, acknowledgements and in-source credits ... - New docs/CPU_DATA_ANALYSIS_REFINEMENT.md (§15): the ML amplitude target and its FFT gradient, scale/robust fit/fine solvent mask, D and the variance (work/free), the embedded Monomer Library subset and topology, geometry/metal/KL-B restraints, the minimiser, the weight trials, refined maps, blobs, GPU/CPU, and a "Not yet" list; with its own References. Linked from TECHNICAL.md and CPU_DATA_ANALYSIS.md; §14 no longer says the structure is never refined. - RUGNUX_REPORT.md: a Refinement (REFINE_*) section - every REFINE_ and REFINE_SCORE_ key, the --mode refine input keys and the _blobs.json fields. - RUGNUX_MODEL_AND_REFERENCE.md: "What it does not do yet"; RUGNUX_ADVANCED.md: --model mentions the refinement, --no-refine as in the usage, a --mode refine options table; GPU_DATA_ANALYSIS.md: the refinement's CUDA engines and the missing Metal path. - ACKNOWLEDGEMENT.md: Brunger, Karplus & Petsko 1989 (gradient-balanced starting weight), Nocedal 1980 (two-loop recursion), Charbonnier et al. 1997 (pseudo-Huber), Jiang & Brunger 1994 (flat solvent mask), Kullback & Leibler 1951; all DOIs checked against Crossref. - In-source credit lines: the weight balancing (replacing a "Brunger (1992)-style" line that named the R-free paper), Nocedal 1980, Charbonnier 1997, Jiang & Brunger 1994, Emsley et al. 2010 at FindDensityBlobs, Read 1986 at the refined maps. Comments only. - docs/THIRD_PARTY_NOTICES.md regenerated from the root file with update_version.sh's own step (it lacked the CCP4 Monomer Library row). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 09375dba351bae0cc73e9a6f0f79455dc92b3a10)	2026-10-26 11:07:43 +01:00
leonarski_f	f960bd7a80	Docs landing page: a real diffraction frame in the hero, and the command with a model ... The generated spot-and-ring drawing is replaced by one 0.2 deg rotation frame of the in-house lysozyme standard (EIGER2 16M, SLS X10SA): Bragg spots on their lunes, the beam stop and its arm, the module gaps. docs/landing/hero_frame.py makes the 920 px WebP (67 KB) from the master file - crop about the beam centre, log tone map from the frame's own background, block mean+max downsampling - and is run by hand; the WebP is committed and copied by the landing-page step. The dark theme shows it inverted through a CSS filter. The hero's command line drops its explanatory sentence and gains the second form, rugnux <diffraction image file> <CIF/PDB model file>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ff26000bfe7446cb4fda739c3b72596f8a8b7047)	2026-10-26 10:53:25 +01:00
leonarski_f	4dbcb35708	Stills partiality: the rocking width measured in shells of equal width in d*, used as a table ... MeasureRunWidth measured the width in eight shells of equal count in d*^2 and fitted the three-term formula gamma0^2 + (gamma_e d*)^2 + (bw |q_z|)^2 to them. On runs predicting ~1000 reflections a frame the equal-count shells all sat at 3.8-1.8 A, so the low-resolution width was the formula's extrapolation, and the fit's form flipped with the number of frames (to a pure bandwidth at 50k frames). The same window-group moment is now also measured in eight shells of equal width in d*, made non-decreasing (pool adjacent violators, weighted by each shell's summed signal) and kept in RunWidth as a table that the partiality interpolates linearly in d*. The formula is still fitted on the equal-count shells and still sets the prediction window. With fewer than three measurable shells the formula is used as before. Still battery, 12 non-float sets, together with the sigma-following prediction window (the previous commit; run 20261026-1021_1b28d6_still-part-comb against rc180 20261026-0857_3882b0_still-part-base1, 0-cycle REFMAC R-free vs deposited, both --no-refine): 6 better, 3 unchanged, 3 worse (by 0.002), sum of changes -0.027 (the window alone: -0.007); largest gains pal90 -0.014, alvra103_int16 -0.011, lcls222 -0.005; largest losses euxfel152 +0.005, alvra213_int16 +0.005 (+0.010 with the window alone), euxfel83 +0.003. Win/draw/loss against the depositions unchanged at 9/2/1. Rotation: p.hkl byte-identical on 5reo, lyso_x06da_ref and thau_x10sa_16keV. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit de779bb10b1813eeae625e3994b1ca73d6079d7a)	2026-10-26 10:50:52 +01:00
leonarski_f	0829b9a72d	Stills prediction: every frame predicts through two rocking widths of the run, measured first and centred on its radial scale ... Each still predicted through |dist_ewald| <= 2x its own profile radius (the RMS excitation error of its indexed spots): one width at every resolution, differing eightfold between frames. Against the run's measured rocking width that was 5-8 sigma at low resolution on one SACLA run (59 % of the 40-8 A predictions empty) and below one sigma at high resolution on a crowded LCLS run (half of the curve never integrated). Now the stills image pass first integrates a sample (a tenth of the run, at least 2000 frames) the old way, scales it against its own merge and measures the run's rocking curve on it - centre (MeasureRadialScale) and width (MeasureRunWidth), as the merge measures them - then predicts every frame through 2 sigma(d) of that curve, centred on the reciprocal lattice scaled by the radial scale (BraggPredictionSettings::window_*, CPU and CUDA). The post-refinement uses the same curve instead of measuring it again: re-measured through windows that follow it, the constant term falls to ~0 and the result swung by up to +-0.015 per run. Still battery, 12 non-float sets, against the st-partial baseline (0-cycle REFMAC R-free vs deposited): 9 better, 3 worse, sum of changes -0.013; the off-centre run -0.012, worst +0.008 on one JUNGFRAU run (its sample measures a different width form than the full run). Integrated reflections 0.8-1.7x, battery wall +13 % on a loaded box. Rotation untouched (window fields zero unless set); p.hkl byte-identical on three rotation sets. Tests: BraggPrediction_WindowFollowsResolution, BraggPrediction_CPU_GPU_consistency_window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 1b28d671d780bc88525ceac5120017963d937bbc)	2026-10-26 10:50:52 +01:00
leonarski_f	13da79b788	Refinement GPU: the atom gradient kernel walks its box without integer division ... Bitwise. One warp per atom, each lane taking the box's points t = lane, lane + 32, ...: every point cost three integer divisions/modulos to turn t into (u, v, w) and three more (imod) to wrap them into the grid. The lane now steps (iu, iv, iw) on by 32 points with adds and carries, and wraps from the box's first point, wrapped once per atom (a box is narrower than the cell, so one conditional subtraction per axis). The same points in the same order per lane, the same arithmetic. atom_gradient_kernel (nsys, side by side): 5reo 426 -> 237 us, 5f6m 887 -> 523, 9yl4 1436 -> 802, 7d98 692 -> 388 us per launch. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Wall: 5reo 1.43 -> 1.35, 5f6m 2.20 -> 2.00, 9yl4 6.88 -> 6.82, 7d98 3.58 -> 3.38, 5f6m MD start 4.16 -> 3.71 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6554e9dcbe4b448eda582eb76d6a24c59fe6ab0c)	2026-10-26 09:16:34 +01:00
leonarski_f	4a8e13ddfa	Refinement GPU engine: its waits on the stream asleep (blocking-sync event) rather than spinning ... Bitwise. cudaStreamSynchronize spins a core for the whole wait; the weight trials' engines wait on a shared card for most of a run, so about half of the user cycles of a refinement were the spin (PROFILE.md: 38-50%). A StreamWaiter (CUDAMemHelpers.h: an event made with cudaEventBlockingSync, recorded and waited for) now takes the refinement engine's waits - evaluation, Fcalc, Fmask, the uploads of positions, row model, blur and restraints. The short waits of the host-driven loops stay spinning: the scale fit's Levenberg-Marquardt (one launch of 66-120 us per step) and the gather's pair count; making those block too cost 2-3% of wall (wake-up latency), here the wall is unchanged. No process-wide cudaDeviceScheduleBlockingSync: that would change every GPU wait of the broker. CPU use (-N 8, time -v): 5reo 301-315% -> 190%, 5f6m 326% -> 181%, 9yl4 333% -> 222-228%, 7d98 307% -> 181%, 5f6m MD start 309-320% -> 151%. Wall, two interleaved runs each: 5reo 1.43-1.52 -> 1.43, 5f6m 2.19-2.20 -> 2.19-2.20, 9yl4 6.83-6.84 -> 6.88-6.89, 7d98 3.50-3.51 -> 3.58, 5f6m MD start 4.16-4.34 -> 4.16 s. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d23d2cf7bb99b30617a3f57c531b03db8bc03e05)	2026-10-26 09:16:34 +01:00
leonarski_f	f34d2f4cca	Refinement: the weight trials' own slots made while the placement runs ... Bitwise. MakeOwnSlot copies the model as it was before the topology and reads only what the placement does not change (cell, reflections, the engine's kind, the B ceiling), so the slots are now built on their threads right after the start score, beside the rigid-body and group-B placement, and joined where they were built before (also where the placement throws). The "slots" stage goes from 0.23 s to 0 on 9yl4; the placement is not slowed (0.59 -> 0.60 s). Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU); side by side = one at a time. Wall: 5reo 1.46-1.53 -> 1.54, 5f6m 2.26 -> 2.21, 9yl4 7.03 -> 6.83, 7d98 3.65 -> 3.56, 5f6m MD start 4.20 -> 4.15 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0752a921a4293d4213894e3242c4ef36d8ce2a05)	2026-10-26 09:16:34 +01:00
leonarski_f	1f8b5b1986	Refinement: a Refit's solvent grid starts around where the context's last one ended ... Logic change (deterministic), kept on evidence. Every Refit searched the whole k_sol/b_sol box - an 88-point coarse pass of plain least squares, then the robust fine pass around its winner. ModelScaleBox (and ModelScaleGPUPool::Fit / ModelScaleGPU::FitSolvent) take an optional start; with one, the coarse pass is skipped and the fine pass is laid around it. Refinement passes the grid winner of the context's previous Refit (before MoveOverallB moves the overall B into b_sol); it travels with a trial's fit, and a trial's first Refit without one (Context::ForgetStarts, which also clears the likelihood's D start) searches the whole box, so a trial still depends on nothing its slot ran before (side by side = one at a time, md5). Evidence (GPU, -N 8): full refinement of the 14 parity sets from their placed models - 12 refined CIFs identical (md5), 9ig7 and 6iu9 differ, final R-work/R-free equal to 4 digits but 6iu9 R-work 0.2911 -> 0.2910; of the 5 timing cases 4 identical, 7d98 (cif start) R-free 0.3096 -> 0.3095; 5f6m MD starts 01-04 median R-free 0.1692 -> 0.1692; 6o2h on the CPU engine identical (md5). The score-only parity table is not touched (one Refit, no start). refit.scale (TIMING sum): 5f6m 1.52 -> 1.10 s. Wall: 5reo 1.68 -> 1.53, 5f6m 2.43 -> 2.24, 9yl4 7.19 -> 7.02, 7d98 3.70 -> 3.65, 5f6m MD start 4.57 -> 4.22 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ff285fb525ecf4b480698d4aafe212a34301a8fe)	2026-10-26 09:16:34 +01:00
leonarski_f	1f1aa18b67	ModelScaleGPU: the robust fit's median of the residuals on the device ... Bitwise. The first robust round copied every fine-pass fit's residuals to the host (up to 25 x n floats, 9 MB at 92k reflections) and took std::nth_element of each there. The median is now a radix selection on the card (one block per fit, four byte-wide passes of integer counts over the float bits - the residuals are not negative, so the bits order as the values): the element nth_element puts at n / 2, the same value; only the fits' c come back. FitSolvent and RobustBatch lose the host threads they took for it. Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start. Wall (-N 8, quiet): 5reo 1.72 -> 1.68, 5f6m 2.57 -> 2.43, 9yl4 7.28 -> 7.19, 7d98 3.78 -> 3.70, 5f6m MD start 4.76 -> 4.57 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dc53896c335fdabdf4d417323038e05b8b63535d)	2026-10-26 09:16:34 +01:00
leonarski_f	ca0a6dd343	Refinement: each slot's likelihood fit on all of the run's threads ... Bitwise (the bins are fitted independently: the same bits at any thread count). The weight-trial slots Refit in lockstep while the GPU idles, each with its 1/slots share of -N for the ML bins; the bins now go to the shared worker pool with -N tasks per slot, so the threads of slots that are not fitting help the ones that are. After the Newton searches (previous commit) the ML fit is 0.1-0.5 s summed over a run, so the wall gain is within noise: 5reo 1.63-1.75 -> 1.72, 5f6m 2.60 -> 2.57, 9yl4 7.34 -> 7.28, 7d98 3.78 -> 3.78, 5f6m MD start 4.82 -> 4.76 s. Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 56815f5390cc01e8da5d420fff73cdf4394c385c)	2026-10-26 09:16:34 +01:00
leonarski_f	2c1df0ebfa	Refinement: the likelihood's D and beta searches by Newton's method; D on the working set from the last fit's ... Deterministic, rounding changes. The per-bin ML fit took 4 rounds x (40 + 40) golden-section steps on the free set and a 50-step golden section for D on the working set - each step a Bessel I0/I1 per row, ~4.6M per Refit at 92k reflections, 27-37% of a trial's wall on the high-resolution sets. Each one-dimensional search is now a safeguarded Newton iteration on the analytic first and second derivatives (bracketed on the sign of the derivative, bisection where a step leaves the bracket or the curvature is not positive), within the same brackets; the free-set alternation keeps its 4 rounds from the moments, so beta is the one it was (a converged joint fit was tried: the 4 rounds are not converged in the low-resolution bins, and converging them moves 6iu9's FOM by -0.003, so it was not kept). D on the working set starts from the bin's D of the context's previous Refit (XrayRowModel::bin_d), restored with a trial's fit and cleared at a trial's first Refit, so a trial depends on nothing its slot ran before (side by side = one at a time, md5). 14 parity sets, score only: R-work/R-free identical to 4 digits, mean |FOM - REFMAC| 0.01186 -> 0.01186, D within 1e-5 and var within 1e-4 (relative) per reflection. refit.ml (TIMING sum, -N 8): 5reo 1.47 -> 0.11 s, 5f6m 3.88 -> 0.26, 9yl4 1.37 -> 0.11, 5f6m MD start 4.97 -> 0.46. Wall: 5reo 2.03 -> 1.75, 5f6m 3.45 -> 2.60, 9yl4 7.99 -> 7.36, 7d98 4.04 -> 3.78, 5f6m MD start 6.10 -> 4.82 s. 5f6m MD starts 01-04 median R-free 0.1693 -> 0.1692. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a91511945e0ccd7145a93877c0c231e87f86fb44)	2026-10-26 09:16:33 +01:00
leonarski_f	a8d3a8dbc2	Refinement: the mask grid sized without allocating it; the L-BFGS pairs keep their s.y ... BulkSolventMask::MaskGrid, documented as "sizes and spacing set, values not allocated", allocated and zeroed the whole fine mask grid (up to 2^26 floats) - in BulkSolventMask's constructor and in every GPU engine's, i.e. once more for each weight-trial slot, only for the grid's size. It now takes good_grid_size() as set_size_from_spacing() does, and the CPU engine's Fmask() allocates its copy. rugnux_refine_test, RTX 5080, -N 8: 9yl4 8.6 -> 7.8 s, 7d98 4.85 -> 4.0 s (engine 0.66 -> 0.16 s, slots 0.60 -> 0.23 s on 9yl4); the high-resolution sets, whose mask grid is small, are unchanged. RefinementMinimiser keeps s.y of every pair it stores and Direction() takes rho and gamma from it instead of two more dot products per pair per call (the same Dot() of the same vectors): Direction's CPU time -22% on 9yl4; no wall change with the trials side by side, where other slots fill the card meanwhile. Bitwise: refined CIF identical on 5reo, 5f6m, 9yl4, 7d98, a 5f6m MD start (GPU) and 6o2h (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6f5caf5e1b1183a752acc5cca78bfb7ef002b12a)	2026-10-26 09:16:33 +01:00
leonarski_fandClaude Opus 5.5	6da51dc5d7	rugnux: no --refine flag - a placed model is refined by default, --no-refine skips it ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 06:54:15 +01:00
leonarski_f	3882b0f062	Serial stills with a reference or model: crystals their own cells anchor keep that indexing ... The per-image reference/model choice of indexing ignored the crystals' refined cells. On a pseudo-merohedral cell, where at least 90% of the crystals' cells fit the run's cell better as indexed than after a twin law, the cells decide that law - the rule the data-only resolution already uses (AnchorOnCells, factored out of ResolveReindexReferenceFree and shared). The resolver now keeps each image's CC per indexing (IntegrationOutcome::reference_cc); before the stills merge, AnchorReferenceReindex puts every image back as indexed, applies the cell rule, and lets the reference choose only among the indexings left open. Exactly merohedral lattices (no metric signal) are unaffected. Still battery (REFMAC 0-cycle R-free, model before -> anchored; data-only in brackets): pal90 0.1883 -> 0.1834 (0.1832), alvra213 0.2073 -> 0.2011 (0.2013), alvra213_int16 0.2081 -> 0.2021 (0.2024); euxfel83 0.2037, euxfel152 0.2452, sacla237, lcls222 unchanged; lcls242_t 0.2847 -> 0.2952 (0.2942): there the model had moved about half the crystals against cells that decide all three laws (96/96/100%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85c804676ffd15ffd66e2ba05928000d6f4e765f)	2026-10-26 06:49:20 +01:00
leonarski_fandClaude Opus 5.5	ba52aadbce	rugnux: the geometry refinement algorithm is -r, --geom-refine; --refine was declared twice and its no-argument form shadowed it ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 06:49:20 +01:00
leonarski_f	a78bfe951d	rugnux: a model and a reference among the inputs, in any order, told by their contents ... `rugnux <images> <model> <reflections>` in any order, without --model or -z. Each positional file is classified by its contents (RugnuxInputKind.h, in JFJochRugnuxOutput - the library still reads no files): an MTZ (magic) or an mmCIF with a _refln / _diffrn_refln loop is reference data (-z; the input itself in --mode refine), a PDB file (a PDB record first, then ATOM/HETATM) or an mmCIF with an _atom_site loop is a model (--model), anything else is images for the readers. CBF (###CBF or _array_data) counts as images; a file whose first line is neither data_ nor a PDB record is decided there, so image files are not read through. An mmCIF holding both atoms and reflections is refused as ambiguous. At most one model and one reference, --model/--mr and -z included. Gzip read through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8f844b0bf86f338d20e8aed426a387722dd8d7cc)	2026-10-26 01:38:25 +01:00
leonarski_f	794db0ca34	Refinement cut by the run's cancel; the hero shows the refined R-work / R-free ... The refinement of a placed model gets the run's cancel flag (RefinementSettings::cancel, through CancelFlag::RunFlag), so the broker's deadline or a cancel - and Ctrl+C in rugnux - cuts a running refinement short with what its finished trials found. RefinementResult::cancelled goes to the live status as rugnux_live_refinement.cancelled (API, regenerated clients) and to the step's note. Hero: once refined, the model card's headline is "R-work / R-free, refined" (e.g. 0.32 / 0.35), its green/red still the placement verdict, the placement and fit in the context line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 395016a30c2bc2aa26fdf2cec2ced9cebe6fad80)	2026-10-26 01:38:25 +01:00
leonarski_f	baec34685d	Rugnux live: the uploaded model is refined after placement; refined scores, model, maps and blobs in the status, the viewer and the hero ... The live run refines a placed model as `rugnux` does (ProcessConfig::refine), after every decision and before the end message; a cut before it starts leaves it out, as the model. The writer still gets the placed model; the refined one stays with the broker. API (broker/jfjoch_api.yaml): - rugnux_live_model.refinement (rugnux_live_refinement): performed, reason, on_gpu, seconds, r_work, r_free, rmsd_bonds_A, rmsd_angles_deg, blob_count, blobs (at most 50, rugnux_live_blob: kind, x, y, z, peak_sigma, volume_A3, nearest_atom, nearest_atom_distance_A) - rugnux_live_status.refined_model - /rugnux/model.cif and /rugnux/map.ccp4 take stage=placed|refined, by default the refined model where there is one; the anomalous map stays on the placed model's phases (the refined coefficients carry none). Regenerated broker/gen, frontend/src/client, redoc, and the python client docs (which also brings the hero fields and rugnux_live_model it was missing). Frontend: the model card keeps its green/red and gets "R-free 0.xx (refined)" as its context line, "refining..." while the step runs; the model section adds refined R-free and blob tiles; Mol* shows the refined model and maps, with the blobs as small spheres coloured by reading (switch, off at first). The Refinement step is in the timeline like any other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3b47fb9a53b7dfad1c1ed289e169094c8d3d9d3e)	2026-10-26 01:38:25 +01:00
leonarski_f	7bf906e4a2	rugnux: a placed model is refined by default - giving a model is the opt-in ... Refinement (Refinement.h) now runs whenever --model or --mr places the model (rigid body fits, or molecular replacement placed it), on the GPU or the CPU: a run that should not spend the time is given no model. --no-refine skips it (for debugging); --refine is still accepted and ignored. A model that was not placed (rejected by the rigid body, not placed by MR - e.g. the battery's wrong-model null) is no longer refined. Processing-inert as before: it runs after every decision; R_WORK/R_FREE and every key outside REFINE_ (section 5), and <prefix>.mtz / _model.cif / _maps.mtz, are unchanged. The block moved after the placement notes so they stay on the step that placed the model. The viewer's processing jobs refine too. The equivalent command line gives --no-refine back where refinement was off. Battery: score.py reads keys exactly, so REFINE_R_FREE cannot alias R_FREE; new report-only fields rfree_refined, rwork_refined, refine_s (part of rugnux_wall_s), shown in the per-set table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit b0bb741831f517b553e59152b3b482a3f75d953a)	2026-10-26 01:38:24 +01:00
leonarski_f	3b06bcee78	Refinement: the trajectory ends at its best step, steps try the positional weight halved too, the final ladder must win ... Three changes to how the trajectory of commit 2bfb84bc4 moves and ends: - A step is taken while its trial lowers R-free or R-work by more than the tie and its R-free stays within the tie of the best so far; the trajectory then ends at its best step. On a deposited model, whose free set it was fitted to, R-work falls while R-free rises, and the trajectory now ends where it began instead of walking on while R-work falls. - Positional steps try the weight halved as well as doubled: at a fixed weight the geometry loosens as the model improves (the X-ray gradient grows with D) until the trials leave the bounds, which stopped 8dqb's trajectories at 2 A (angles > 2.0 deg at w0/128 by step 6). - The final ladder's choice replaces the trajectory's end only where it displaces it (geometry within bounds, R-free lower by more than the tie); before, its first rung was taken by default, two more cycles of movement that on deposited models cost R-free. Deposited models (14 parity sets), R-free after refinement vs refine-combined: mean +0.0058 -> +0.0016 (e.g. 7n2s +0.012 -> +0.003, 9rcs +0.013 -> -0.008, 7d98 +0.015 -> +0.003, 9yl4 +0.010 -> +0.003), the time halved against refine-combined. MD starts, median R-free (REFMAC 0-cycle scorer), previous commit -> this: 5f6m 0.167 -> 0.168, 8dqb 0.294 -> 0.297, 7arr 0.278 -> 0.290 (REFMAC 0.311), 11if 0.244 -> 0.245, 5reo 0.258 -> 0.259, 7d98 0.335 -> 0.335. Determinism: run A == run B == --trials-at-once 1 on 5reo, 9rcs and a 5f6m MD start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c2b55aab1aaff637d9484dec0d25003b5c4a3857)	2026-10-26 01:38:24 +01:00
leonarski_f	bda90dbdd5	Refinement: B restraints in proportion to B (Kullback-Leibler form) with contacts, an own ADP weight, overall B into the atoms ... - B similarity: E = (B_i - B_j)^2 / (kappa^2 B_i B_j), the symmetric Kullback-Leibler divergence of two isotropic Gaussians (Murshudov et al. 2011), so sigma = kappa sqrt(B_i B_j) instead of a fixed 1.5-4.5 A^2. kappa 0.1 / 0.133 / 0.2 / 0.3 for main-chain bonded / angle-related and side-chain pairs (REFMAC's logs list sigma ~0.1 B for main-chain bonded pairs at B 11-140; the ratios are those of the old sigmas, which these equal at B = 15). New class: atoms within 4 A that no bond or angle relates, waters and symmetry mates included, kappa 0.5 (BuildBContacts, sharing the neighbour search with BuildRepulsions). CPU and GPU. - The B factors' X-ray weight is balanced on its own (BalancedADPWeight: B-restraint over X-ray B-gradient on the model with B shaken by 20%) instead of being a multiple of the positional weight; the ladder tries B held and refined at that weight, the trajectory's B steps divide and multiply it by four, positional steps and the final ladder leave it. - Before each B-refining macro-cycle the overall isotropic B of the scale moves into the atoms and the solvent's B (F_model unchanged), so the two do not drift against each other. Tried and not kept: xyz and B as one L-BFGS vector (the B block scaled by its curvature, the positions by the median X-ray curvature): no better than alternation with a memory per block on 12 starts (mean R-free 0.265 vs 0.262), worse on 7arr/7d98; a refit before each B step: no gain for +20-40% time. Refinement battery MD starts, median R-free by the REFMAC 0-cycle scorer, previous commit -> this (REFMAC to convergence): 5f6m x8 0.167 -> 0.167 (0.168), 8dqb x8 0.291 -> 0.294 (0.284), 7arr x4 0.270 -> 0.278 (0.311), 11if x4 0.245 -> 0.244 (0.251), 5reo x4 0.312 -> 0.258 (0.261: the ladder no longer keeps B held where B is the model's largest error), 7d98 x4 0.342 -> 0.335 (0.361). Deposited models: unchanged (B stays held there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ed292f35b482b696d81f71f1d97e0016ca05b144)	2026-10-26 01:38:24 +01:00
leonarski_f	6369a263b1	Refinement: one trajectory of short weight trials instead of a ladder of full refinements from the start ... Every rung of the old ladders (6 positional, then 3 B) was a full 10-cycle refinement from the placed model, so ~90% of the evaluations were thrown away and the kept trajectory had 100 iterations. The run is now: 1. a ladder from the placed model, 2 macro-cycles per trial: three positional weights (w0/1024, /128, /16), each with the B factors held and refined (adp = 4 w), chosen by the old rule (tighter first, a later trial wins only within the geometry bounds and by more than the R-free tie); 2. a trajectory of steps from where the last one ended, each two trials side by side - the weights as they are and the X-ray term raised, in turn the positions' (x2, at most w0/8) and the B factors' (x4) - until a step lowers neither R-work nor R-free by more than the tie (at most 8 steps); 3. a final ladder from there at x1/2, x1, x2. A macro-cycle is 30 positional and 20 B iterations (the minimiser's memory kept within a trial). A trial starts from the scale, solvent and likelihood fit its predecessor ended with (the same computation, so result-neutral), and the slots that run share the threads. On 5 battery MD starts (start 1 of 5f6m 1.09 A, 8dqb 2.04, 7arr 0.92, 11if 1.37, 7d98 3.32; R-free by rugnux's scorer, REFMAC 0-cycle within 0.002), previous commit -> this, against REFMAC run to convergence (40-100 cycles): 5f6m 0.215 -> 0.167 (REFMAC 0.168), 8dqb 0.341 -> 0.290 (0.284), 7arr 0.356 -> 0.289 (0.288), 11if 0.295 -> 0.236 (0.244), 7d98 0.393 -> 0.337 (0.360). Time 5.4 / 8.6 / 7.3 / 3.9 / 10.0 s on the RTX 5080 (base 6-7 s). Halving a weight during the trajectory never won in the logs, so steps only try raising it; the final ladder can lower it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 2bfb84bc4f3c72e8da37e54b728839b65a636519)	2026-10-26 01:38:24 +01:00
leonarski_f	4424d8884d	Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature ... Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every 10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian. The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints' Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser, since that stiffness is in directions that move bonded atoms together. Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345, 7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393; REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung further). Time unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825)	2026-10-26 01:38:24 +01:00
leonarski_f	626afc49dd	Refinement likelihood: the searches over D alone with the variance taken once ... EstimateLikelihoodParameters' golden-section searches over D at a fixed beta - four per bin on the free reflections, and the one on the working reflections the D/FOM change added, 50 evaluations over 95 % of the data - evaluated -ln L reading every row through the index and taking log(var) at every evaluation. The rows are now gathered per search with their variance and its logarithm; the sum is the same expressions in the same order, so D, beta and everything after are the same to the bit (refined model and report identical on the seven timing sets). refit.ml 200 -> 140 ms per cycle on the 1.1 A set, one thread per trial slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 23ad3bfc254d10af310f17dbb8f85616f666a783)	2026-10-26 01:38:24 +01:00
leonarski_f	6075239be0	Refinement likelihood: D on the working set, the measurement error per component, as REFMAC 5.8 estimates them ... The likelihood's variance per bin (beta) is fitted by maximum likelihood on the free reflections as the whole spread of Fo about D Fc, and D is then fitted on the working reflections with the variance eps beta + 2 sigma^2 (acentric; + sigma^2 centric), which is also the target's variance. Murshudov et al. (2011) Acta D67 355: D from the working set, the variance from the free set. The measurement error is thereby counted twice, so D and the figure of merit fall where sigma is a large part of the spread - at high resolution. Measured on REFMAC 5.8.0431 0-cycle output (black box) of the 14 parity sets: its FOM inverted per reflection gives var = eps a + 2 sigma^2 (acentric, fitted coefficient 2.00) and eps a + sigma^2 (centric); its D (FC_ALL/FC_ALL_LS = the log's SigmaA_Fc1) and a are reproduced by this estimate (on REFMAC's own Fc, one set to rms 0.009 in D over all 20 bins). The previous per-bin free-set ML put D at 0.39-0.48 where REFMAC has 0.06-0.13 in the outermost bins. rugnux score vs REFMAC 0-cycle, 14 sets: mean |FOM difference| 0.056 -> 0.012 (max 0.112 -> 0.042), per-bin D rms 0.107 -> 0.068; R-work/R-free unchanged (D does not enter them). The score's FOM is now the mean over the working reflections, as REFMAC reports it, and the refined model's maps are weighted by the refinement's own m and D rather than a separate sigma_A estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d00d635e2874bff6cd216b7f24367362d0cc2e56) (cherry picked from commit b93e0ea2fc1ffcc6bb40dae9b77816a321f4d6ec)	2026-10-26 01:38:24 +01:00
leonarski_f	bcb71136ab	Refinement restraints: metal ions bound to their donors by the model's LINK records ... Metal sites were held only by the 1.9 A metal contact of the repulsions; REFMAC restrains the metal LINKs of the coordinate file (its documentation), which was the one geometry term the 0-cycle comparison did not share (one 2.8 A set: bond rmsd 0.018 against REFMAC's 0.042). - monomers/metals.cif: the Monomer Library's own statistics of metal sites (metals.json, LGPL data as the rest of monomers/), cut by update_monomers.py to the metal ions of its compound list and O, N, S donors: mean distance, spread and number of sites per metal, donor element and coordination number. - AddMetalLinks (before prepare_topology): every LINK / struct_conn record between a metal ion and an O, N or S that no link of the dictionary describes (its ZN-CYS, ZN-HIS, FE-CYS, FE-HIS are taken as they are) gets a link of one bond at EmbeddedMetalDistance for the two elements and the ion's coordination - the number of its donor records, symmetry mates included - from the rows resting on at least 20 sites (the coordination's own row, else the neighbours interpolated), sigma their spread (>= 0.02 A); a His or Cys donor loses the proton the bond replaces (DEL-HD1 / DEL-HE2 / CYSmod1, as the library's links do). Records only, no distance search; records to a symmetry mate count for the coordination but are not restrained (bonds to symmetry mates are not, anywhere). - The metal bonds are a restraint set of their own (RefinementTopology::metal_bonds), evaluated with the geometry and reported apart (GeometryStatistics::rmsd_metal, REFINE_RMSD_METAL_BONDS, REFINE_SCORE_RMSD_METAL_BONDS / _BONDS_ALL): their sigma is the spread of real sites, so they sit further from ideal than covalent bonds, and the weight ladder's 0.02 A bound stays a bound on the covalent ones. refine_parity.py compares REFMAC's bond rmsd (links included) with the combined one, and takes --gpu. Score-only geometry against REFMAC 5.8 0-cycle, all bonds (count): 2.8 A Zn/Fe set 0.0402 (4934) vs 0.042 (4935), was 0.0178; a Zn-Cys set 0.0186 (7102) vs 0.019 (7102); a Ca set 0.0297 vs 0.026 (REFMAC also refines the sugars and uses one Ca-O distance); a Mg set 0.0114 vs 0.017 (REFMAC restrains 8 symmetry links and puts Mg-O at 1.99 A, the library's statistics at 2.07). R of the scored model unchanged (+0.0001 on the first from the removed His/Cys protons). Refined R-free with the links vs without: +0.0016, -0.0002, +0.0005, +0.0004. Test: RefinementRestraints_metal_links (Zn-Cys4 by the dictionary's link, Mg(H2O)6 by the new ones, both back at their distances). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0fcffc5c71e03e184cff18f07524473462a57b68)	2026-10-26 01:38:24 +01:00
leonarski_f	9da3851844	Refinement: at most n weight trials at a time, and a cancel flag the caller can set ... - RefinementSettings::max_trials_at_once (rugnux_refine_test --trials-at-once): 0 as many as the ladders and the cards take, 1 one after another - the check that side by side changes nothing. Verified: refined model and report identical, side by side vs one after another, GPU (5reo, 9yl4, 6o2h, 9rcs) and CPU engine (6o2h, 9rcs, 5reo), and run to run. - RefinementSettings::cancel, an atomic flag for the broker's live deadline: checked at every iteration of the L-BFGS (positions, B, rigid bodies, group B: the minimiser stops where it stands), at every macro-cycle and before every weight trial. The trials that ran are chosen among as usual; cut before the first, the model is the placement's. RefinementResult::cancelled says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f708d3d227bb42215ffe3d9a15c766dbd73b783c)	2026-10-26 01:38:24 +01:00
leonarski_f	8718065a7c	Refinement GPU: the bulk-solvent mask on its own fine grid on the card ... The score draws the mask on a grid of its own (min(d_min/3, 0.4 A), BulkSolventMask) on the host; on a GPU engine that was a CPU mask and transform every cycle, per trial slot on one thread. Now the engine masks on that grid itself: - RefinementGPUSetup::mask_grid (BulkSolventMask::MaskGrid, the grid the host mask uses); where it is not the structure factors' grid the engine reserves a grid, a spectrum and an R2C plan for it (counted in DeviceBytes and the FFT work area), and Fmask masks (ModelMaskGPU), transforms and reads the rows there. Refit takes the engine's Fmask on a GPU, BulkSolventMask on the CPU. - ModelMaskGPU's shrink: each edge point marks its stencil as solvent instead of every macromolecule point searching its stencil for an edge - the same idempotent stores to the same points, so the same mask to the bit (ModelMaskGPU_* tests unchanged), and a thin shell of points visited instead of half the grid. - BulkSolventMask finds its symmetry orbits at its first mask: a GPU refinement never draws one on the host (1.1 s of trial-slot setup on the 3.6 A set). The card's mask is gemmi's but at points exactly on an atom's radius (ModelMaskGPU.h): R-work/R-free of the 14 parity sets on the GPU are the CPU engine's to 4 digits. Refinement deterministic run to run on the GPU (7 timing sets, refined model and report identical over 5 runs of successive builds that did not change results). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 89b5d00907c9ad99e8b389f6272a3724f981cd10)	2026-10-26 01:38:24 +01:00
leonarski_f	46ad2c10ed	Refinement GPU: the robust scale fit on the card, the pool back in every trial slot ... The score's robust fine pass (pseudo-Huber IRLS of ModelScaleBox::robust) on the device, so that refinement on a GPU no longer fits its scale on the host every cycle - with the trials side by side that was a robust grid search on one thread per slot (13-133 s per refinement on the seven timing sets, against 1.7-8 s before the score commit). - ModelScaleGPU: the weights multiply the WSSR and J^T J / J^T r in scale_sums (a weight row per fit, reserved at the first robust fit); robust_residuals / robust_weights kernels; the median of each fit's first residuals taken on the host as FitModelScale takes it (nth_element at n/2, on the fits' threads); LevMarBatch split out of FitBatch so that the rounds restart from where a fit stands. FitSolvent(..., robust) refits the coarse winner in the fine pass, as FitModelScale's robust fit does; the plain fit is unchanged. - ModelScaleGPUPool::Fit(..., robust); its streams at the highest priority: a fit is a chain of small launches the host waits on, and on a card shared with the trials' long kernels each wait queued behind them (refit.scale 58-87 ms -> 14-45 ms per cycle, the result the same to the bit). - Refinement: the main context and every trial slot fit on their pool (robust); the CPU engine keeps FitModelScale's robust fit. The card's fit is FitModelScale's to rounding (float residuals and weights, the steps' anisotropic factor in float), deterministic run to run and whichever engine a fit takes: R-work/R-free of the 14 parity sets on the GPU equal the CPU engine's to 4 digits (largest difference 1.4e-6 on the 3.6 A set). Test: ModelScaleGPUPool_RobustMatchesFitModelScale (one reflection in fifty a 4x outlier, five cells). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d15f048cbd7ed2d6b430f9f9cb5018ca284da8f0)	2026-10-26 01:38:24 +01:00
leonarski_f	a218c0bec8	rugnux --mode refine: refine a model against merged structure factors, nothing processed ... rugnux --mode refine --model model.cif -o out data.mtz The input is an MTZ or a structure-factor mmCIF, gzipped or not, by content (writer/RugnuxReflectionFile: parsing in the file layer, the library reads no file). Columns are found, not named: F/SIGF-like amplitudes (calculated amplitudes and map coefficients passed over), else intensities made into amplitudes by French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set, or the minority value), else rugnux's own 5 % set. RunRefineOnly (rugnux/RugnuxRefine) runs RefineModel; the outputs are --refine's (_refined_model.cif, _refined_maps.mtz, _blobs.json) and a report with the REFINE_ keys, the input's REFINE_FRENCH_WILSON / REFINE_FREE_FLAGS, and every term of the start score under REFINE_SCORE_ (also in --refine's section). --refine-score-only scores the model as given; --refine-d-min cuts the resolution. tools/battery/refine_parity.py runs REFMAC 5.8 at 0 cycles and rugnux --refine-score-only on the same files and prints the terms side by side, with R by resolution from both programs' model amplitudes on the same reflections. Tests: [refine_only] - column choice (observed over map coefficients, intensities, 1-marks-free flags, no flags, no usable column), an MTZ read from its bytes, and the mode end to end on a synthetic cell (French-Wilson, generated flags, score-only, the report keys). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 63ee830c39c988588cffb8c9d4638a678815ec2c) (cherry picked from commit d7397311e6ce0d3bb37ac91e8973b6052e38589f)	2026-10-26 01:38:24 +01:00
leonarski_f	c4d2d419ce	Refinement score: the model's structure factors as REFMAC 5.8 scores them at 0 cycles ... Three terms of the score of a model - the same F_model at every stage, the as-given start and the refined end alike - and a breakdown of it: - Riding hydrogens from the start: the dictionary's hydrogens are added before the model is first scored (REFMAC's 0-cycle restores them; they move R by 0.010-0.013 at 1.1 A). The engine is made once, with them. - The bulk-solvent mask on a grid of its own, min(d_min/3, 0.4 A), at most 2^26 points. On the density grid (1.1 A at 3.3 A) gemmi's 0.8 A shrink found no grid offset and was skipped, which put R 0.015-0.03 high at >= 2.5 A. 0.3 and 0.4 A agree to 0.001. - The overall scale and anisotropic B by robust least squares (pseudo-Huber, iteratively reweighted, c = median |residual|) in the fine pass of the k_sol/b_sol grid (ModelScaleBox::robust, refinement only; MR and validation keep the plain fit). Plain least squares on |F| is the exact LS optimum but lets a few badly-fitted strong reflections set the scale (1.1 A set: R 0.1706 against REFMAC's 0.1600). The scale fit of refinement now runs on the CPU (no ModelScaleGPUPool). RefinementSettings::score_only scores the model as given and stops; RefinementResult::score_start carries every term (k, B, anisotropic B, k_sol, b_sol, mask spacing, R in 20 shells, -lnL work/free, the likelihood FOM, hydrogens; per-reflection terms with score_only). GeometryStatistics gains rmsZ. rugnux_refine_test: --score-only, --dump. Measured against REFMAC 5.8.0431 ncyc 0 (refi bref iso) on the same model, data and flags, 14 open-arm sets 1.1-3.6 A: R-work/R-free within 0.002 on 12 (was +0.003..+0.03); 7n2s and 9yl4 score 0.003/0.005 below REFMAC, which a 0.6-0.8 A mask grid reproduces. REFMAC with ANISOU stripped scores identically, so isotropic B is not a parity term. Open: the likelihood's D/FOM (ours 0.01-0.11 higher at high resolution); geometry differs only by REFMAC's metal LINK restraints and residues outside the embedded dictionary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 4f94b21611874ad9dcf89e6771fe311a0e10bb06) (cherry picked from commit 00db6a6f539ac2da515c0a649474335f4c4b07b8)	2026-10-26 01:38:23 +01:00
leonarski_f	8933a72a89	Refinement: the weight trials side by side, each on a model, topology and engine of its own ... Every rung of the weight ladder starts from the same model and depends on no other, so the six position trials, and then the three B trials at the chosen weight, run at the same time: on the main model and engine and on up to five slots beside it, each a copy of the model before the topology was made (the topology is built again on it, hydrogens included) with its own engine. The slots go on the GPUs in turn from the main engine's, as many as each card's budget takes, so a machine with several cards spreads them; a slot whose engine cannot be made is left out, and with none the trials run one after another on the main engine as before. The threads are shared out among the slots. The log lines and the choice are made afterwards in ladder order, so the log, the trial table and the refined model are the same, bit for bit (GPU and CPU engines checked). 5reo 3.9 -> 2.5 s, 9yl4 13.4 -> 7.8 s, 6o2h 3.1 -> 1.9 s, 8dyz 4.6 -> 2.7 s, 9rcs 2.9 -> 2.2 s (one RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0ea6a6db8aba640afc5249fe748922ddfaa2537a) (cherry picked from commit 0c0176c8bf571ed26f8a68a9de40222487c6200e)	2026-10-26 01:38:23 +01:00
leonarski_f	5569f3d1f2	Refinement GPU: the X-ray target and the restraints in one submission with one wait ... RefinementEngine::EvaluateWithRestraints evaluates both targets at the model's atoms. On the GPU the X-ray chain is queued first, the restraints' coordinates are written into a pinned buffer while the device works on it, the restraint kernels follow on the same stream, and the host waits once; the sums and gradients come back through pinned buffers. Same kernels on the same inputs, same bits. The CPU engine keeps calling the two in turn. 9yl4: evaluation 5.1 -> 4.7 ms, refinement 14.2 -> 13.4 s; 5reo 4.0 -> 3.9 s (RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 51434b98b62a9e6322ffe811bf272a45ff6ce003) (cherry picked from commit 0bcfa3f8041cec4bb4bc376b8559f30c36d3ac80)	2026-10-26 01:38:23 +01:00
leonarski_f	001e61c1a4	Refinement: the repulsion search on the refinement's threads, hydrogens skipped before the neighbour walk ... BuildRepulsions walked the neighbours of every atom and only then dropped the pairs of a riding hydrogen; those atoms are now skipped before the walk. The atoms are searched in 64 blocks under ParallelFor and the blocks' pairs joined in atom order, so the restraint list is the same as before for any thread count. The model is no longer copied for the NeighborSearch. Same bits. 9yl4: repulsions 3.37 -> 1.09 s (91 calls), refinement 17.7 -> 14.2 s; 5reo 0.38 -> 0.12 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ded11388a18f5a4138a73150c4bb073570152727) (cherry picked from commit 9c96836a90c9f170008d343af2cd885a4e59b321)	2026-10-26 01:38:23 +01:00
leonarski_f	e3782cabe4	Refinement GPU: atom densities uploaded only when some B moved, one device copy, positions through a pinned buffer ... Every evaluation uploaded both copies of the atoms' densities (the gather's and the gradient's) and their boxes, although in a positions-only minimisation no B changes between two Refits. The engine now keeps one copy (the gradient kernel reads ModelDensityGPU's atoms and boxes), uploads it only where some atom's density was made again, and sends the positions from a pinned buffer without waiting. The densities that are made again (every atom in a B minimisation) and the grid positions are worked out on the refinement's threads. Same bits. 9yl4: upload 3.29 -> 0.56 s, host side 1.29 -> 0.61 s, refinement 22.3 -> 17.7 s; 5reo 4.8 -> 4.4 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 46c3b67c91aee0aec3473b0fb1a4945c00bda070) (cherry picked from commit 0aebc98110c8155932af647d6204891aec662294)	2026-10-26 01:38:22 +01:00
leonarski_f	dd49d4e2f2	Refinement: the likelihood's D and beta fitted bin by bin on the refinement's threads ... EstimateLikelihoodParameters fits each resolution bin on its own and writes only that bin's values, so the bins run under ParallelFor; the result is the same, bit for bit. It was the largest single stage of a refinement (107 calls, one core): 5reo 2.42 -> 0.56 s, 8dyz 2.46 -> 0.57 s, 9yl4 3.54 -> 0.88 s; whole refinement 5reo 6.3 -> 4.8 s, 8dyz 6.9 -> 5.3 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85aa34d3031c8e36be4462fca544bc801af45d5a) (cherry picked from commit d84ba55252adf455d5b4ba62052dfbafa6290a64)	2026-10-26 01:38:22 +01:00
leonarski_f	24b1711d64	Refinement: time per stage - an NVTX range per call and one TIMING line per stage with its sum and calls ... RefinementStage sums the time of each stage over a refinement (Refit and its parts, the evaluations and their X-ray, restraint, host and upload shares, the repulsion build, the riding hydrogens, the rungs) and logs it at debug level at the end; rugnux_refine_test -v shows it. No change to any result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 9491f54e0beccf36c65ebcec9afff90d08485f86) (cherry picked from commit 0ce206a88df960754476ce2f75ffaed90fbf3123)	2026-10-26 01:38:22 +01:00
leonarski_f	07ae590e54	Serial stills: the indexing ambiguity resolved from the data when no reference or model is given ... Where the space group is given (-S) and its lattice admits alternative indexings, a stills run without -z or --model used to merge the crystals' random indexings together (and warned). The crystals are now made consistent with each other before anything is scaled (ReferenceFreeReindex): each takes the indexing in which it agrees best with all the others (Brehm & Diederichs 2014), agreement being the sum over shared reflections of the products of shell-normalised, per-crystal standardised intensities, weighted by 1/(1+(sigma/sd)^2). The sum over all pairs is (|M|^2 - const)/2 for M the running sum, so a pass over the crystals is one pass over the reflections rather than over all pairs: coordinate ascent, four seeded starts, stop when <= 1 crystal in 1000 moves, the overall relabel that moves the fewest crystals. Deterministic (fixed summation order, sequential passes). A twin law of a pseudo-merohedral cell whose axes differ by a few per cent is decided by the crystals' own refined cells, which the indexer has already used: where >= 90% of the crystals' cells fit better as indexed, those crystals are anchors and only the others are decided by the intensities. Without this, two public sets got worse (pal90 R-free 0.183 -> 0.207, lcls242_t 0.296 -> 0.350). With a reference or model the per-image choice stands; the same resolution runs as a check, reported as AMBIGUITY_DATA_DISAGREES. New report keys (section 2): INDEXING_AMBIGUITY_RESOLVED, AMBIGUITY_CRYSTALS, AMBIGUITY_REINDEXED, AMBIGUITY_SEPARATION_SIGMA, AMBIGUITY_DECIDED_FRACTION (+ AMBIGUITY_TIME_S, dev). --mode scale is not changed. Still battery, no --model (R-free, REFMAC 0-cycle against the deposited model; before -> after; with --model for comparison): euxfel83 0.321 -> 0.207 (0.204), euxfel152 0.346 -> 0.257 (0.245), alvra213 0.209 -> 0.201 (0.207), alvra213_int16 0.210 -> 0.202 (0.208), pal90 0.183 -> 0.183 (0.188), lcls242_t 0.296 -> 0.294 (0.285); controls sacla237 / lcls222 unchanged (no ambiguity, no-op). The step takes 0.2-1.6 s on 2.7k-54k crystals; synthetic 64k crystals x 200 reflections 0.5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 80d378a1fe92364fd5b99a17f5c9ee8498412d36)	2026-10-25 23:26:32 +01:00
leonarski_fandClaude Opus 5.5	2daa173859	docs: --model on serial stills - breaking the indexing ambiguity is the one thing taken from it before the merge, and why ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-25 20:10:38 +01:00
leonarski_f	c982ba4f65	FFBIDX on the GPU: the same spots index to the same cells, whatever order the thread blocks finish in ... Each block of the candidate-vector and the cell search sorts its own candidates and merges them into one shared top list under a lock, in the order the blocks happen to get there. The merge compared values only, so where two candidates scored the same - common on the 30-spot seed the stills path tries first, since the score is a sum of trimmed terms - the place went to whichever block merged first. The same frame then indexed to a slightly different lattice, or not at all, from one run to the next. Ties are now broken by the candidate's identity - the sample index for a vector; the cell vector (with its flip), candidate and rotation sample for a cell - which makes the order total, so the merged lists are the top-N of everything whatever the merge order. The blocks' own lists are already in that order: the block radix sort is stable and its samples ascend with the thread index. Measured on a 100k-frame serial set: 4-21 of ~33k indexed frames came out with a different lattice in every pair of runs, which the stills scaling and partiality chain turned into 0.97-0.99 CC1/2 and thousands of observations in or out of the merge. With the fix, three runs at the default thread count and one at -N 8 give the md5-identical merged .hkl; a per-frame dump (spot list, indexer answer, refined lattice) of 20k frames is identical across three runs where it differed in 4-6 frames before. FFBIDXIndexerCPU_MatchesGPU and Portable_FFBIDXIndexerCPU_SyntheticStill pass. No regression test: a repeat-and-compare test on the library's spot lists, threaded, passed before the fix too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 9cff12d652088851b81f6a353ac52aa8318799f2)	2026-10-25 19:58:38 +01:00
leonarski_f	64bbc44e11	fast-feedback-indexer vendored in image_analysis/indexing/fast-feedback-indexer instead of fetched ... The GPU known-cell indexer behind FFBIDX is now built from an in-tree copy of upstream commit a94fd4f5c55289e70860feea358182d2072e02b5 (the archive the FetchContent block pinned), so it can carry local fixes. No behaviour change. Copied: indexer/src/{indexer.cpp, indexer_gpu.cu, log.cpp} and the ffbidx/ headers the build uses (indexer.h, indexer_gpu.h, refine.h, log.h, exception.h, envvar.h), each with a header line naming the upstream URL, commit and licence; LICENSE.md beside them; and the library's test spot lists (data/simple/files/image{0..8}_{local,peakfinder8,radial}.txt) that FFBIDXIndexerCPU_MatchesGPU reads, now found in-tree. ffbidx/version.h is written once ("1.1.3-a94fd4f") instead of generated - the generator ran git in the parent build, so it used to stamp Jungfraujoch's own hash. The in-tree target keeps the name fast_indexer_static and what upstream's indexer/src/CMakeLists.txt gives the static library: the source list, CUDA_RUNTIME_LIBRARY Static, cxx_std_17 (PUBLIC), the source directory as PUBLIC include, CUDA::cudart_static and Threads::Threads (PRIVATE), Eigen3::Eigen (PUBLIC, the project's), the CUDA-architecture default only where none is set, and -Xcompiler=/Zc:preprocessor for the CUDA sources under MSVC (CUB needs the conforming preprocessor). Left out: the shared library, the C API (c_api.cpp, unused), install/pkg-config rules, the version_txt generator target, the top-level project's data/tests/python/examples subdirectories, and -DVECTOR_CANDIDATE_REFINEMENT=1 (no source reads it). Licences: COLLECT.sh takes the text from the vendored LICENSE.md; the THIRD_PARTY_NOTICES row moves from the fetched to the vendored table. CLAUDE.md's list of FetchContent downloads no longer names it. Verified: clean configure and build of rugnux and jfjoch_test; FFBIDXIndexerCPU_MatchesGPU and the Portable_FFBIDXIndexerCPU tests pass; a 3000-image stills run gives the md5-identical merged .hkl with the fetched and the vendored library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0e4fcb87de0fbb7c0022750ff4d62d8c5a5c3c18)	2026-10-25 19:58:38 +01:00
leonarski_f	cf7676f5d5	Stills merge: outlier rejection on by default (10 sigma), the band linear in the error model's sigma ... The stills merge had no working outlier rejection. --reject-outliers was 0 for stills, and when set its band took the systematic part as a factor exp(+-n s), s the spread of ln(I / median) - on stills a reflection's observations are partials and per-crystal scales, many near zero, so s is wide and the band let through observations hundreds of sigma out. On a high-multiplicity broad-bandpass run with a spotty powder contaminant, single grains in a weak reflection's box were that reflection's whole merged intensity (half-set means -2 vs 2284 on 141 observations); the shells holding the powder rings and the lowest shell merged at CC1/2 0.6-0.75 beside 0.99 elsewhere, at multiplicity ~400-1000. Stills now test the plain linear band, n sqrt(a sigma^2 + (b <I>)^2) about the median - the error model's b is the spread as it is - at n = 10 by default (rotation keeps 6 and its multiplicative band). The stills merge statistics now report the rejected count (OBSERVATIONS_REJECTED was always 0 there). Measured (REFMAC 0-cycle R-free of an isomorphous room-temperature model, depositor's free set, scale mode on stored integrations, vs the same code with --reject-outliers 0): - broad-bandpass in-house stills, 50k frames: 0.318 -> 0.265; 20k: 0.274 -> 0.244; R_split 0.29 -> 0.06, CC1/2 0.79 -> 0.98. 0.1% of the observations are rejected. - monochromatic in-house stills, 30k frames: 0.216 -> 0.212; R_split 0.126 -> 0.087. - n = 6 gains ~0.003 more there but costs the public XFEL sets 0.003-0.006 on three of four; 20 keeps most of the gain. Still battery, 12 non-float sets, against rc180: R_split better on all 12; R-free change mean -0.0005 (range -0.006..+0.003, plus pal86 +0.009 that is within that set's run-to-run spread of ~0.007 - a rerun of rc180's behaviour gave 0.192 against the baseline's 0.185); one draw becomes a win (lcls242_s), nothing else changes class. Rotation is untouched (the stills band and default are gated on the experiment not being a rotation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3a7142801056f21c720956255f1207e0bf1a8f43)	2026-10-25 19:58:38 +01:00
leonarski_fandClaude Opus 5.5	47960ccaf7	rugnux_refine_test: the CUDA device-link step a portable build needs for the static cuFFT, as rugnux and the portable test have ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-25 12:54:19 +01:00
leonarski_fandClaude Opus 5.5	f077320efa	Refinement restraints: n_slots, not slots - Qt defines slots as a macro, which broke the viewer's build through Rugnux.h ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-25 10:22:02 +01:00
leonarski_f	d3673d17fc	RefinementTest: the GPU restraint check on an engine made for a cell its gather takes ... The restraints do not read the cell; the 14 A cell that makes the symmetry contacts is too narrow for the gather's boxes, so the engine was refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a6e00e10ce1798a97d1f054c7553cc9eb6b5cf4b)	2026-10-25 07:20:32 +01:00
leonarski_f	789bd37606	Refinement: ten short macro-cycles instead of five long ones ... The scale, the bulk solvent and the likelihood's parameters are held within a macro-cycle; refitting them twice as often, with 10 L-BFGS iterations on the positions and 5 on B per cycle (was 25 and 15), lowered R-free on a 1.65 A deposited model from 0.2218 to 0.2189 with B held, and halved what B refinement cost there, at the same number of evaluations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8021d0cfc0bd56b1cc149a40df72ed78f35367c8)	2026-10-25 07:20:32 +01:00
leonarski_f	9cb5cc6237	Refinement: placement steps kept only where R-free drops; a wider weight ladder ... On deposited models the per-residue B shift raised R-free at every resolution tried (2.56 A: 0.320 -> 0.327) and the rigid bodies rarely helped; each is now kept only where it lowers R-free by more than the 0.001 tie, otherwise the model goes on as it was. The positions' weight ladder runs from 1/1024 to 1/32 of the gradient-balanced weight: the R-free optimum sat near 1/500 at 2.6 A and near 1/50 at 1.1-1.7 A. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d444fb03f610db70903de04f53f0489a71540b37)	2026-10-25 07:20:32 +01:00
leonarski_f	a3082c1500	DensityBlobs: an ion is called by its anomalous peak alone; ligand size scales gentler with resolution ... Measured by stripping the deposited ligands and ions of 12 open-arm models (placement-refined, then the blobs of mFo-DFc): ligands with a blob 34/39, called LIGAND 26/39, LIGAND blobs on a removed ligand or ion 24/33; ions with a blob 23/40. The ligand-size threshold was four resolution spheres, which called no glycan at 3.3 A and every ion at 1.1 A; it is now one and a half, and never under 8 A^3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8e66d58ba87c35119df3ba8779ca193053f5cf00)	2026-10-25 07:20:32 +01:00
leonarski_f	7759643d4c	rugnux --refine: refine the placed model, list its difference-map blobs, report under REFINE_ keys ... Opt-in. After the model's placement and everything it decides (setting, indexing, enantiomorph, MR, twin choice), the placed model is refined against the merge it was placed on (RefineModel) and the blobs of the refined mFo-DFc map are listed (FindDensityBlobs). Nothing reads the outcome: the reflection files, the space group and R_WORK/R_FREE of the model as given are those of a run without --refine. Writes <prefix>_refined_model.cif/.pdb, <prefix>_refined_maps.mtz (FP FC PHIC FWT PHWT DELFWT PHDELWT FOM FREE) and <prefix>_blobs.json; the report's model section gains REFINE_* keys (start, placement and restrained R-factors, rmsd bonds/angles, clashes, fixed residues, weights, blob counts by kind) and the five strongest blobs. Documented in RUGNUX_MODEL_AND_REFERENCE.md and CPU_DATA_ANALYSIS.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit aaee92711a205bf6a575a47c4db4555d34e7e8fe)	2026-10-25 07:20:32 +01:00
leonarski_f	b4b68d368f	Restrained refinement on the GPU: ML amplitude target, FFT gradient, dictionary restraints, R1 + R2 driver, blobs ... RefineModel (Refinement.h) refines a placed model against the merge, reading and writing nothing: - placement stage (R1): every chain of 50+ atoms as a rigid body at once (L-BFGS on six parameters per chain, their gradient from the per-atom one through Ceres jets), then one B shift per residue; - restrained stage (R2): positions and isotropic B of every atom of a residue the embedded dictionary describes, the rest held fixed but kept in Fcalc and the mask; riding hydrogens added by gemmi's topology, moved with their parents, re-placed every macro-cycle. Macro-cycles refit the mask, the scale and bulk solvent (FitModelScale / ModelScaleGPUPool) and the likelihood's D and beta (ML on the free set in resolution bins), rebuild the repulsion pairs (symmetry mates included), then L-BFGS on positions and, where chosen, on B. - weights: a ladder of full refinements at multiples of the gradient-balanced X-ray weight, chosen by R-free within bond/angle bounds (ties to the tighter geometry), then the B refinement tried at its own weights and kept only where it beats B held by more than the tie. The target and its gradient (RefinementXray, RefinementGPU): the Rice/centric amplitude likelihood (RefinementMath.h, shared host/device code), the gradient by Agarwal's FFT method - the adjoint of the reciprocal-space symmetry composition into a half-l (CPU) or half-u (cuFFT) spectrum, one inverse real transform, then each atom's density derivatives (xyz and B) summed over its box. On the GPU everything per evaluation stays on the device: ModelDensityGPU's gather, r2c, a fused compose + likelihood kernel, a CSR gather for the adjoint (no atomics), c2r, one warp per atom for the gradient; the restraints are one kernel per restraint plus a per-atom gather of gradient slots; cub reductions. Deterministic. Restraints (RefinementRestraints): bonds, angles, torsions (with esd, no phi/psi, no alternative sugar puckers), chiral volumes, least-squares planes from gemmi's Topo; repulsions beyond 1-4 (vdW radii from the dictionary's energy library, 2.6 A for H-bond pairs, 1.9 A metal-N/O/S); B similarity of bonded and angle-related atoms. DensityBlobs (D1): gemmi's flood-fill blobs of mFo-DFc above 3 sigma peaking above 4 sigma, placed beside the model and labelled WATER / ION / LIGAND / MODEL_ERROR by distance, contacts, volume and the anomalous map. tools/rugnux_refine_test refines a model against a finished run's p.mtz (developer utility, not installed). Tests: likelihood derivative, the embedded library, restraint values on ideal residues and every restraint kind's gradient against finite differences (CPU, and the GPU against the CPU), the X-ray gradient against finite differences in P1, P212121, C2 and P6122 (CPU and GPU), and a synthetic refinement that recovers a shaken model. Credits in docs/ACKNOWLEDGEMENT.md and at the algorithms. Measured on the open arm's battery outputs against REFMAC 5.8, 10 cycles, same p.mtz, free set and start model (R-free): 5reo 0.2228 vs 0.2236, 8dyz 0.1733 vs 0.1651, 9chw 0.1995 vs 0.1969 (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8efe6f2322b3bfdae580bb34f256e67730dfc3c2)	2026-10-25 07:20:32 +01:00
leonarski_f	e58453a693	Refinement restraints: the CCP4 Monomer Library subset embedded in rugnux, read through gemmi's MonLib ... A restraint dictionary carried in the binary, so refinement reads nothing from disk and needs no CLIBD_MON: the 20 amino acids and MSE, the DNA/RNA nucleotides, ATP/ADP/AMP/GTP/GDP/GMP, DMSO, PEG fragments (PEG, PGE, PG4, 1PE), GOL, EDO, SO4, PO4, ACT, MPD, water and common ions - copied verbatim from the CCP4 Monomer Library (LGPL-3.0) - plus links_and_mods.cif cut down to the links and modifications those compounds can take (peptide, cis-peptide, proline, N-methyl, disulfide, nucleic-acid backbone, termini, metal-Cys/His) and ener_lib.cif cut to its atom-type table. monomers/update_monomers.py holds the compound list and regenerates the subset; CMake embeds every .cif of the directory as string pieces under MSVC's literal limit; EmbeddedMonomerLibrary() parses them from memory into a gemmi::MonLib once per process. About 0.7 MB of text in the binary. gemmi v0.7.5's monlib.cpp, topo.cpp and riding_h.cpp vendored verbatim (prepare_topology). Licence text in licenses/, rows and a note in THIRD_PARTY_NOTICES.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 38034718052c4a300691393b5b817f476c83dfc2)	2026-10-25 07:20:32 +01:00
leonarski_f	8ff2abf3aa	docs: a standalone front page with the three programs and the downloads, the Sphinx tree one page below it ... docs/landing/ is a self-contained page (HTML, CSS, a little JS, no framework): a hero, a card per program, the download section with the archives of the newest release and the reader's own platform marked, the licence in one paragraph, light and dark. The jfjoch_downloads extension renders it into the HTML output root as index.html at the end of every Sphinx build, so it is served wherever the documentation is - Read the Docs, the gitea-pages branch and the broker's frontend bundle - and its download buttons are the same ones as on the Sphinx download pages, built from the configured release and repointed at the newest published one by the shared script. The Sphinx root page moves to documentation.html (root_doc), and the frontend's documentation frame opens that page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 64cde61f0b941b3e8c99eaa9b74afeaa75aefc78)	2026-10-25 07:20:29 +01:00
leonarski_f	9c543f60b6	docs: home page with the three programs, download pages for rugnux and the viewer, technical reference grouped below ... The home page introduces Rugnux, the viewer and the acquisition system in three cards, with the licence stated once, and the sidebar is regrouped by program: what a user of Rugnux or the viewer needs comes first, and the algorithms, formats, interfaces, hardware and FPGA pages sit under one collapsed "Technical reference" hub (TECHNICAL.md). No page moves, so every existing URL stays. Download pages (RUGNUX_DOWNLOAD.md, VIEWER_DOWNLOAD.md) are rendered by a small Sphinx extension (docs/_ext/jfjoch_downloads.py) from the `release` in conf.py, which update_version.sh rewrites, so no version string on them is maintained by hand; a script then asks the Gitea API for the newest published release and repoints the buttons at its assets, naming any archive that release does not carry. Without network access the build-time links stand. JUNGFRAUJOCH.md is the landing page of the acquisition side with its install pointers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 050442592504ce879dc51d78b6d3d1c58268a800)	2026-10-25 07:20:28 +01:00
leonarski_f	2fb4b1d015	Rugnux live page: a hero panel with resolution, ISa, space group and whether the model was placed ... Under the header of the Rugnux live page, four big-number cards in the manner of jfjoch_viewer's merge statistics window: Resolution (d_min of the merge), ISa, Space group (short symbol, screw axes subscripted) and the model. The model card shows the fit over random orientations - R-work of the placed model below its random-orientation null, in sd - which is the test rugnux places a model by, both for molecular replacement (after the TFZ/LLG-Z gate) and for a model that fits where it is. The card is green where rugnux placed the model (placement MR or RIGID_BODY), red where it did not (with MR's reason), grey where no model was uploaded; no threshold of its own. "—" while the run is going, "searching…" during the MR step. /rugnux/status gains `space_group`, `d_min` and `isa` (set with the result; absent until then, isa also where the error model did not determine it), filled from ProcessResult in RugnuxThread with the values the summary's Space group, Resolution and Signal rows print. Generated code regenerated from jfjoch_api.yaml: broker/gen (cpp-pistache-server 7.20.0), the TypeScript client (npm run openapi) and broker/redoc-static.html. Python client not regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit fcc019707751aab9d8c9f349309eeadcf364b9c2)	2026-10-25 07:20:28 +01:00
leonarski_f	76b0229d82	rugnux: the pinned host pool is handed back by the process that outlives the run, after the results are out ... Rugnux::Run ended by giving the pinned buffers its engines kept for each other back to the driver (cudaFreeHost of up to 512 MB, 0.07-0.12 s on a quiet workstation, 0.13-0.37 s loaded) - after the last result was made and before the caller could show it. Now Rugnux::ReleasePinnedPool() is called by the broker once the live status holds the result (and the run itself is destroyed after that, not before), and by the viewer once it has emitted its result; the CLI, which exits after writing its files, leaves it to process exit. Results unchanged; the broker's last image -> Done is shorter by that time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit b80f6bc3280be9e55a86cd8036f7db83a5742c15)	2026-10-25 07:20:25 +01:00
leonarski_f	3c73b469f5	rugnux: TIMING marks for the rest of the rotation tail - ingest sub-steps, the analyses, the pass's teardown and the pinned-pool release ... The tail after the last image had unmarked gaps: the device ingest's second upload, geometry smoothing and partiality recompute were one "smoothing" mark; the analyses step was one mark although the anisotropy future, not tNCS, is what it waits on; and the time between "Processed N images" and the end of the run (the pass's locals going, the pinned pool handed back) had none. Logging only: outputs unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 4ce9e5d6d489324a759f155ad9718271c52ba86b)	2026-10-25 07:20:25 +01:00
leonarski_f	78d3b7ef75	Stills geometry: fit only the given cell's crystal form; a stray frame is no cell population ... RefineStillsGeometry fitted distance, beam and cell on the strongest sampled frames whatever their cell, so a sample holding two crystal forms put the cell, and with it the distance, between them - while the merge later keeps only the given cell's population (GivenCellPopulation). The sampled frames are now filtered by the same population before the bundle and the module placement are formed, the population told from the strong frames (>= 20 indexed spots), whose cells show the valley between two forms; a weak frame's cell at the start geometry spreads across it. A run of one form has no valley and keeps every frame. GivenCellPopulation: a stray frame or two far out in a population's tail is a local maximum of the histogram with no valley of its own, so its range was everything, both forms included, and its mean - the mean of all frames - could sit nearer the given cell than the given form's own and win. A population is now represented only by its highest peak. Test CellPopulation_StrayFramesBeyondTheTail (fails before, passes after). Still battery, against the same code without the change: the two runs of a two-form sample -0.0019 and -0.0015 R-free (0-cycle REFMAC; the second repeated three times, -0.0013..-0.0021, run noise ~0.001), R_split 10.6 -> 9.8 % on the first; single-form controls fit the identical geometry (R-free within 0.0003). Rotation does not call either path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ada1b08003fa9748175cce381fc8f28569f28296)	2026-10-25 07:20:25 +01:00
leonarski_f	c62a67975e	Polarization plane: measured on the pooled background, refused beyond what a source can give, logged, and kept across passes ... The pre-scan's plane test read per-pixel sector medians of the mean projection. On a large photon-counting detector with a background of a fraction of a count per pixel over the 60 frames read, those medians were zero and the test returned no harmonic at all: on 21 of 255 battery sweeps (EIGER2 16M and JUNGFRAU 4M sweeps, the laboratory EIGER sweeps, the small-molecule sweeps at 20-25 keV) the correlation was NaN or within 0.01 of zero, so the plane could not be decided there. The background is now pooled over 9 x 9 pixels first, as the mask itself already is; 216 of 244 synchrotron sweeps then show their (horizontal) plane past both gates, against 159 of 229 before. Pooling also lifted one long-distance sweep (500 mm, 2theta under 21 deg, polarization can move its background by at most 4 %) over the correlation gate: its background carries a 10 % cos 2phi harmonic along image y, 2.9 times what a fully polarized source would give, on a horizontally polarized beamline. No polarized source exceeds a scale of 1, so a harmonic of twice that or more now decides nothing (PLANE_MODULATION_MAX). The pre-scan logs the correlation and the scale on every run that detects the beam stop (GetPlaneEvidence). A turned plane is kept on the run and re-applied where a pass starts again from an experiment taken before the decision while the shadow is already in the mask (the rotation measured-centre arm, the no-lattice pass), which previously went on with the stated plane. No battery sweep turns; where the plane is not turned the applied coefficient is unchanged. The degree is still never fitted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 50a145a4f1ddf6d58332a8ab745627dc36a2dc3d)	2026-10-25 07:20:25 +01:00
leonarski_f	6ee1a5759e	rugnux: a rotation scale is integrated with pass 1's lattice refit at it, and walked to the refit's fixed point ... Two defects let a stage turning 1.9% slower than its stored angles cost 9qw8 half its signal (R-free on the depositor's flags 0.346 against their 0.251, ISa 5.2, c 100.24 against the deposited 100.92): - Pass 1's lattice, integrated wherever the refined-geometry re-index finds less (Pass1Fallback), kept the cell and orientation pass 1 fitted at the stored angles. The rate error sits partly in that cell, so the adopted scale was integrated with a cell that undoes it. The lattice is now refit (cell + orientation, detector and axis held, no re-index) on the validation frames of the whole sweep at the angles the pass runs at, and kept where it indexes no fewer of them. - The walk took a refit step only where the validation frames significantly preferred it. A first pass indexes through a degree of drift late in the sweep, so the frames score 0.98772 and 0.98198 alike while integration at the first loses half the signal. Off the stored angles a refit step is now refused only where the frames score it significantly worse; leaving the stored angles still needs significant evidence. 9qw8 (CPU, battery protocol): walk 0.98772 -> 0.98087, c 100.94, ISa 5.2 -> 11.2, R_meas 0.270 -> 0.157, CC1/2 0.975 -> 0.993, R-free on the depositor's flags 0.346 -> 0.267 (ratio 1.381 -> 1.066; forcing --rotation-scale 0.9816 gives 0.255). The other eight sets that adopt a scale (2wnq, 2wnn, 6ttn, 8r5r, 8pqd, 8xte, 9i80, 9gdj) move by at most 0.0006 in R-free; the photon-counting controls lyso_x06da_ref, thau_x10sa_16keV, cytc_x10sa, 5reo and 3mc4/3r6o are unchanged (lyso p.hkl byte-identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 5daa4ed635b22e29b25e3e02f8ad8c058f9176a2)	2026-10-25 07:20:25 +01:00
leonarski_f	fca2a77b0c	Scaling: the low-resolution limit bounds the scale fit, not the merge ... Reflections coarser than --scaling-low-resolution (default 50 A) are now merged and written - what maps and refinement want of the lowest resolutions - but stay out of every fit of the scale model, by the same path an ice-ring observation takes (excluded from scaling, kept for merging), under a name of its own: RotationScaleMerge's per-observation flag is now `unscaled` (UNSCALED_ICE | UNSCALED_LOW_RES; on the device OBS_UNSCALED = OBS_ON_ICE_FLAG | OBS_LOW_RES_FLAG), set from the reflection on the host and in the device ingest, and only its ice bit marks a merged reflection as on an ice ring. The stills merge drops its low-resolution cut; ScaleOnTheFly, StillsPartialityRefine and the reindexing correlation keep it. The merge statistics run over the whole sphere unless --report-resolution declares a range, so the reflections the beam stop hides count as missing. Which coarse reflections exist at all is the beam-stop shadow test's decision (a7610d847). Why not simply lift the limit: on 5nw5, 35 reflections beyond 50 A bent the scale model enough to move the merged 20-50 A intensities by +61 % and 10-20 A by +19 % (Wilson B 319 -> 382, REFMAC R-free +0.006). Kept out of the fit, the merged intensities at 10-50 A are unchanged to 0.2 % and Wilson B is 307. CPU subset battery against the previous commit (lp-cpu-v2 / lp-cpu-v1): REFMAC 0-cycle R-free 5nw5 +0.0004, 6oel +0.0005, others within 0.0001; rugnux's own R-free/R_model rise where the new >50 A terms enter it (lyso R_model 0.2714 -> 0.2739; the model is poorest there). Kept beyond 50 A: 7qij 61, 5nw5 35, 6oel 14 unique. In-house photon-counting sets unchanged, except insu_I_x06da_ref, where the delta-CC1/2 frame exclusion now removes 449 instead of 78 frames: refres R_meas 0.246 -> 0.167, ISa 33.5 -> 35.4, R_free 0.2443 -> 0.2427, anomalous signal 3.31 -> 3.38 (XDS: R_meas 0.349, ISa 25.1). Stills (lcls242_s/t, lcls222, euxfel83, sacla236) within 0.0002 of the previous commit. GPU (device ingest) and CPU give the same 5nw5 merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 32a9fbcb3ca67879ceb051c13f58d61fe85530f4)	2026-10-25 07:20:21 +01:00
leonarski_f	e97b902603	Beam stop: a reflection whose signal disk reaches the shadow is not integrated; no-data pixels are dead, not shadow ... The profile fit keeps a signal disk that lost pixels to the mask and fits the pixels that remain. That holds for a gap or a dead pixel, but not for the beam-stop shadow: the mask is drawn where the background is dimmed, and the Bragg half-shadow does not end there, so the rest of such a disk is attenuated too. Measured on summed rocking events against their symmetry mates (stratified by resolution and zeta, the detector edge excluded), observations whose disk touches the shadow read low by 12-60 % on 5uth, 7qij, 5nw5, 3r6o, 3meb (CCD and PILATUS alike) and level on the in-house photon-counting sweeps; observations with a clean disk are unbiased except on 5nw5, whose half-shadow reaches ~20 px past the mask. The test sits in MXAnalysisWithoutFPGA::IntegrateOnImage, so rotation, stills, the _process.h5 and --mode scale see the same reflections; with no shadow in the mask (the broker) it does nothing. The shadow finder also reports pixels holding a detector's no-data mark (PILATUS CBF module gaps at -1): they are the darkest pixels there are. The pre-scan now masks those as dead (bit 10) instead of beam-stop shadow (bit 9), so that the shadow test does not drop every reflection clipped by a module gap and the reported shadow share is the shadow's. A mean of zero is not taken as dead: a deep shadow on a still counts nothing in the projected frames. CPU subset battery vs rc180 (REFMAC 0-cycle R-free): 3meb -0.0004, 5uth -0.0004, 6h2p -0.0003, 6oel -0.0002, 3r6o +0.0001, 7qij/7rji 0, 5nw5 +0.002 (its lowest-shell CC1/2 0.78 -> 0.89, ISa 8.4 -> 9.3); in-house refres ISa up 0.1-0.3 on lyso/thau/insu/cytc/myob, half_image refres R_meas 75 -> 64 %, no set lost quality. Stills (lcls242_s/t, lcls222, euxfel83, sacla236): R-free within +-0.002, the noise of a perturbed run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a7610d847694b5540dacc9ef609344ea7c7ae8fd)	2026-10-25 07:20:21 +01:00
leonarski_f	925701a1ce	ScalingSettings::MinPartiality: validate the argument, not the value it replaces ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit df5f2ae22f979a5569f4d569fe2a8cdd5999d242)	2026-10-25 07:20:21 +01:00
leonarski_f	2dbb9d5951	Stills partiality: merge only partials of p >= 0.5, adopt the refined model on every crystal, centre the rocking curve on the run's radial scale ... Three changes to the stills post-refinement and merge, decided on 0-cycle REFMAC R-free against the deposited models (diff = ours - deposited data, noise ~0.002): - Stills merge only observations whose model partiality is at least 0.5 (RugnuxDefaultScalingSettings; rotation keeps 0.02, and --min-partiality still overrides). Diagnosis on per-observation dumps: the empirical rocking curve has a floor of 5-15 % of its peak out on the tail (mis-oriented crystals), and dividing such observations by a small model p makes them biased, not noisy. Variance terms in (1-p) or (1-p)/p recovered a third of what the cut does; a narrower (correct-looking angular) width or a Lorentzian form was worse. On one set in scale mode the optimum is flat over 0.3-0.6. - The CC adoption gate in RefineOne is gone: it left 25-33 % of crystals at p = 1 next to the refined ones, and collapsed to 3 % on a run whose curve is off-centre. - The run's radial scale of the reciprocal lattice (a wavelength or cell a little off, invisible to spot positions since they fix only lambda d*) is measured from where the curve is centred against d*^2 (MeasureRadialScale, iterated three times) and applied to q in the width measurement and the post-refinement. One run had its curve centred at +2-3e-3 d*^2 (scale -1.6e-3 measured); the others measure below 7e-4. --mode scale on stills now starts where the full pipeline starts: p = 1, the excitation error recomputed from the stored lattice (the window was read as zero before, so no run width was ever measured), and collapsed scales rejected after the refinement. Still battery, the 12 non-float sets, change of diff vs the previous build: -0.0035 mean, 9 better, three within +-0.0006 except one +0.0053 (an edge win becoming a draw); the off-centre run -0.005 here and -0.017 in an earlier run of the same algorithm (that set varies by ~0.01 between runs). Rotation p.hkl byte-identical on three sets. Test StillsPartialityRefine_MeasureRadialScale added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0d85e5b975ba863828bd17ebc3b5308e9aa294ce)	2026-10-25 07:20:21 +01:00
leonarski_f	e374ea269f	Rugnux live page: MR scores and the placed model right under the timeline, carved difference maps ... - Fo-Fc (and the anomalous map) are carved within the same 4.5 A of the atoms as 2Fo-Fc, not only cropped to the model's box + 6 A, and Fo-Fc starts at +-3.5 sigma: uncarved at 3 sigma it filled the frame with noise peaks. A departure from AareDB's viewer, which carves 2Fo-Fc only. - The anomalous map is on by default only where the status says the data carry anomalous signal. - The model section (molecular-replacement scores + viewer) sits directly under the timeline and slides in when the run ends with a model; the stage stays dark until model and maps are in, then fades up while the camera eases in from further out and turned ~35 deg (1.6 s, ends exactly at the reset view). prefers-reduced-motion turns all of it off. - The page uses up to 2400 px instead of the 1300 px column; Processing and Result are 1:2, the summary tiles flow into as many columns as fit, the viewer is clamp(520px, 64vh, 820px) tall. - Score tiles: placement (copies, group, MR time), TFZ (RFZ, LLG-Z, search resolution), R-free (R-work, unrefined), CC(model, data), fit over random orientations, 2Fo-Fc at atoms, and the strongest anomalous peak only where there is anomalous signal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3bb891c60acd2c5618bbc461fabcb7513f5df947)	2026-10-24 19:06:04 +02:00
leonarski_f	128830d15e	rugnux live status: the model's scores, and whether the data carry anomalous signal ... /rugnux/status gains `model` (rugnux_live_model) - placement (RIGID_BODY/MR/NOT_PLACED), R-work and R-free of the placed, unrefined model, CC(model, data), R-work over its random-orientation null, mean 2mFo-DFc at the atoms, the rigid body's move, molecular replacement's decision, group, copies, TFZ/RFZ/LLG-Z and search resolution, and the strongest anomalous site - taken from ProcessResult::model_validation and ::model_prior (LiveModel). `anomalous_signal` says the summary's "Anomalous" row reports usable signal (CC_anom > CC_ANOM_SIGNAL, now a named constant shared with the report), so the viewer can show the anomalous map by default only then. The step that placed the model (Molecular replacement, or Model validation) now carries notes: what MR placed and its TFZ/RFZ, R-work/R-free (unrefined) and CC(model, data), the rigid-body move. Generated code regenerated from jfjoch_api.yaml: broker/gen (cpp-pistache-server), the TypeScript client (npm run openapi) and broker/redoc-static.html. Python client not regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f602e931dc6afcccb3767b8c6df87650d9496033)	2026-10-24 19:06:04 +02:00
leonarski_f	789e064e6a	Stills scaling: a model error in the per-crystal scale fit, and the rocking width measured per window group ... Two changes to the stills scale model; rotation is untouched (p.hkl md5 identical on two rotation sets). The per-crystal scale fits (ScaleOnTheFly and the partiality post-refinement's G and tilt) measured residuals in counting sigmas. The strong reflections of a still scatter about the model by tens of percent (robust sd of ln(I/<I>) 0.45-0.8 by shell on one run), so with a Cauchy loss at 3 counting sigma every strong reflection was an outlier and the scale was set by the weak ones: re-fitting each frame's scale against the final merge moved it by a factor 0.45-1.5 (5-95%). The fit's variance is now the expected one at the model, background + counting + (0.3 * model)^2. The coefficient is not delicate: 0.1-1.0 fit alike in an offline replay; what matters is that the term is there. The run's rocking width came from the intensity-weighted moment of the excitation error over all frames pooled. Frames predict through windows of their own that differ by an order of magnitude, and the widest windows dominated both moments: on a run whose windows are mostly narrower than the curve at high resolution the measurement read the window at every resolution, 2.8e-4 1/A with no angular term, several times too wide at low resolution (low-partiality observations there merged at 0.1-0.3 of their reflection's mean). The moment is now taken per group of frames of similar window (4 per shell), each against its own window, and the shell takes the median of the groups whose curve is narrower than their window. On that run the width becomes 6.7e-5 1/A + 0.018 deg angular; where the pooled measurement was already right (a SASE-bandwidth run), it stays where it was. 0-cycle REFMAC R-free against the deposited models, 9 still battery sets (6 used while developing, 3 held out), change from rc179: -0.0100, -0.0095, -0.0057, -0.0055, -0.0051, -0.0045, -0.0030, -0.0024, +0.0009 (held out: -0.0055, -0.0051, +0.0009). The width change alone accounts for most of the largest gain; the scale-fit variance for 0.000-0.003 per set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ccc0e01f56827bbe0648ed7c5b8049912d80650f)	2026-10-24 16:57:12 +02:00
leonarski_fandClaude Opus 5.5	672a375b31	v1.0.0-rc.180: VERSION and update_version.sh ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-24 16:57:12 +02:00