9 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	46da7fb359	Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons ... The pre-scan measures the gain of an SMV/marCCD detector (background variance per unit mean) but the integrator used it only for the background clip: sigma, var_bkg, the strong-spot flag (I/sigma >= 5) and the profile-vs-box-sum guard (10 box sigmas) all treated one ADU as one photon, so on a Saturn (g 4.7-5.8) the guard sat at ~4.5 true sigma. The new BraggIntegrationSettings::PixelGain (set by rugnux from the same measurement, rotation only, at max(1, g)) takes every variance the engine forms at the gain, and Finalize converts I, sigma, bkg and var_bkg to photons, which is what the merge's variance model (var_bkg + I) assumes. CPU and CUDA engines; Metal runs the CPU code for this. Gain 1 (every photon-counting input, ADSC/marCCD whose measured ratio is below 1, and stills) is bit-identical. Battery --only, 12 open-arm CCD/background-dominated sets vs the base build (987e812e2), GPU: the six Saturn sets (g 4.7-5.8) all improve - guard fallbacks 4.1-17.3% -> 1.2-6.1%, anomalous peak height 3mc4 x1.10, 3p85 x1.02, 3r6o x1.06, 5cc8 x1.03, 5uth x1.05, 5vml x1.04; R_meas 0.880-0.993x on all six; R_model shell-scaled better on 5/6 (5cc8 +0.0005); ISa 5uth +1.5%, 5vml +15%, 3p85 +3%, the twinned 3mc4/3r6o -5%/-1%. 6cee, 6i3j, 6v2r, 7kcn, 8v4j, 9jzo unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-23 01:26:44 +02:00
leonarski_fandClaude Opus 5.5	5905d34cba	Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed ... The lower bound counts each saturated pixel at the saturation limit, so it needs one. An observation overloaded only by pixels unreadable on this frame (a per-frame bad-pixel flag, data value 0) was kept as a box sum with those pixels counted as zero, which bounds nothing; it now stays overloaded and is dropped, per observation. With the stuck pixels masked by the pre-scan, the run-level gate (kept overloads must be enriched coarser than 4 A) has nothing left to catch: without it every still set gives the gated result (clipped CCD and Rayonix sets identical to the fourth decimal, photon-counting sets unchanged), so it is removed with Reflection::overload_kept. A neighbour-brightness test per saturated pixel was tried and not kept: after the mask it only dropped 9-20% of the Rayonix overloads and worsened R_split on one set (0.245 -> 0.278). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-22 17:27:44 +02:00
leonarski_fandClaude Opus 5.5	6b5647e8ab	Merge stills-sat into stills-easy ... The one conflict, tests/BraggIntegrationEngineCPUTest.cpp, where both branches add cases at the same place: integ-pal's side is taken and stills-sat's test diff applied on top, which keeps all four cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-22 12:46:00 +02:00
leonarski_fandClaude Opus 5.5	72cc542d78	Stills keep an overloaded observation as a lower bound instead of dropping it ... The stills merge dropped every observation with a saturated pixel. On a still the saturated observations are the fully recorded ones of the strongest low-resolution reflections, so what survived were their partial observations, whose partiality correction then set the merged intensity: on clipped CCD stills the merged low-resolution intensities came out at 0.4-0.6x of the deposited ones, a ramp no Wilson B absorbs, invisible to CC1/2 and R_split. The pixels are really clipped (flat-topped plateaus a few per mille wide, a pile-up in the pixel-value histogram from the per-pixel ADC limit), so the auto ceiling is right; the drop was the defect. A still now keeps the observation: the box sum with every saturated pixel counted at the saturation limit, unflagged and marked clipped, counted in BraggIntegrationCounts and reported as OVERLOADED_REFLECTIONS_KEPT. Rotation data (IsRotationIndexing) are unchanged and keep the flag for the rocking-event drop. Measured on a clipped-CCD stills subset (dimple R-free, deposited free set): dropped 0.247, lower bound 0.230 (same as switching saturation off). A Kabsch fit to the unsaturated wings read lower than the lower bound (0.237), and inflating the lower bound's sigma lost (0.255). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-22 10:36:33 +02:00
leonarski_fandClaude Opus 5.5	61def00a1a	Stills: background clip follows the radial curve; signal disk from the lattice spacing ... Two stills-only changes to the integration stencil. 1. Where the whole-frame radial curve is in use (the correction on), the ring's high-side clip follows it: a ring pixel is clipped against the threshold plus the curve's rise from the reflection's radius to its own (BraggIntegrationEngine::RadialClipOffset, rounded to whole counts so CPU, CUDA and Metal keep the same pixels). On a background falling steeply away from the beam the flat mean + 4 sqrt(mean) threshold cut the beam-side half of the ring as outliers, so the background read low and reflections near the beam stop read 3-150x too strong. The empty-aperture test's corrected column clips the same way, so the per-run correction decision sees it. Measured with r1 6 (dimple R_free, battery arguments, all frames): lcls222 0.196 -> 0.1856, pal86 0.216 -> 0.1997, sacla236 0.2559 -> 0.2559, alvra213_int16 0.1986 -> 0.1986. 2. The signal disk is taken from the lattice: it must not reach into the next node's spot. The closest two nodes can come on the detector is the shortest reciprocal-lattice vector the centring allows, s = F_px tan(2 asin(lambda / 2 d_max)), and a spot reaches 2 r80 (measured in the pre-scan); r1 = the largest of 4, 5, 6 px not above s - 2 r80, never below 4, never above the disk the run starts with (6). The empty apertures are measured for each candidate disk and the ring is chosen on the chosen disk's tally. Pixel detectors keep 6 (sacla236 s 22 px, alvra213 20 px); the binned CCDs with a long axis shrink to 4 (lcls222 5.4 px, pal86 6.7 px), which the radius scans there measured as the optimum; pal90 (s 21 px) and lcls242_s (15.5 px) keep 6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-22 09:20:52 +02:00
leonarski_fandClaude Opus 5.5	e8145d8d2f	Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame ... Two changes to the stills integration background, both stills-only (rotation keeps its path bit for bit): 1. The r2..r3 background ring's outer radius is measured per run. The pre-scan lays apertures on a grid over its frames, away from every found spot, and for each candidate r3 (r2+2 .. 14) takes the mean square difference between the r1 disk mean and the clipped ring mean. With nothing in the aperture that is exactly the background error the ring makes (noise + bias); the run keeps the minimum. On a photon-counting pixel detector the noise wins and the shipped r3=14 is kept; on a binned CCD the background structure wins and the narrowest ring is taken. r1 stays 6: a smaller disk (4,6,14) costs 0.0013 dimple R_free on a JUNGFRAU set, so it does not pass the photon-counting gate. --adaptive-integration-radius now defaults on for stills too (=off restores the fixed 6/8/14). The pre-scan bandwidth estimate is not applied to stills. 2. The radial background-curvature correction on stills takes its curve from the adaptive spot finder's peak-excluded per-ring background (whole frame) instead of the predicted reflections' own ring pixels, and auto applies it to every stills image (no ice-score gate: a solvent/carrier hump curves the background as much as ice). New engine call RadialBackgroundFromRings; CPU, GPU and Metal read the handed-over curve. The GPU box sum now sizes the radial bin index separately from accumulating, so a handed-over curve is read at the right radius. Measured (dimple R_free, deposited free set, battery arguments, all frames unless noted): pal86 (Rayonix 4x4, 156 um): 0.354 -> 0.218 (ring 10; at 4,6,10 0.213) lcls222 (Rayonix 4x4, 177 um): 0.205 -> 0.196 (ring 10) sacla236 (MPCCD): 0.2588 -> 0.2558 (ring 11) alvra213_int16 (JUNGFRAU): 0.1986 -> 0.1986 (ring 14, unchanged) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 18:10:08 +02:00
leonarski_fandFilip Leonarski	19b135186d	v1.0.0-rc.175 (#85) ... * Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`). * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS). * Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead. * Rugnux: On macOS, Bragg integration runs on the Apple GPU. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. * Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`. * Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. * Rugnux: The beam-stop shadow is found correctly on tilted detectors. * Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement. * jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps. * jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images. * Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. --------- Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home> Reviewed-on: #85 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-16 15:34:43 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00