7 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	f199f23c54	Radial background: one mechanism for stills and rotation, decided per run on empty apertures (D5) ... Wherever the correction is set for the whole run its curve is the frame's peak-excluded ring background (the adaptive spot finder's, whole frame) and the ring's clip follows it - as stills have done since integ-pal - now rotation too. One empty-aperture measurement (BackgroundAperture) decides it for both workflows: - The apertures' corrected difference is now taken as the integrator applies the correction: the clip follows the curve (each ring pixel's threshold raised by the curve's rise from the centre, in whole counts) as well as the kernel correction. Before, the tally clipped flat, so it measured a correction the engine never applies. - Rotation measures the stencils the spot width can give it (r1 = 4, 5, 6 with the ring-area rule, RotationStencil) and decides on the one it gets, and separately on the pre-adaptive stencil the geometry pre-pass and the starvation guard return to. - Stills: on wherever the corrected ring predicts the empty disk better (unchanged rule). Rotation: the frame's curve on every image only where it takes >= 10% of the apertures' mean square error away (ROTATION_RADIAL_MIN_GAIN); otherwise the existing per-image ice gate with the reflections' curve is left exactly as it was. Why a margin on rotation only: a still has no other correction, a sweep has one. Measured over 18 rotation sets (forced on every image): photon-counting sweeps gain 0.1-5% on the apertures (z 3-4) and lose ISa/anomalous signal when corrected (insulin, lysozyme, cytochrome c; 9yzk R-free +0.009 at 1.2%), as the earlier stills-rule trial found; the home-source CCD and smooth-ice runs gain 13-31%. Result (CPU A/B vs 87684012f, 18 rotation sets): 12 .hkl byte-identical (every photon-counting control without ice), 5uth R-free 0.210 -> 0.201 / ISa 10.4 -> 10.8, the ice set neutral; stills: lcls242_s turns the correction on (the clip-following tally), R-free 0.2399 -> 0.2406. No integration kernel changed; the tally is host code. Tests: BackgroundAperture_CorrectedClipFollowsTheCurve, RadialHelps margins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dd3f3f2da39f5338e7dda1874bbdd535229bce98)	2026-10-24 07:27:12 +02:00
leonarski_f	711f902aa7	One gain mechanism: frames analysed in their own units, the gain applied where a photon count is assumed ... The gain of an input in detector units (SMV, marCCD, XTC, the XFEL formats) is measured once, before anything reads the frames (RugnuxMeasureGain: photon-peak spacing and background variance per unit mean), or given with --gain, and handed to the integration and the spot finding by one function (ApplyRugnuxPixelGain). The rotation pre-scan no longer measures it, and the stills reader no longer converts frames of a detector whose photons are not resolved: SACLA MPCCD, Rayonix and CCD frames are integrated in ADU at the gain, rotation and stills alike. Where a photon count was assumed, it is now taken at the gain (bit-identical at gain 1): - integration: the background clip's one-photon floor, the sigma floor (one photon), the profile fit's background floor (0.01 photon), and a still's overload kept as a lower bound, which was left in ADU while every other reflection left the integrator in photons - the cause of the earlier loss of the engine route on clipped CCD stills (merged low resolution inflated by the gain, the rocking-width fit's bandwidth collapsing); CPU and CUDA, Metal through the shared CPU code; - the adaptive spot threshold (Poisson arm and its one-photon floor), CPU and GPU; - the pre-scan spot-width measurement (strong-spot SNR, the ring clip) behind the stills bandwidth and the rotation signal radius; - the hot/dead pixel tests, CPU and GPU; the empty-aperture ring clip. BackgroundPixelGain is gone: one pixel_gain. A detector whose background resolves single photons (photon peaks: JUNGFRAU in keV, AGIPD) is a photon counter and its frames are still counted in whole photons as read: analysed in keV at the gain instead, a photon-counting still set lost 0.008 R-free (fewer spots, a noisier beam-centre trial) - the rounding is the detector's own discrimination. --gain sets the value, the background still says which kind. Validation, against rc178: photon-counting and g<=1 rotation p.hkl md5-identical (lyso, thau, cytc, myob, insu, a small-molecule set, 5reo, 6cee, 6i3j, 2xfw, 6s1u, 8v4j); CCD g>1 at or within noise of rc178 (5uth R_meas 0.113 -> 0.108). Stills R-free (0-cycle REFMAC): MPCCD sets 0.184 -> 0.185 and 0.199 -> 0.196, Rayonix identical, photon-resolving sets back on their rc178 numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit da8dbfc8147354b63e375f2d9521dffb80938b6d)	2026-10-23 17:56:51 +02:00
leonarski_f	c74678b06f	Polarization plane decided once, in the pre-scan, and followed by the integration too ... The shadow finder already compares the background's cos 2phi harmonic with the profile a polarized source gives it, and applies the stated degree in the plane the background shows where it shows one clearly (correlation > 0.5, at least 0.3 of a fully polarized source's modulation). The integration's Lorentz-polarization factor still assumed the plane along lab x, so on a run whose background is suppressed along the image's y axis the two disagreed, and only a negative --polarization made them agree. ShadowFinder now reports what it applied (GetAppliedPolarization), the pre-scan writes it into the experiment (and keeps it in the beam-stop memo with the mask), and everything built after the pre-scan - the Bragg integration, the azimuthal mapping - reads the same plane. The degree is never fitted; only the plane can turn, and only where the shadow finder already turned it. Where the plane is not turned the experiment is not touched, so every other run is bit-identical: six rotation sets give the same p.hkl md5 as rc178, and the stills sets that do not turn give the same R_free, R_split, indexed fraction and d_min. On the one stills run whose plane turns, R_free and the resolution cut are the ones a stated -0.998 gave before, now from a bare invocation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f50bfe15f7dccf588bd8bdd6114a32e3ac9b36ed)	2026-10-23 17:56:51 +02:00
leonarski_fandClaude Opus 5.5	0a515bb086	rc179: everything on rc178 after its merge into main (#88), except the CHANGELOG ... The rc178 branch continued after #88 was squash-merged; this is its content from b9ada7914 (the tree #88 squashed) to 094eee31a, as one change on main. The individual commits stay on the rc178 branch: - still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes - AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets - DetectorModules: panel look-up by id, plane -> panel, plane bounds and a per-pixel walk; mask size checked - tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout - Dense per-pixel maps through the panel table (panel layout stage 1) - Panel id on spots and the panel table in the start message - Viewer and preview show a panel detector as it lies on the detector plane - Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster - Panel layout stage 2: prediction and Bragg integration through the panel table - XTC reader: frames handed up as stored, streams indexed in parallel - XFEL reader: integer frames stay integer, one gain, placement by panel - EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery" - still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed - float32_to_int16.py: a non-finite pixel becomes -32768, the int16 error marker, not a zero count - CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them - Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table - Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane - still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good - still battery: take the deposited model as mmCIF where the RCSB has no PDB-format file - still battery: dimple gets PDB-format coordinates converted from an mmCIF-only deposition - stills fixed cost: geometry refinement and pre-scan do the same work in less wall time - stills geometry sample and trial spot finding run as many workers as the image pass - Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame - StillsPartialityRefine: the partiality width is the run's, measured on the crystals, not the prediction window's - MergeOnTheFly: capture term and multiplicative outlier band for the stills merge (both opt-in) - Stills given a cell: the written cell is the mean over the frames that are merged - Stills: the radial curvature correction decided per run on the same empty apertures - StillsPartialityRefine: measure the run width only on shells with signal - Stills pre-scan: report the paired z of the radial-correction comparison - module placement: the five concurrent fits each keep every thread - still battery: score R-free against the deposited data through the same dimple protocol - still battery: the R-free target is the lower of the deposition's dimple and published values - Stills: background clip follows the radial curve; signal disk from the lattice spacing - Stills keep an overloaded observation as a lower bound instead of dropping it - Stills: drop the lattice-spacing signal disk; the correction decision back on the plain clip - battery: R-free against the deposition without refinement only, one protocol for rotation and stills - still battery: EuXFEL sets run on a 2,000-image subset for now - stills geometry refinement: fit the detector distance and the cell together - stills geometry refinement: a distance given with --detector-distance is held in the joint fit - Metal integration engine: back to its panel-s5 kernels; Mac stills keep the old clip, no saturation count - Metal integration test: the CPU comparison runs on a rotation experiment - Stills: kept overloads go back to the drop where they are not concentrated at low resolution - stills distance search: an absolute 30-3000 mm scan, a sample found without d-limits, the fit iterated - battery results: XFEL still battery 20261022-1358 on rc178 0cf62452c - 12 sets, cold, R_free unrefined - battery results: still summary gains the resolution of our merge and of the deposition - Pre-scan masks a pixel at the saturation limit on most frames, as one holding the error value - Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed - XFEL reader: a per-frame mask is read the way the frames are, its chunk inflated outside the HDF5 lock - stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes - rugnux usage: --beam-center-check describes the absolute 30-3000 mm stills distance scan, not the old 60-160 % grid - stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree - CLAUDE.md: the changelog is written once, just before the merge into main - condensed one-liners of what users get - broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4) - frontend: Mol* viewer of the placed model and its density on the Rugnux live page - writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does - rugnux live maps: one unit cell centred on the model; the viewer opens on all of it - Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer - Mol* viewer: open with the maps drawn within 10 A of the centre of view - Mol* viewer: aaredb's structure-viewer setup, on a black stage - rugnux: the gain of a stills file that states none is read off its single-photon peak where the background resolves photons - Mol* viewer: in line with AareDB's StructureView at its current head (8ece2c2) - ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI) - stills tilt decision: the modules stay where the run placed them - still battery: no distance pins and no per-set tuning flags; gain from the data except where a file's baseline hides it - battery results: rc178 6f6f69 - full still battery and full rotation battery with REFMAC - Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons - BackgroundGain: the gain of a frame that resolves photons is the spacing of its photon peaks, so a baseline left in the frame no longer hides it - ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization - BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain - ShadowFinder: measure the modulation robustly under a wide one-sided shadow - ShadowFinderTest: a one-sided shadow is not fitted as part of the background's modulation - ShadowFinder: back to the predicted polarization factor, in the plane the background shows - MXAnalysisAfterFPGA: the Bragg engine's edge mask in the image's layout, so raw / untransformed FPGA runs start again - battery results: still battery on rc178 e54f55 replaces the earlier rc178 still tables - XFELReader: per-frame mask and photon conversion without a library call or a branch per pixel - stills geometry trial: the hypotheses a search asks are indexed side by side - stills tilt decision: the fits start beside the pre-scan and the geometry trial - XFELReader: a deflated frame is inflated and converted a piece at a time, while it is in the cache - XFELReader: a byte-shuffled frame is unshuffled a piece at a time, a whole pixel per step - GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block - GeometryRefiner: a spot's residual differentiates its observed and predicted halves apart - docs: EXTERNAL_TEST_DATA lists every open-arm battery set, with corrected values and counts - battery: doc_check.py cross-checks the manifests against the dataset pages - stills module placement: the placement fit gets half the threads, each held-out fit an eighth - still battery: the EuXFEL sets run on all their frames, as the published still runs did - battery results: still battery on rc178 a0ce6c, cold on a quiet machine, replaces the earlier rc178 table - macOS build: explicit size_t in the XFEL reader's row bounds; parentheses in the CPU preprocessor's select Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-23 17:56:42 +02:00
leonarski_f	afbe844b20	v1.0.0-rc.178 (#88) ... * Rugnux: Processes serial crystallography data from XFELs - SACLA (MPCCD), PAL-XFEL and European XFEL (CXI), LCLS (XTC) and SwissFEL Alvra (JUNGFRAU with a CrystFEL module geometry) - with several files or Jungfraujoch master files merged in one run (`rugnux *_master.h5`). * Rugnux: Stills processing takes a known unit cell and space group as a starting point, finds the beam centre (and, with a cell, the detector distance) from the images, detects the saturation level in other facilities' files and refines the placement of individual detector panels. * Rugnux: The stills report gives R_split and CC1/2 from frame half-sets, comparable with CrystFEL. * jfjoch_viewer: Opens the same XFEL file formats. * Validation: A stills battery checks Rugnux against deposited structures from SACLA, PAL-XFEL, LCLS, European XFEL and SwissFEL. * Rugnux: More reliable convergence of rotation scaling, and live molecular replacement starts as soon as the model is uploaded. * jfjoch_broker: Initialize clears the previous dataset's bearer tokens, image buffer and live Rugnux status, and a slow CUDA JIT cache is reported at start. Reviewed-on: #88 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-21 11:20:01 +02:00
leonarski_f	8261b7354e	v1.0.0-rc.176 (#86) ... * jfjoch_broker: Optional live Rugnux analysis of DECTRIS rotation collections inside the broker, with merged data, maps and the placed model written beside the master file, the rotation lattice in the scan result, and a "Rugnux live" tab and dashboard row in the web frontend. * Rugnux: Faster processing, with molecular replacement and model validation spread over several GPUs and run on the Apple GPU on macOS. * Rugnux: The polarization stated in miniCBF and d*TREK headers is used, twinned crystals' space groups are decided more reliably, and a run that finds no lattice still reports every image. Reviewed-on: #86 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 07:03:28 +02:00
leonarski_fandFilip Leonarski	19b135186d	v1.0.0-rc.175 (#85) ... * Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`). * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS). * Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead. * Rugnux: On macOS, Bragg integration runs on the Apple GPU. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. * Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`. * Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. * Rugnux: The beam-stop shadow is found correctly on tilted detectors. * Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement. * jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps. * jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images. * Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. --------- Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home> Reviewed-on: #85 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-16 15:34:43 +02:00