v1.0.0-rc.178 #88
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mx/Jungfraujoch#88
Reference in new issue
Block a user
rugnux *_master.h5).The stall rule ('step not below 0.9 x its best for five rounds') called a steady slow fall a stall - 82 of the 245 loops that ended unsettled in the rc176 full battery were still falling at < 2% a round - and every unsettled loop returned its LAST state: a fulls loop that diverged from round 1 handed on the state after five rounds of walking. Now a loop is stalled when the median of the pairwise slopes of log(step) over its last six rounds (Theil-Sen) is not negative - a median, because a least-squares slope let one round's jump (the smoothness re-chosen five decades away) end a loop falling by a quarter a round - and however a loop stops (settled, stalled or at the cap) it ends on the plain round with the smallest step. On a settled loop that is the last round, so nothing changes there; on any other it is the state closest to a fixed point the loop reached. The partials' loop restores it through the round's own rescale (ratio fitted G / best G), so the corr bookkeeping of the CPU and GPU paths is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4The live rugnux run wrote its whole processing log into the broker's journal - every [Rugnux] line of the library (tables included) plus a "rugnux:" line per step and note - about 300 lines per collection, all of them results the user already gets through /rugnux/status, the end message and the written files. - Logger::ServiceLevel(service, level): every Logger of that name made afterwards keeps only that level and above. The library makes its own Logger("Rugnux") (and Logger("ScaleOnTheFly")) wherever it logs, so this is the one switch: the broker sets them to error, or to debug with the existing `verbose` setting. The per-Logger "Version" line goes with it. The rugnux CLI routes every Logger to its own sinks (RouteAll) and never calls this, so its console and _log.txt are unchanged. - The live observer feeds steps and notes to the live status only. - The run's outcome is one line at the end: state, START -> first image and frame rate, done after the last image, last image -> all written. The probe result, the warm-up time and the cut reason go to debug (the cut is in the outcome line). - Per-collection step chatter goes to debug: receiver init/teardown, thread start/stop, "Wait for receiver done" and friends, the per-image mean processing time. The JFJochBroker logger now follows `verbose` too, so these are available again with it. - Kept: Start request, measurement start, receiving started/finished, receiver efficiency (and the per-module packet counts on loss), calibration sequence, all warnings and errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4f217f180a tightened SCALING_TOLERANCE to 1e-5 for the per-frame scaling loops, but ApplyCellSurface used the same constant for its own stop. The surfaces then ran to their 30-round cap ("NOT settled"), and that - not the per-frame loops - is what lost ISa on the tol-1e-5 full battery (20261019-2106 vs 20261019-1726): on hepes, myob_x06da, 9rcs and 7bgt the per-frame loops settle on the same trajectory either way, while the modulation/absorption fits and their half-set gates changed. The surfaces' own fixed point is the worse one: run to 1e-5 with 300 rounds allowed, the detector-frame surfaces walk along the resolution direction the gauge projects out (7bgt modulation: 15 -> 283 rounds, 1.13x -> 8.47x ramp removed), their held-out half-set gain drops (hepes +0.026 -> -0.031, 6o2h -0.28 -> -0.78), and ISa falls (hepes 37.0 -> 18.3, myob_x06da 22.8 -> 19.6, 9rcs 7.4 -> 5.3, 7bgt 17.0 -> 15.3, 6o2h 31.0 -> 18.1). The surfaces' prior and gates were calibrated at 1e-3, so they get their own SURFACE_TOLERANCE = 1e-3 again; the per-frame loops stay at 1e-5. Tried and rejected (23 validation sets, runs 20261020-0040/0104/0129 cvfold-*): - leave-one-out reference in the surface fit: 7bgt R_meas 0.14 -> 0.31, R-free 0.207 -> 0.277, 9rcs CC1/2 0.994 -> 0.956. Regressing the observation on a mean that excludes it puts the rest of the group's noise in the regressor alone and attenuates the slope; the mean that includes it is matched by the same noise on both sides. - leave-one-out reference in the per-frame fit (CPU + GPU): same attenuation; 5reo R_meas 0.23 -> 0.49, 3meb 0.26 -> 0.37, myob_x06da_powder_2 d_min 1.54 -> 1.94 (aspirin and yag gained). - cross-validating the fulls smoothness on the held-out per-frame fit while scaling on the usual one: thau_bl1a_4p6keV ISa 34.3 -> 36.4, but kdp 23.9 -> 21.3, aspirin 35.5 -> 33.6, L-cystine R1 0.124 -> 0.137, 6o2h ISa 31.0 -> 25.7; no fewer stalled loops. Validation, 23 sets (small molecules, the four losers, stall sets, controls; run 20261020-0152 cvfold-final-sets): hepes, myob_x06da, 9rcs, 7bgt back to their tol-1e-3 values; the other 19 equal to the base within noise (aspirin 20 keV ISa 35.8 -> 35.5, L-cystine 25 keV ISa 8.5 -> 8.2 with SHELXL R1 0.1152 -> 0.1154). No SG, d_min or verdict change. ci tier (20261020-0152 cvfold-final-ci): 29/29 pass, every row within noise of both earlier full runs. [RotationScale] and AndersonMixing* pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4Foreign stills files state their saturation level wrongly or not at all: an MPCCD Cheetah file is dark-subtracted per pixel, so each pixel clips at the ADC limit less its own dark, and its CrystFEL geometry leaves max_adu commented out - the run reported "Overloads: 0" while the inner panels saturated in most frames. saturation_ceiling (image_preprocessing/SaturationCeiling) histograms pixel values on a log scale (64 bins per octave from 256) and returns the lower edge of the lowest bin holding over twice what any of the 8 bins below it held (and at least 50 values): a bright tail only falls, a clipped one piles up below where it stops, as a band where the level differs per pixel. DetectSaturation runs it on 40 frames spread over every stills run and logs it beside the file's limit; for any input but a Jungfraujoch-written file (ProcessConfig::detect_saturation) a level below the file's replaces it. The CrystFEL/CXI reader, where the geometry states no max_adu, now takes the stored integer type's maximum (an int16 Cheetah file clips at 32767). The geometry-file-free CXI path (previous commit) is limited to square frames: a tiled detector's Cheetah slab (CSPAD, AGIPD, MPCCD, ...) read as one flat panel would place every pixel wrongly without a word, and those slabs are not square. Measured: MPCCD stills, 1339 frames: pile-up from 1218 photons (by hand 12000 ADU ~ 1200 ph); 5.5% of reflections coarser than 4 A flagged; lowest-shell CC1/2 0.545 -> 0.674, overall 0.741 -> 0.793; same dimple refinement against the deposited model R-free 0.2586 -> 0.2556 single-panel Rayonix CXI, 1316 frames: no pile-up (20 pixels at the int16 limit in 40 frames); the int16 limit flags 123 reflections; CC1/2 0.737 -> 0.787, rugnux R-free 0.382 -> 0.358 JUNGFRAU stills (Jungfraujoch files: lysozyme and two private sets): no pile-up below the stated limit, results bit-identical Cost 0.5 s per run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>Three defects that stopped known-cell stills runs: - The consensus cell compared every frame's cell to -C axis by axis. A run whose frames index in another setting of the same lattice (axes in another order, another monoclinic choice) had no frame accepted, no consensus cell and no output ("No crystal lattice was determined"), although it merged fine. A frame whose lattice contains the given cell now counts on the given cell's axes (CellOnReference, the same FindSublatticeBases the merge-stage relabel uses). - The per-image ambiguity resolver relabelled a frame's reflections but left its lattice on the old axes. Everything after it predicts from index and lattice: the sublattice relabel put such a frame through a second transformation, and on a monoclinic cell with a near-symmetry (a+c nearly as long as a) the merge came out at indices finer than anything measured, shells over 100% complete. The lattice now follows the indices (ReindexLattice, B' = R^-1 B). - With -C/-S the stills path replaced the known-cell indexer's lattice, already on the given axes, by LatticeSearch's conventional choice, which on that near-symmetric cell is the other description on half the frames. The indexer's lattice is now kept when it is close to the given cell. Measured (cell + space group given, dimple against the deposited model and free set): - monoclinic MbCO-type still set: was over-complete/broken with --model; now 2.06 A, R/Rfree 0.233/0.314 at 2.11 A (P1 merge before: 0.275/0.306). - C-centred orthorhombic set: no MTZ before; now 3.27 A, R/Rfree 0.298/0.403 at 3.68 A (de novo 0.336/0.448). - monoclinic two-form set, one form: cell P 1 21 1 with beta (was alpha); Rfree 0.310 -> 0.260, 3.05 -> 2.72 A. - No change on seven other stills sets, nor on two rotation battery sets (5reo, lyso_x10sa_90deg_1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>