168 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	a29c36600f	Beam-stop shadow detection, and a low-resolution limit for scaling ... rugnux finds the beam stop and its holder in a projection of 60 images and marks them in the pixel mask as bit 9 (--detect-beam-stop[=N|off], on by default). Reflections behind the stop are attenuated but not flagged, so they integrate low with a plausible sigma and nothing downstream catches them: the signal-box gate requires 100% valid pixels and shadow pixels are valid, the background clip is high-side only, and the |zeta| cut applies only to the space-group search merge. The detection compares each pixel's background against the typical background at the same radius on two channels. An azimuthal one (the ring median) finds the holder arm, which is a minority of its ring; a radial one (the background just outside) finds the disk, which the ring median cannot see because inside a fully blocked ring the median is the shadow itself. Pixels are pooled over a 5x5 box and tested only where the background has actually been counted, so low-background data no longer masks the whole detector. Recorded reflections are carved back out - a beam stop cannot block a reflection that was measured. Bit 9 belongs to the run that found it, not to the dataset: it is cleared when a run starts, so a mask read back from a file that carries one starts clear. The user mask (bit 8) is left alone. Scaling and merging gain a low-resolution limit, default 50 A (--scaling-low-resolution <num>, 0 removes it), applied per observation before scaling so it also protects the per-frame scale fit and the space-group search. 50 A is the value XDS configurations use; rugnux_vs_xds.py now matches both of XDS's resolution limits instead of only the high one, so the lowest shell is the same shell in the two programs. The viewer draws the detected shadow in coral with a "Show beam stop" switch in the side panel, exposes the low-resolution limit in the settings dock, and offers detection in its processing jobs. Adding an image marker meant giving the reader a MIN_REAL_PXL_VALUE, because several places classify a pixel by range rather than by equality and would otherwise read the new marker as a very negative intensity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 01:05:31 +02:00
leonarski_fandClaude Opus 5	df9a9c2a2c	Fix the defects found reviewing the branch before merge ... Image buffer: the per-image CBOR metadata headroom had been re-derived from the online reflection cap alone, which cut it from 4 MiB to 2.55 MB while the measured worst case - reflections plus the capped spot list plus the three azimuthal arrays - is 2.9 MB, so the receiver dropped the frames with the most to say. Restore it and give it a name that both the code and its guard test read: written down twice, the two had drifted and the test kept passing against the value the code had left. Spot finding: an unset low_resolution_limit means no limit at that end, as an unset high_resolution_limit already did. An optional rather than a zero sentinel, because zero is not a natural "no limit" here - every pixel lies above it, so the plain comparison masked the whole image instead of none of it, and nothing validated the zero. The API field is no longer required; a zero is folded into the unset case at the boundary, where older clients still send it, so one spelling reaches the analysis code. The FPGA takes its fixed-point ceiling instead, since ap_ufixed<16,9> wraps above 512 A and would have masked everything. image_preprocessing: check the CUDA calls on the fused decode path - the one new GPU file with none, and the path fed by bytes we did not produce. An unchecked synchronise returned the host-written sentinel as if it were a measurement, so the decode looked successful and the fallback to the host decoder never fired. rugnux: --stride no longer writes one past the end of the per-image arrays, whose count floored where the worker loop ceils, and the written process file links the images actually processed rather than the first N - each frame's picture now sits next to its own analysis. Powder calibration: the face-centred calibrants no longer list their systematically absent rings, so the distance fit starts from a reflection that exists rather than an extinct one; the triclinic calibrant covers both signs of h and k instead of a single octant, which is only valid for a diagonal metric. The test asserted the old behaviour - one ring formula for every cubic standard - and is rewritten. CBOR: skip an unknown tagged value in the end block, as the other four blocks already do. One advance lands on the tagged item rather than past it, so an older reader fed a newer end message threw and never finalized its file. Viewer: a settings value the setter rejects no longer escapes as an uncaught throw from a worker slot, and the field offers only what the setter accepts. Space-group search: judge stage B on the same "present" cut stage A already computes. Merged sigma is floored so no reflection reads above ISa, so on a low-ISa merge the fixed cut left both stage B tests unsatisfiable - every screw axis passed unchallenged and the centering rescue switched itself off on exactly the weak data it exists for. Where the fixed cut is the smaller of the two they are equal and this is inert: over the 37-crystal rotation battery every crystal reports the identical space group and identical merge statistics, so it is a no-op there and the low-ISa case it targets remains unmeasured. rugnux: --polarization reaches --mode azint, which parsed the flag and then dropped it; that mode also applies the same polarization default as every other mode. Acknowledge the ACTS/traccc project, whose sparse connected-component labelling both spot extractors take their algorithm from, with its citation and its license. The rc.161 change list is brought back to one line per entry, and the user-visible changes that were missing from it added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 18:18:46 +02:00
leonarski_fandClaude Opus 5	5830f78d57	Revert the azimuthal-integration sigma clip ... Removes azim_int_settings.sigma_clip / rugnux --azim-sigma-clip and the clipping machinery in AzIntEngine. This is a partial revert of a6be35ccd - the ice-ring-mask removal that commit also carried stays. Sigma clipping remains where it started and where it is needed: inside the adaptive spot finder, at a fixed 3 sigma on raw counts, feeding the detection threshold and the ice score. The option made the workflow harder to reason about than the quantity was worth. It gave azimuthal integration two meanings behind one setting - the bin mean and the background under the peaks - which the azimuthal-integration workflows do not need. It also did not compose with the fused GPU engine, which supplies the profile from its PLAIN pass: on the default rugnux, viewer and receiver path the setting was silently doing nothing (measured, the profile came out identical to the unclipped run to 1e-6 with identical per-bin pixel counts). Making it correct is not a matter of gating that one shortcut - it means separating the workflows (azimuthal integration, MX rotation, MX stills, geometry calibration) and deciding per workflow what the profile is for, which is a larger change than the option earns. The default path is unaffected: over 20 images of a rotation dataset the radial profile, the per-bin pixel counts and the spot counts are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 15:31:19 +02:00
leonarski_fandClaude Opus 5	a8d289e7cf	Powder calibration: write Poni1/Poni2 in pyFAI's frame, not ours ... The same frame mismatch as the rot2/rot3 fix, in the other two fields. Our pixel coordinates are pixel-centred - 948.0 is the CENTRE of pixel 948 - while pyFAI measures from the edge of the sensor and puts the centre of pixel i at (i + 0.5) * pixel size. Poni1/Poni2 went out as beam * pixel size, so anything reading the file placed the pattern half a pixel (37.5 um at 75 um pixels) off ours. The previous commit's "Poni1/Poni2 need no such change" was right about the axis directions and wrong about the origin. The proof was already in the tree. The pyFAI reference values in DiffractionGeometryTest were computed for a .poni with Poni2: 0.150 and a 75 um pixel, which the tests translate to beam_x = 2000 - but pyFAI's numbers are reproduced only at 1999.5. At 2000 every one of them is out by 2.6e-3 nm^-1, which the 1e-2 tolerance hid. The tests now use the beam centre those headers actually mean, and agree with pyFAI to 1e-6 - float precision - across untilted q, azimuth, rot1, rot1+rot2, rot3, rot1+rot2+rot3 and the solid-angle correction. Tolerances drop to 1e-4 (1e-5 for solid angle): ~100x the observed float noise, and 26x tighter than the half pixel they were blind to. The viewer's calibration window printed "PONI x = ... mm" from the un-offset value beside the path of the file it disagreed with; it now matches the file. Also moves the viewer's beam-centre cross half a pixel down and right, where the spot, prediction, top-pixel and saturation markers already are. Our coordinates are pixel-centred and the Qt scene's are pixel-cornered, so the map between them is +0.5, and DrawBeamCenter was the one overlay missing it. The convention itself is now written down in docs/DETECTOR_GEOMETRY.md, with the conversions to XDS ORGX/ORGY and to the edge-of-sensor programs, this being the second bug to come out of it. Only exported and displayed values change; the fitted geometry, spot positions and integration were always self-consistent. A .poni written by an earlier build is half a pixel off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 03:55:42 +02:00
leonarski_fandClaude Opus 5	6194fe6fbf	viewer: calibrate the whole dataset from "Analyze dataset" ... The powder panel could only calibrate the image on screen. Calibration is now a third page beside MX and AzInt, so the dataset button runs it over every image the same way it runs the other two - which is the point, since a powder ring is measured far better by summing a run than by one frame. The page carries the calibrant and the method (rings or spots); the interactive Guess/Refine buttons stay where they were and now share the one calibrant selection, so there is no second combo to drift. analyzeDataset() carries the ProcessMode rather than a bool: a third state was coming, and two bools would have had one combination that cannot be valid. The calibrant list gains ICE, which it could not offer before: the widget worked in unit cells, and hexagonal ice has none that generates its rings correctly (P6_3/mmc would include systematically absent ones). FindCenter now takes the ring list its first line used to derive, so the interactive path gets ice as well. The result window leads with the residual rms rather than the fitted sigma. The sigma is a formal scatter estimate and understates a bad fit badly - measured on ice, 0.215 px reported against a 1.70 px residual - while the rms separates a usable fit from one that has locked onto the wrong thing. A rings run needs the profile binned in azimuth; below four sectors it returns nothing at all. The viewer raises the count to 32 exactly as the CLI does, and says so in the panel and in the job dialog rather than doing it silently. Also fixes a CLI inconsistency this comparison exposed: rugnux's calibration branch never applied the standard offline analysis defaults, so it measured the rings in a profile built with the file's polarization factor while every other mode - and the viewer - uses 0.99. Found because the two disagreed by 0.005 px in PONI x, and confirmed by reproducing the viewer exactly with --polarization 0.99. With it applied the CLI and the viewer write byte-identical .poni files on LaB6 by rings, LaB6 by spots, and an iced dataset over 1800 images. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 10:45:39 +02:00
leonarski_fandClaude Opus 5	6468dd13be	rugnux: --mode, and detector calibration from powder rings ... --azint-only and --scale are replaced by --mode mx|azint|scale|calibration, with mx the default. The old flags are removed rather than aliased. Calibration mode fits the detector geometry - PONI x/y, the two tilts and the distance - to a calibrant's powder rings and writes a pyFAI .poni alongside a report of how far each parameter moved from the header. Bragg data constrain the beam centre worst, because it is gauge-coupled to the crystal orientation; a powder ring has no orientation to couple to. --calibrant takes lab6, agbh, ceo2, si or ice. A calibrant is a list of ring positions rather than a unit cell, because hexagonal ice is P6_3/mmc: rings enumerated from its cell would include systematically absent ones. So the crystalline standards generate their rings from a cell and ice carries the measured list, and RingsFromAzimuthalProfile, GuessGeometry and OptimizeGeometry all take ring q. The calibrant table is shared with the viewer's powder panel, which previously carried its own copy. --calibration picks how the rings are measured: rings (default) sums the (q x azimuth) profile over every processed image and fits the arcs in it; spots pools the found spots and fits those. Both use the whole run, with -s/-e/-t selecting images. rings defaults --azim-phi-bins to 32, since a profile with one azimuthal bin has averaged the ring over every direction and cannot locate it. Two fixes this exposed: The extraction window is capped at half the gap to the neighbouring ring. The background under a peak is taken from the ends of its window, so a window wider than half that gap measures the next ring's flank as this ring's background - and hexagonal ice has three rings within 0.06 1/A. Ice calibration was 3.5 px out before this and 0.29 px after; LaB6 is unaffected. RingOptimizer holds rot1/rot2 fixed when only one ring is present. A tilt and a centre offset both move a ring as cos(phi) and are separated only by the tilt's amplitude growing as the ring radius squared, so on a single ring they are exactly degenerate. Measured. LaB6 at five distances: the fitted direct beam is within 0.36 px of an independent implementation out to 300 mm, and D = -0.046 + 1.000788 dtz with an rms of 0.011 mm. At 500 mm one ring is fully on the detector and a second only clips the corners, which is not enough to constrain a tilt - restricting the q range to the resolved ring recovers 0.06 px. Ice: 5.53 -> 0.29 px on one crystal and 4.71 -> 0.80 px on another, against XDS's refined direct beam. On an ice-free crystal the fit is worse than the header, which is the correct outcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 10:00:03 +02:00
leonarski_fandClaude Opus 5	cfd3697ddb	docs: cover the azimuthal sigma clip, the indexing flag, and the metric-symmetry check ... Three things had reached the code without reaching the documentation. The azimuthal sigma clip had a RUGNUX.md row and a CPU_DATA_ANALYSIS section but no changelog entry - and the only "sigma clip" the changelog mentioned was the background ring's, which is a different thing at a different stage. --index-ice-rings was in the options table but nowhere in the changelog, so the entry describing the ice gate still implied that whether indexing uses the ice-band spots is decided per run, which it no longer is. CPU_DATA_ANALYSIS section 6 still described the Bravais class as simply "the highest-symmetry class that matches within tolerances", which is the behaviour that lost a crystal outright. It now records that the class is chosen from the UNREFINED candidate against a fixed angular tolerance, that a pseudo-symmetric lattice therefore gets promoted a class too far, and that the first pass settles it on validation-frame counts with a clear-majority bar - including why the bar is a majority rather than a margin, since that distinction is the whole reason the check is safe. It also records that the first pass finds its own spots rather than reading the acquisition's, which was not written down anywhere. Also a build note: M_PI is not standard C++ and MSVC does not define it, so the Bragg integrator's use of it broke the Windows viewer build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:56:08 +02:00
leonarski_fandClaude Opus 5	1df9556ec1	Bragg integration: default the radial background correction off again ... Auto rode in with the ice work rather than on its own evidence, and measured over the 37-crystal rotation battery it does not carry itself yet. It TARGETS correctly - it fires on ten crystals and every one is ice-positive, no failures, no space-group changes - but it costs 1.35x the wall clock (median +3 s per crystal, worst +29 s) and on the merge statistics it is the familiar sign-mixed trade: high-shell CC1/2 worse on three of the four crystals that move materially, mean -0.76. The case for it is real but rests on agreement with a fixed external model - 43 % of the ice bands' excess amplitude removed on smooth ice, the effect 7x stronger inside the bands than outside - which is the better arbiter and also the narrower one. That deserves settling on its own, not riding along with a set of ice defaults. `--background-radial=auto` keeps it a flag away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:29:42 +02:00
leonarski_fandClaude Opus 5	a6be35ccdb	Azimuthal integration: optional sigma clipping of the reported profile ... The profile is the MEAN of each bin, so a few strong reflections landing in a bin lift it exactly as a smooth powder ring does. That is the wrong quantity whenever the profile is wanted as a background rather than as a measurement of what is in the bin - the ice score being the case in point, where reading a plain profile INVERTED the metric: over 37 rotation crystals the two highest-scoring crystals had no ice at all. The adaptive spot finder already computes the right thing, a sigma-clipped per-resolution-ring background, as a byproduct of its own threshold. Where it runs, the ice score uses that. Where it does not - --no-adaptive-spots, --azint-only, and anything reading the profile the broker wrote - there was no way to get it. This adds one: azim_int_settings.sigma_clip (rugnux --azim-sigma-clip), 0 = off, minimum 2 because a tighter clip rejects a large part of a clean Gaussian bin and biases the estimate low rather than removing outliers. Two clip passes follow the plain one, matching the finder's recipe - the first pass's standard deviation is itself inflated by the peaks being removed, so one pass leaves a threshold that is still too generous. A bin with fewer than eight pixels is left alone: at the detector edge and behind the beam stop there is no spread to clip on. Both engines do it. On the GPU the accept range is computed by a small kernel and stays resident, so a clip pass is one more read of the same pixels and no round trip; the two accumulation kernels take the range as a pointer that is null on the plain pass. Measured on a JUNGFRAU rotation dataset, non-adaptive path: azimuthal integration 0.02 -> 0.06 ms per image, exactly the 3x the extra passes predict, against a 0.34 ms per-image total. Note what the result IS: the smooth background under the peaks, not the bin mean. It should not be switched on where a ring's integrated intensity is wanted - the powder-ring geometry fit reads ring peaks, and those are what a clip is designed to remove. Off by default, so nothing changes unless it is asked for. Not exposed over the REST API - that needs the generated model regenerated, which is a separate step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:18:52 +02:00
leonarski_fandClaude Opus 5	3d4209d803	docs: the first-pass ice measurement, the promotion fix, and powder-ring geometry ... Changelog entries for the three changes above, and a new CPU_DATA_ANALYSIS section on determining detector geometry from powder rings: why a ring is an independent constraint on the beam centre (it has no crystal orientation to be gauge-coupled to, unlike everything else that fits geometry here), what a ring can and cannot determine, and how the ring points are obtained. The section states the harmonics correctly, which is worth writing down because the intuitive version is wrong: a detector tilt shows up as cos(phi), the same harmonic as a beam-centre error, and the two are separated by the amplitude growing as the ring radius SQUARED - so it takes at least two rings, and on one ring they are exactly degenerate. The genuine cos(2 phi) term is hundredths of a pixel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 22:12:21 +02:00
leonarski_fandClaude Opus 5	f0cdb027e1	Ice: default the merge mask off, gate the radial background on smooth ice, and pick detection by geometry ... Three defaults, each settled by measurement rather than by argument. The arbiter throughout is structure-referenced - anomalous peak height where a crystal can carry it, and otherwise the agreement of the ice bands with a fixed external model against resolution-matched DECOY bands carrying no ice. The band-versus-decoy contrast is used because R-free here tracks completeness, and every one of these switches moves completeness. The damage is real and it localizes: over the rotation battery the ice bands' excess amplitude reaches +9.6% on a smooth-ice crystal and +35% on the worst, while a clean control sits at +0.6% (z +0.45). On the worst crystal, nine of the ten largest excess peaks in a q scan land on hexagonal ring positions. Turning ice handling off leaves the contrast unchanged and forcing it on a clean crystal does not create one, so it is the ice and not the machinery. MERGE-TIME RING MASK -> OFF. It deletes reflections, which no other program does by default - AIMLESS, DIALS, xia2, XDS and CrystFEL all keep ice-band reflections in the merge and exclude them only from the model fit; autoPROC is the sole exception. On the one battery crystal where the mask fires and an anomalous arbiter can score it, dropping the band moved the mean peak height at the known sites by -0.001 +- 0.018 sigma, 2% of the site height, while removing 1149 unique reflections whose mean I/sigma was 3.62 against the dataset's own 3.05 - better than average data - and costing 17 completeness points in that shell. It fires on 5 of 37 crystals, changes no space group, and those 5 disagree in sign: it clearly helps the two most heavily iced, is a wash on two and costs a third. So it stays as a switch, worth setting by hand on a badly iced crystal where it shows in the high shell, but it is not a default. RADIAL BACKGROUND -> AUTO, gated per image. The correction models the background as a function of radius alone, and that is exactly when it works. On a crystal with pure smooth powder ice it removes 43% of the bands' excess amplitude, with the improvement 7x larger inside the bands than outside; on a crystal whose ice is discrete crystallite spots - no smooth ring to model - the excess amplitude GREW by half; on clean data it is inert to four decimals. The two ice channels already separate those morphologies, so --background-radial takes on|off|auto and auto applies it to an image when that image's peak-excluded score reaches --ice-min-score. Auto never engages without such a score, because the plain profile carries the Bragg peaks and cannot support an absolute threshold. Per image rather than per run, and that was tested rather than assumed: the gate fires on 100% and 94% of frames on the two crystals that want it, and on 1.5% of frames - 32 blocks, 23 of them single frames - on the textured-ice crystal. A seam statistic against off + f*(on - off) is null on both mixed runs, every merge statistic is bracketed by the pure arms, and the textured crystal's auto arm lands on `off` rather than on `on`'s harm. A run-level gate would need the score before the pass that integrates, i.e. rotation-only plumbing, and buys nothing measurable. The kernel was already built unconditionally, so flipping the flag per image is free - except on the GPU, where the launches were gated on a construction-time n_rad. That is why the buffers are now allocated whenever the correction could run, and Run() decides per image. DETECTION -> the geometry's default when the file is silent: on for rotation, off for stills, with the command line and then the file taking precedence. A rotation sweep sits on the same rings for the whole run, so ice there is a coherent systematic and the presence gate keeps it inert on a clean crystal; a serial stills run has too few spots per image to spend any on flagging. The master file's key is kept as written rather than collapsed to a bool, so "the file said nothing" is distinguishable from "the file said no" - it used to fall silently to off, taking the exclusion from the scale fit with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 19:06:30 +02:00
leonarski_fandClaude Opus 5	09fb8e0306	Bragg integration: clip the background ring high side instead of trimming it ... The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A symmetric trim is not a consistent estimator of the mean of a right-skewed (Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true mean at every level, and with ~50 signal pixels in the r1 disk that under-subtraction adds ~5 counts to every partial on every frame. Measured two independent ways on four rotation datasets - stored background_mean against a plain ring mean over the same pixels on reflection-free frames, and directly on apertures that provably hold no reflection. Empty-aperture pedestal, counts: plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip +0.02..+0.54. Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for monochromatic data. It rejects the same one-sided contamination the trim was there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds ~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The trim stays reachable with --background-trim for back compatibility; setting either estimator clears the other, so they can never stack. --integrator boxsum does not take the clip (matching what the shipped clip already did), so it now uses the plain ring mean unless --background-trim is given. The intensities get measurably more accurate: per-shell agreement with an independent processing of the same images improves on 14 of 16 crystals (weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR - absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and CC1/2 in the outer shell improves on 14/7. EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by information-free counts, so both get worse when the bias is removed; neither is evidence against this change. That fingerprint is exactly how the trimmed mean was accepted in the first place. Known cost: over the 37-crystal rotation battery the de-novo space-group count goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal whose merge fails the absolute-sanity gate under either background (R_meas 63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the very operator being scored, so its operator CC is diluted by construction. No other crystal changes space group, and twin protection is not weakened - the H-ratio veto that refuses genuinely twinned crystals gets MORE decisive (1.63 -> 1.84, 2.83 -> 3.99). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 18:03:11 +02:00
leonarski_fandClaude Opus 5	fb55645b81	Revert "rugnux: fit the profile radius from the strongest spots too" ... Reverts the profile-radius part of 457b1bfd1; the comparison-script and mosaicity-column changes from that commit are kept. The cap was validated on the rotation battery, which cannot test it: the profile radius feeds `ewald_dist_cutoff` in IndexAndRefine, and that is read only by the STILLS predictors (BraggPrediction/BraggPredictionGPU). The rotation predictors gate on the mosaicity window instead and never look at it. So "no space-group changes, 36 of 37 crystals bit-identical" showed the quantity is inert for rotation, not that capping it is safe - and the one regime where it does act was never exercised. Validating it needs the serial-stills battery, which is a much larger exercise. Until then the arbitrary constant is not worth carrying in a code path nobody measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 21:35:17 +02:00
leonarski_fandClaude Opus 5	5eb386e333	docs: describe the per-frame geometry smoothing ... Goes in §10.3 next to the per-frame scale and mosaicity smoothing, since it is the same mechanism applied for the same reason, and trims the changelog line to one sentence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 20:23:19 +02:00
leonarski_fandClaude Opus 5	017f64690c	rugnux: smooth the per-frame geometry before scaling ... Geometry is re-refined independently on every frame, against that frame's spots alone - as few as a dozen on a sparse crystal, where XDS fits its equivalent to about sixty times more data. Measured over ten datasets the per-frame orientation carries two components: a slow drift that is real, with rugnux and XDS agreeing to R^2 0.83-0.88 on the two crystals that genuinely slip by 1.5 and 0.54 degrees, and a fast jitter that is fit noise, scaling with spots-per-frame at exponent -0.79 where counting noise alone would give -0.5. The jitter is worth 1-8% on merged intensities, 24% on the sparsest crystal. It cannot be fixed by refining less. Turning per-image refinement off entirely loses six space groups and a whole crystal, and even a 624-spot-per-frame crystal collapses; dropping the beam-centre terms holds the space groups but is worse on 31 of 37 crystals. The freedom is earning its keep, so keep it and suppress only the band that cannot be physical - a crystal does not re-orient and snap back from one frame to the next. So smooth the orientation in frame order after integration and recompute each partial's delta_phi, and hence its partiality, from the smoothed lattice. Batching at integration time was not an option: frames are processed independently and the online path depends on that. This runs before the GPU upload, so the device path picks it up with no separate kernel. The window is chosen per dataset by leave-one-out cross-validation, because the two components vary far too much for one number - drift spans 0.018 to 1.288 degrees and jitter 0.005 to 0.221, so any fixed window over-smooths one crystal while under-smoothing another. Chosen windows range from +-1 to +-20 frames. It is capped: cross-validation scores how well neighbours predict a frame's orientation, which on a barely-drifting crystal keeps improving with width, but the per-frame fit is also absorbing a real per-frame systematic and smoothing too wide destroys it - uncapped, one crystal chose +-60 and lost 16% of its ISa. Battery over 37 crystals: space groups unchanged at 34 matching XDS, R_meas better on 31 and worse on 6, low-resolution R_meas 30/7, ISa 26/10, high-resolution CC1/2 23/12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 20:06:41 +02:00
leonarski_fandClaude Opus 5	457b1bfd1d	rugnux: fit the profile radius from the strongest spots too ... Same defect as the mosaicity in 2c94f3013, in the same file's sibling fit. The profile radius is an RMS of the excitation error over whatever spots were kept, and weaker spots sit further off the Ewald sphere, so it grows with the depth of the list: measured over a 150 -> unlimited spot budget it rises ~60%, and on a clean dataset as much as on a hard one, so this is general rather than something one awkward crystal provoked. That made it a function of --max-spots, which is an indexing budget, rather than of the crystal. Its one consumer treats it as a membership gate (ewald_dist_cutoff is twice it) where reflections at the cutoff carry near-zero partiality and are removed downstream anyway, so the integrated data barely notices: with the mosaicity already pinned, the partial count moves 0.2% across a 34% change in the radius. It is also reported per image as a diagnostic, though, and a number that slides with an unrelated setting is misleading to anyone comparing two runs - and it would stop being benign the moment anything used it as a width rather than a gate. Battery over 37 crystals: no space group changes, 36 of 37 bit-identical, no failures, one crystal marginally better. Also in the comparison script: report XDS's mosaicity next to rugnux's. XDS has two and they are not interchangeable - CORRECT.LP's REFLECTING_RANGE_E.S.D. is post-refined, while INTEGRATE.LP's per-batch SIGMAR is its integration-stage estimate, and the two differ by up to 2.3x. The XDS cell now prints both as postrefined|MLE so a per-image estimate is compared against the one measured the same way. Fixes a scoping bug in the same addition where every crystal read the last directory's INTEGRATE.LP. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 10:52:30 +02:00
leonarski_fandClaude Opus 5	2c94f3013e	rugnux: fit the mosaicity from the strongest spots only ... The per-image mosaicity MLE ran over the whole indexed spot list, so it rode on --max-spots, which is an indexing budget. A spot is detected when I_full * R(tau) clears the finder threshold, so selecting by intensity censors on R(tau): a deeper list holds proportionally more large-|tau| partially recorded spots and the fit widens with it. Raising the budget 250 -> 1000 widened sigma_M 0.059 -> 0.075 deg on a rotation dataset whose measured rocking width says 0.054. That is not cosmetic. An over-wide mosaicity mis-states every partiality in scaling: forcing the mosaicity across that range moved the merge error model from b 0.039 / ISa 26 to b 0.167 / ISa 6, and the space-group search lost a genuine 422 with it, merging the crystal in 222 instead. Cap the fit at the strongest 250 spots. FilterSpotsByCount leaves the list strongest-first, so this selects exactly the spots a smaller --max-spots would, and the mosaicity becomes invariant: 0.0538 deg at 250, 500, 1000 and 2000 spots, with the correct space group at each. Trimming or down-weighting the tau tail does not work - the censoring is multiplicative in R(tau), so it widens the whole distribution rather than adding a tail. Battery over 37 crystals: exactly one change, the demoted crystal repaired (33 space groups matching XDS -> 34). 23 of 37 are bit-identical, never reaching 250 spots. Unaffected elsewhere: the default spot count is 250, and stills have no goniometer so they return before the fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 07:34:40 +02:00
leonarski_fandClaude Opus 5	cd0c43c65c	docs: add the packaging and documentation work to the rc.161 change list ... Covers the Debian package rename and the DKMS scripts, the writer's --verbose, the version plumbing, and the documentation pass - the release-contents page, the corrected repository URLs and package names, and the rugnux, viewer and writer pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 20:53:48 +02:00
leonarski_fandClaude Opus 5	ceb92fc4cc	docs: bring the rc.161 change list up to date ... Covers the GPU decode work (fused un-transpose, the memory it frees, corrupt-chunk detection, large bitshuffle blocks, host fallback), the two reproducibility fixes (integer ring statistics, striped geometry-refinement sampling), the connection-teardown and written-extent fixes, and the thread-count, max_hkl and compression-time repairs. Folded into the existing entries where they belong rather than added as new ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 16:58:03 +02:00
leonarski_fandClaude Opus 5	b6c4a59d69	docs: record the stored-format break, and say how the adaptive finders actually accumulate ... The per-image image-scale B factor was dropped from the CBOR stream and from the written HDF5, which is a change for anything reading those files, but the changelog listed it only under the OpenAPI breaking changes. The GPU adaptive finder test claimed both finders sum the rings in double. The CPU one does; the GPU one stages a block's contribution in float before reducing across blocks in double, deliberately, to keep the hot loop's shared footprint down. Say so, and say what follows from it - detection compares integer pixel values, so a threshold that crosses an integer flips every pixel of that value in the ring at once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 14:14:24 +02:00
leonarski_fandClaude Opus 5	47277674fa	docs: split the breaking API changes out of the rc.161 change list ... Earlier releases put breaking changes in their own paragraph after the bullets (rc.139, rc.29) rather than as one item among them. Follow that: the OpenAPI changes now sit under their own heading below the list, with the client-side action in the lead line, and are listed one per change instead of run together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 07:46:57 +02:00
leonarski_fandClaude Opus 5	737cbde3ff	docs: tighten the changelog and the decoding section for release ... The rc.161 changelog had grown entries several hundred words long and listed the same area three or four times over. Collapse them by subject - spot finding, resolution limits, space-group search, scaling, performance, correctness - and hold each to one line, keeping the actionable detail in the breaking API entry. Add the performance work that had not been written up: device-side image decoding and the parallel first-pass candidate-cell refinement. Section 0 of the CPU analysis document was the longest thing in it after two core algorithm sections, and most of that was a profiling narrative and the measurements that motivated the change rather than a description of what runs. Cut it to the two kernels, the host-side block scan and the fallback rule. The attribution stays; it is also in the reference list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 07:44:55 +02:00
leonarski_fandClaude Opus 5	0ca159449f	Bragg integration: integrate as far as the detector reaches, not to a fixed 1.0 A ... BraggIntegrationSettings::DMinLimit_A had a setter that nothing anywhere called, so it was always its 1.0 A default - in rugnux, the viewer and the broker alike, with no option or API field to change it. It feeds the predictor as high_res_A, which discards any reflection with |q| > 1/d_min, so integration simply stopped at 1.0 A however far the detector reached. Five of the 33 rotation test datasets have detectors reaching past it, down to 0.981 A. On one of them, run with no resolution limit, the shell table ended dead at 1.00 A with that shell still at CC1/2 55.6% and <I/sig> 3.4 - cut mid-shell rather than fading out. This branch had already made the sibling limits detector-driven (spot finding, scaling), so the pipeline was finding spots the detector could see and then refusing to integrate them. Make it a std::optional: unset means as far as the detector reaches, a value limits. The limit is only a bound on how far the lattice walk goes, never a second opinion on what is measurable - both predictors independently drop reflections that miss the detector (BraggPrediction.cpp, BraggPredictionRot.cpp) - which is what makes the detector's own reach the right default. rugnux gains --integration-high-resolution (0 = no limit, as for --spot-high-resolution); the derived per-axis prediction range resolves against the same number, so the two cannot drift. Full battery: 30/33 space groups, unchanged from before, 0 failures and the same three known mismatches; 22 of 32 crystals bit-identical and nothing worse than 5 observations in ~500k. The datasets that gain do so because their detector reached past 1.0 A - the effect is understated here because the harness caps each merge at the XDS resolution anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 14:03:34 +02:00
leonarski_fandClaude Opus 5	b0e315e73c	Bragg prediction: derive the lattice walk from the cell, and expose it in the API ... Follow-up to making max_hkl a setting: it is now an optional, and unset means "take it from this crystal". The predictor keeps only |q| <= 1/d_min and h = a.q for the real-space axis a, so |h| <= a/d_min exactly - and likewise |k| <= b/d_min and |l| <= c/d_min. max(a,b,c)/d_min therefore bounds all three at once: nothing that could be predicted lies outside it, and nothing inside it is reached by a shorter axis. It applies to rotation and stills alike, both going through the one place the prediction settings are built. Offline (rugnux, viewer) the default is unset, so every crystal gets its own range; --max-hkl overrides it. Online the broker holds a concrete number, because the cost is the cube of it per image and a live acquisition should not have its frame rate decided by whichever sample is mounted: max_hkl joins bragg_integration_settings in the OpenAPI with a default of 100, so an omitted field arrives as that default (the generated model carries it) rather than as "derive it", and the frontend exposes it next to the integration model. Measured against a fixed 100 on six rotation crystals: three are bit-identical, two were being truncated and recover 419k and 5.8k observations with the high-shell CC1/2 going 15.1 -> 25.8% and 52.1 -> 55.3%, and the space group is unchanged 6/6. It reproduces a fixed 200 exactly, which is the bound being tight rather than merely safe. The sixth is worth recording: a 149/83/226 A cell derives 227, and because a single scalar has to cover the longest axis the cube is ~16x what a per-axis box would be - 22% wall clock, for a net 22 observations out of 364k (the per-frame 65536-reflection cap re-selects at the margin when more candidates are offered) and identical CC1/2, ISa and space group. Per-axis limits would remove that; the predictors already map a thread index to h, k and l separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 13:10:58 +02:00
leonarski_fandClaude Opus 5	a06c06931f	Bragg prediction: how far to walk the lattice is a setting, not a literal ... max_hkl was hardcoded to 100 at the one place production builds the prediction settings, so the only way to change it was to edit and rebuild - and it is not a constant of the method, it is a property of the cell. An axis is truncated once a/d_min exceeds it: 100 covers a 150 A axis at 1.5 A, but the same axis at 1.0 A, or a 250 A axis anywhere, loses its outermost reflections with nothing said. Move it into BraggIntegrationSettings next to the other prediction/integration parameters and add rugnux --max-hkl (1..511, default 100 - no behaviour change). Like the integration radii and the background trim it stays out of the OpenAPI, so the broker keeps the default it has today and live analysis cannot be handed a range that would not finish; the offline front end, which knows its cell, can ask for more. RugnuxCommandLine emits it when it is not the default. Measured on five rotation crystals at --max-hkl 200: two are bit-identical at no cost, and three were being truncated - one gains 419k observations (+17%) and takes its high-shell CC1/2 from 15.1% to 25.8% for +14% wall clock, the other two gain 12k and 5.8k observations with CC1/2 76.6->82.4% and 52.1->55.3% for +9% and +1%. ISa is unchanged throughout, and no frame overflowed the prediction buffer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 12:00:45 +02:00
leonarski_fandClaude Opus 5	3e56d96921	Stills partiality: adopt the refined tilt only when it fits better ... RefineOne re-measured the image's correlation to the reference after writing the refined partialities - because --min-image-cc drops images by it - and then ignored what it measured. A crystal the tilt model suits worse than the fixed partiality it replaces kept the refined model anyway, and the refinement is on by default. Compare against the CC the crystal arrived with and put it back untouched when the refinement does not improve it, which is the same state a crystal with too few reflections to fit ends in. Also four things noted in review and left until now: AdaptiveThresholdTest.cpp was listed twice in the test target, AdaptiveThreshold.h was the one header in image_analysis/spot_finding not in its library's source list, CLAUDE.md said update_version.sh rewrites VERSION when it only reads it, and the CHANGELOG did not mention that image_scale_b is gone from the plot_type enum - which breaks a client that asks for that plot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 22:03:18 +02:00
leonarski_fandClaude Opus 5	a770f0857b	CHANGELOG: record the rc.161 fixes ... Also corrects the reader bit-depth entry: taking the depth from the file was the wrong fix for the 32-bit EIGER2 case and broke 8-bit files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:42:04 +02:00
leonarski_fandClaude Opus 5	0f1851cf82	rugnux: drop the 1.5 A spot-finding limit on rotation data ... Rotation kept a 1.5 A high-resolution limit for spot finding on the strength of one indexing-rate measurement (100.0 -> 96.8% on a strong set). Measured properly, over the whole 33-crystal rotation battery, it does not earn its place: * no space-group decision changes - the same 30/33 agree with XDS, and the three that differ are the known pre-existing cases, unchanged; * 29 of 33 crystals are identical to the digit - same indexing rate, R_meas, CC1/2, ISa. The limit was doing nothing on the large majority; * where it does bite, the limit is the worse setting. The one crystal that loses appreciable indexing rate without it (99.50 -> 94.22%) comes back with lower R_meas (29.4 -> 28.1), higher high-resolution CC1/2 (27.9 -> 29.1) and higher ISa (5.77 -> 6.17). Another loses 0.4% of frames and gains 2.8 points of CC1/2_hi. Fewer frames indexed, better data from them; * runtime is unchanged (16m46s vs 17m32s over the battery). So the indexing-rate cost is real but does not carry through to the merged data, which is what the limit was protecting. Unset now means "as far as the detector reaches" for rotation as well as stills; --spot-high-resolution still sets a limit for weak, high-background data where the extra high-resolution spots are genuinely noise. This also removes the flag that distinguished "the user asked for no limit" from "the user said nothing" - with no rotation default left, both mean the same thing. While rewriting the comment block, corrects its neighbouring claim that rotation keeps the fixed-threshold finder; adaptive detection has been the default for both workflows since 6f4917dce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 10:39:23 +02:00
leonarski_fandClaude Opus 5	648e8c62c8	CHANGELOG: record the two settings fields that are no longer required ... Making azim_int_settings.high_q_recipA and spot_finding_settings.high_resolution_limit optional changes the wire format: when unset they are omitted rather than sent with a placeholder, and a client generated from an older spec does j.at() on them. The azimuthal-integration limit now defaults to unset, so a stock broker omits it out of the box - the break needs no operator action to hit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:38:38 +02:00
leonarski_fandClaude Opus 5	40acf1ffc4	rugnux: search the space group twice by default on rotation data ... --search-min-zeta now defaults to 0.85 for rotation, so the de-novo search runs on a merge of all the observations AND on a merge of only the well-measured ones, and keeps whichever found more symmetry. Previously it shipped off and the second opinion had to be asked for. Rotation battery, 33 crystals, NO flags beyond the resolution limit: fixed-threshold finder 30/33 - zero crystals differ from the single search adaptive finder 30/33 - the same three mismatches Both arms now agree crystal for crystal, which they have not done before. The last disagreement was a thaumatin set whose 4-fold sits 88.9 deg from the spindle: at defaults it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken) where it read P222. The classic arm is a strict no-op - zero differences against both the explicitly-flagged run and the run predating the dual search - so the default costs nothing where the geometry is not the problem, and 47.2 s against 47.8 s on the same crystal back to back. The default is safe to set because the two searches can only disagree by a LOST operator: discarding observations starves an operator correlation, it cannot invent one. That also makes the 0.85 itself uncritical - too aggressive a cut only means the second opinion contributes nothing and the full merge wins. --search-min-zeta 0 restores the single search. Docs: CHANGELOG gains a 1.0.0-rc.161 section covering the branch, and CPU_DATA_ANALYSIS records the four analysis changes of this work - the confidence-weighted per-image refinement, the collapsed per-frame scale guard, the opt-in per-image rejection, and the operator-disagreement criterion with the two-search rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 21:56:36 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	6136f858af	v1.0.0-rc.154 (#64) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-25 18:12:00 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62	2026-06-17 20:36:24 +02:00
leonarski_f	ef52dac2ee	v1.0.0-rc.151 (#61) ... * jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname Reviewed-on: #61 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-16 14:13:29 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60	2026-06-15 20:24:15 +02:00
leonarski_f	ea575f790a	v1.0.0-rc.149 (#59) ... * XDS plugin: Fix HDF5 mutex to run on multiple processors Reviewed-on: #59	2026-06-13 21:27:41 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	caef26873e	v1.0.0-rc.145 (#55) ... This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144. * **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future. * XDS plugin: Improve performance of VDS reading. * jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD. * jfjoch_writer: Clean-up of HDF5 routines to better handle issues. Reviewed-on: #55	2026-05-06 21:50:02 +02:00
leonarski_f	7d34e8a049	v1.0.0-rc.144 (#54) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Improve performance of preview JPEG image generator at receiver startup (saving about 150 ms on measurement start for 16M) Reviewed-on: #54 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-05-01 17:06:36 +02:00
leonarski_f	6e0bb971ac	v1.0.0-rc.143 (#53) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Avoid copying gain calibration together with DiffractionExperiment Reviewed-on: #53	2026-04-30 22:16:50 +02:00
leonarski_f	87fde1b32e	v1.0.0-rc.142 (#52) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8 * Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent Reviewed-on: #52 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 16:47:53 +02:00
leonarski_f	d760b12a18	v1.0.0-rc.141 (#51) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times * frontend: Fix selection of FFTW in indexing settings Reviewed-on: #51 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 13:04:54 +02:00