10 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	7bb74a9dc8	review fixes: two silent wrong outputs, and one rule for a quantity nobody measured ... From four whole-branch code reviews of rc166. Reviewing the net diff rather than the commits found what per-commit review structurally cannot: a later commit leaving an earlier one's claim standing, and two cases of a later commit quietly undoing an earlier one. THE FITTED RESOLUTION WAS THE P1 CROSS-CHECK'S. The block that merges in P1 to write <prefix>_P1.mtz saves and restores the error model around itself, because that merge is not the run's answer. A later commit taught the same function to report the CC1/2 resolution fit and did not extend the list, so every de-novo rotation run in a non-P1 group has been quoting FITTED_RESOLUTION - the number the report itself calls the one to quote - from a merge with n_ops times the unique reflections at a fraction of the multiplicity. AND IT ROUND-TRIPPED THE SPACE GROUP THROUGH ITS NUMBER. A number names only the reference setting, which stopped being enough when the search learned to adopt P 1 1 2(1) or I 1 2 1. Everything written after that block - the unmerged MTZ included - therefore carried the reference setting while the merged file carried the adopted one: two files describing one dataset in two different settings. It now carries the group. A PONI CANNOT STATE A MIRRORED OR QUARTER-TURNED DETECTOR, and the fits became orientation-aware on this branch while the writer did not. It wrote five numbers that silently described a different geometry from the one measured; it now refuses, and says the JSON beside it has the full one. A REFUSED FIT'S ERROR BARS COULD BE HANDED BACK AS AN ACCEPTED GEOMETRY'S. RingOptimizer::Run writes its uncertainty only where the solve is usable, and CalibrateFromSpots runs it twice - tilt free, then tilt pinned. A failed second fit kept the first's sigmas, valid flag and all. It is cleared on the way in. ONE RULE FOR NOT MEASURED. The report had four conventions for it and printed the same missing quantity two ways on adjacent lines: SIGANO as the literal "nan" and CC_ANOM by absence, for a Friedel-merged run that split no Bijvoet pair - which is the default. A quantity a run did not measure now writes no key, and the shell table's dash follows the same rule rather than a 0.0% that reads as a measured total failure. ANISOTROPY_D_MIN_BEST also stops printing nan when only its first principal direction is unmeasured. Four claims that a later commit made false are corrected where they stand: the merge header promising an order-independence the balancing rule gave up, the reference page arguing against CCanom 28 minutes before it shipped, the screw threshold whose "three dead reflections clear it" the evidence floor caps at 19.2 nats, and a shell comment calling equal width in 1/d^2 equal volume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 17:27:56 +02:00
leonarski_fandClaude Opus 5	01ad1e1743	calibration: only report a tilt the fit actually measured ... The detector tilt is refined by default, and on a pattern that cannot separate it from the beam centre the fit returns one anyway - there was nothing to stop it. Both displace a ring's radius as cos(phi), and only how that amplitude grows with the ring's radius tells them apart, which takes two well-sampled rings. At 500 mm on the LaB6 series only two rings reach the detector and the outer one is barely there: the tilt came out at the opposite sign to every shorter distance, dragged the PONI 28 px, and bought a residual of 0.960 px against 0.962 pinned. The covariance says so plainly - 0.1 sigma, and a beam centre quoted to +-180 px. So ask it. A tilt is kept only where the fit had it free AND it stands at least three times its own uncertainty; otherwise rot1/rot2 go back to the header's values and the beam centre and distance are refitted around them. Over the series the tilt stands at 50, 33, 15 and 8 sigma at 110 to 300 mm and 0.1 at 500 mm, so any threshold between 2 and 5 gives the same verdict on all five - this says which regime a fit is in, not where a line was drawn. The declined 500 mm fit lands on a direct beam of 773.53 px, against 773.56 for the pinned fit measured independently. It is a rejection criterion and nothing more. Clearing it does not certify a tilt: that estimator is limited by systematics rather than by this sigma, and a coherent half-pixel error in the ring positions fakes a tilt of the usual size while leaving sigma small. The report says "refined", never "verified". Writing the gate turned up a related fault in the pass loop. RingOptimizer pins the tilt by itself when every point it is given lies on one ring, and on a barely-sampled pattern a later pass lands in exactly that state - which froze the tilt at whatever the FIRST pass had produced and returned it with sigma zero, an unmeasured tilt wearing the appearance of a fixed one. The gate reads that as "not measured" and refits pinned, which is why it is stated over the geometry that gets reported rather than over what the last fit happened to do. Both paths are covered, profile and spots; the spots path was reporting a refined tilt as declined for the same reason. Two things measured and NOT taken: A robust loss. A Cauchy loss scaled to the previous pass's median residual changed nothing on the series - rms 0.415 to 0.421 at 110 mm, no case improved, every direct beam within 0.06 px. Ring points are per-sector peaks that already had to stand 3 sigma clear of their own background, so there are no gross outliers left to reject. Recorded at the call site rather than left as an unused option. A quality gate that refuses a bad calibration. Three candidate signals, all measured against naming the wrong standard on LaB6 data: sigma(PONI) does not see it at all (0.52-0.65 px, indistinguishable from healthy); the residual only half sees it (3.2-3.6 px wrong against 0.4-1.0 right, but a correct run from a wrong header sits at 1.0-2.4 and would be caught too); and the seed's match score is dominated by how many rings the calibrant lists, scoring 0.29 for a perfect LaB6 fit against 0.21 for a wrongly named silicon. None of the three separates, so no gate is shipped. What the run does say is the recovered distance against the header, and a wrong standard moves that to 446 mm on a 110 mm exposure - unmissable, and the operator's call rather than a threshold's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 17:07:37 +02:00
leonarski_fandClaude Opus 5	fba5435c38	calibration: report what the ring fit knows about its own answer ... The ring fit reported the SCATTER of its measurements (rms, and the beam-centre standard error that follows from it) but nothing about how well the fit pinned each parameter. Those two part company exactly where a calibration is worth doubting: as the rings run out, the tilt and the beam centre stop being separable - both displace a ring's radius as cos(phi) and only the way that amplitude scales with radius tells them apart - so the fit can sit tightly on the few points it has while being free to spend tens of pixels of beam centre on a tilt the data do not support. Take the covariance of the converged problem from Ceres and report it. Measured on a LaB6 distance series, the fitted tilt is 50 sigma at 110 mm and 0.1 sigma at 500 mm, where only two rings reach the detector; at 500 mm the fit quotes its own beam centre to +-180 px and its tilt to +-2.9 deg on a 0.35 deg value, and the correlation between them is 1.000. Nothing acts on this yet - it is printed so the next change can gate on it. Ceres returns the bare (J'J)^-1 of an unweighted problem, so it is scaled by chi2 per degree of freedom; that leaves the sigmas in pixels, mm and radians whatever unit the residual is stated in. Cost is one 5x5 SVD per run, below the noise of the surrounding I/O. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:11:18 +02:00
leonarski_fandClaude Opus 5	03456dcddd	calibration: an option to hold the detector tilt fixed ... --mode calibration fits five parameters - beam centre, distance and the two PONI tilts - and a program that cannot express a tilted detector has nowhere to put the last two. Dropping them after the fact is worse than never fitting them: the centre and the distance of a tilted fit have already absorbed the tilt, so the flattened geometry is right nowhere. rugnux --no-refine-tilt, the "Refine detector tilt" tick box on the viewer's Calib page and RingOptimizer's refine_tilt argument hold rot1/rot2 at the value the geometry came in with and fit the remaining three. That is the best flat-detector answer, and the one such a program would refine to itself. Measured on a five-distance calibrant series. At short distance the tilt is real and reproducible - three independent fits agreeing to 0.01 deg, radial rms 1.4 -> 0.4 px - and its direct beam agrees with the background beam-centre estimator to 0.05 px, so the tilted model is the physically right one. The pinned fit then displaces the centre 2.6 px to absorb the tilt and lands within 0.03 px of the same place at every distance. Past ~300 mm, where only two rings reach the detector, the tilt is instead under-determined: it comes out with the opposite sign to every short-distance fit and drags the PONI 28 px while the rms does not move (0.960 against 0.962). The existing degeneracy guard only fires on a strictly single ring, so it does not catch that; declining a tilt that does not pay for itself in rms is left for a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 20:44:12 +02:00
leonarski_fandClaude Opus 5	fed077e683	geometry: hold the detector plane as axis vectors, and give the discrete part its own home ... The detector plane was three PONI angles and nothing else, so the two things it cannot express - an image mirrored in Y, and one mounted at a multiple of 90 degrees - had no home at all. They are now the DetectorOrientation carried by the detector setup, composed with the PONI rotation into one orthogonal matrix whose columns ARE the fast axis, the slow axis and the sample->PONI normal: lab = R(rot1, rot2, rot3) * Delta * ( (x-bx)*p , (y-by)*p , distance ) GetFastAxis/GetSlowAxis/GetNormalAxis read those columns and DetectorAxes() sets the plane from them, decomposing back to the angles; PoniRotMatrix and PoniAnglesFromMatrix are the conversion in both directions, exact on the canonical branch (rot2 in [-pi/2, pi/2]) and with a stated convention at gimbal lock. The angles stay stored rather than re-derived, so a geometry given as angles is written back as the same angles, to the bit. Delta is never inferred. In particular an arbitrary rot3 is NOT decomposed into a quarter turn plus a residual: rot3 is a fitted quantity, and a least-squares step must not be able to turn the stored image. It is set only where something states it - the detector setup, --detector-mirror-y / --detector-quarter-turns, or the value a file this system wrote records - and defaults to the identity, which makes the whole change a no-op for every existing detector and every existing file. It is a different setting from DetectorSetup::mirror_y, which flips the MODULE LAYOUT while an image is assembled and so decides what the stored pixels are. Merging the two would apply the mirror twice for every modular detector, or change the pixel content of every file written; both are ruled out. The new one earns its keep exactly where the old one is a no-op: a detector whose image arrives already assembled has no layout to flip. Both generators are signed permutations of the in-plane offset, so they preserve the distance from the PONI. That is why almost nothing downstream changes: everything needing an azimuth already goes through LabCoord, and everything that does not needs only a radius. The two hand-written copies of the rotation - XtalResidual and RingOptimizer - take the discrete part as four constants next to cos_rot3/sin_rot3, since it acts in the detector frame where rot3 acts in the laboratory and cannot be folded into it. RingOptimizer needs it despite being a radial fit: it fits the tilt, and the discrete part changes which way the tilt tips a ring. Carried as two optional CBOR keys and two detectorSpecific datasets, both back-compatible; the NXmx module axis vectors and the translation direction stop being hardcoded and are computed from it, reproducing today's values exactly at the identity. GetPoniRotMatrix is renamed GetDetectorMatrix, because it is no longer only the PONI rotation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	1ab257af6c	v1.0.0-rc.125 (#32) ... This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-18 16:17:21 +01:00
leonarski_f	c67337cfe1	v1.0.0-rc.72	2025-09-08 20:28:59 +02:00
leonarski_f	bb32f27635	v1.0.0-rc.70	2025-08-27 06:21:10 +02:00