7 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	555c686205	merge: bridge a rocking event by an angle rather than by a frame count ... A rotation reflection arrives as a run of partials and is cut into rocking events wherever two successive frames are more than MAX_FRAME_GAP apart. That constant was 2.0 frames, in five places, and the quantity it stands for is not a frame count at all: it is an angle, a reflecting range of a tenth to half a degree. At a tenth of a degree per image two frames is a fifth of a degree of dead rotation and the bridge does what it was meant to; at a degree per image it is two degrees, as wide as a whole event, and two genuine crossings of the Ewald sphere are joined into one "full". Reconstructing the events on a paired control - the same photons sliced two ways - the fraction of events whose partialities sum past 1.5 is 14.3% at 1.0 degree against 6.0% at 0.1, the fused ones spanning a median of five frames. So the gap becomes half a degree of rotation, floored at one frame so an event is never cut at its own neighbours and capped at the two frames that were always allowed. For any oscillation of 0.25 degrees or less the quotient is at least two and the cap returns the literal 2.0f, so every finely sliced sweep is bridged exactly as before, on both the CPU and the GPU path. The value is taken once per run and carried to the device in CombineParams rather than recomputed in the kernel, so the two paths compare the same float by construction and the bit-parity contract the combine documents is preserved. The defect above is measured; the repair is not. No arm has been run with this change, which is why it is a commit of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:09:08 +02:00
leonarski_fandClaude Opus 5	8b88074001	rugnux: the joint post-refinement's gate is asked of the geometry it commits ... Five corrections to the joint fit, none of them meant to move a well-conditioned result. EffectiveCellFromParams used M_PI, which is not standard C++ and which MSVC does not define without _USE_MATH_DEFINES. image_analysis/ is on the viewer's dependency path, so it takes PI from JFJochMath.h like the rest of the tree. The bounds test ran on the fit-half solution, and what the run actually commits is the re-fit on all the reflections that follows it. That second solve moves the answer, and nothing re-asked the bounds of it, so the committed distance was held only by the solver's +-5 % box rather than by the 1 % the gate advertises. The test is now a function of a candidate geometry and is asked twice - once of the half that earns the right to fit, once of what will be written - and a re-fit that lands outside leaves the run at its header geometry, named in the log like the other refusals. It is inert on the corpus: over the 98 committed fits the largest committed move is 0.709 % of the distance, 0.885 % of a cell length and 1.07 px of the beam, so nothing that commits today stops committing. The bounds covered the distance, the beam and the cell lengths and left the cell ANGLES - beta for monoclinic, all three for triclinic - free to the solver's +-0.05 rad, which is +-2.86 deg. The uniform scale this replaced could not move an angle at all, so that was a new unguarded degree of freedom on the two systems whose conditioning is weakest. The bound is one degree, chosen from what the fit actually does: the largest angle it moves anywhere on the corpus is 0.35 deg, on a monoclinic beta, and the two triclinic cases with the largest moves are 0.09 and 0.07 deg. A degree is about three times the worst of those and still well inside the solver's box, so the box goes on reaching everywhere the gate accepts and no fit that commits today is refused. The gate pooled the two residual families into one held-out mean. With both caps saturated that is some 30000 positional values against 10000 excitation ones, so the excitation residual - the only thing in the problem that identifies the cell SCALE - is outvoted three to one in the decision that commits the cell. Neither family may now degrade. This one is not free. It refuses 1 of the 98 fits the corpus commits, a cubic case whose held-out excitation rose 15 % while its cell scale moved 0.75 %; refused, that dataset keeps its header geometry and lands where the two-step arm did, R_meas 0.2699 -> 0.2724 and ISa 3.15 -> 3.10 with the space group unchanged, and its cell deviation from the reference goes back from 0.05 % to 0.40 %. That is the shape of failure the split is meant to catch, and on the one dataset where the two disagree it costs a real if small improvement. The changelog claimed the refined distance "no longer depends on how wrong the file's distance was". It does. Swept over a millimetre of header distance the committed distance still moves monotonically and the merge degrades with it; the dependence is reduced about thirteenfold, not removed, and the line now says reduced. Finally the log printed the half-data fit's distance, beam and cell beside the word COMMIT while different numbers were committed - 107.908 mm printed against 107.913 mm applied two lines further down, on one sweep. It now prints whichever geometry the verdict is about, and prints the cell angles beside the lengths, since the angles are refined now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:00:24 +02:00
leonarski_fandClaude Opus 5	bb2a42774a	rugnux: the rotation post-refinement fits the crystal and the detector at once ... Post-refinement scaled the pass-1 lattice by one scalar against the observed rocking angles (step A), then read the detector distance off that scaled cell (step B). The split was there because the positional residual is degenerate with the cell scale - which is true of the positions ALONE, and is exactly what the excitation residual already computed in the same struct breaks. So the degeneracy that motivated the split was resolvable inside one problem all along, and splitting it manufactured error twice over: * Pass 1 frees the whole lattice against a frozen distance, so what it absorbs of a header distance error is ANISOTROPIC (0.16 % on the crystal measured), and no single scale can undo it. * Whatever bias is left in that scale goes straight into the distance, since the distance is only ever determined relative to the cell. Over six runs spanning three wavelengths of one crystal, distance error (%) = 1.33 x step-A bias (%/axis), r = 0.9947 - and the bias changed SIGN with the wavelength (-0.067 % at 1.9 A, +0.049 % at 2.7 A, +0.19 % at 3.3 A), reproducing the whole wavelength ordering of the distance error. The two steps' own printed numbers say it, on 112 of the corpus's datasets. Comparing each run's held-out POSITIONAL residual at the unrefined cell with the same residual at step A's scaled cell and at step B's committed geometry: step A leaves it WORSE than the unrefined cell on 110 of 112 (median 3.41x), and step B - which does fit that residual - cannot get back: its committed value is still worse than the unrefined cell on 110 of 112 (median 2.97x). Step B was improving on a baseline step A had corrupted. This part of the measurement is of the old code alone and owes nothing to the replacement. Now one Ceres problem: free crystal orientation, the cell (every parameter the crystal system leaves free), goniometer axis, detector distance and beam centre; residuals are the positional detector<->reciprocal one at each partial's observed spot and the distance-independent Ewald excitation one at each rocking centroid. Same deterministic hkl-hash split-half gate as before, with the move bounded: distance and every cell length within 1 %, beam within the existing bound. Detector tilt stays fixed (gauge-coupled to the orientation on one crystal). Committed, it leaves the held-out positional residual BELOW the unrefined cell's on the same 110 of 112 (median 0.90x) - though that residual is half its own objective, so read that line as a sanity check, not as an independent arbiter. The rotation-scale fit that followed step A now runs on the jointly committed lattice and axis; it is unchanged otherwise and reproduces its previous k to five digits. MEASURED. Three wavelengths of one rotation crystal, scored against an independent determination of the cell (unconstrained joint fit to XDS spot centroids) and against XDS's own refined cell: set cell error vs XDS committed distance R_meas ISa 1.9 A -0.048 % -> -0.041 % 60.274 -> 60.279 mm .0725/.0725 31.2/31.1 2.7 A +0.057 % -> -0.058 % 60.360 -> 60.266 mm .0681/.0676 25.8/26.3 3.3 A +0.230 % -> -0.033 % 60.491 -> 60.267 mm .1193/.0835 9.2/14.9 The three committed distances now agree with each other to 0.013 mm and with the external value (60.26 +- 0.04) instead of drifting 0.22 mm with the wavelength. Anomalous peak heights at the known sulphur sites rise from 7.82/8.18 to 10.09/9.04 - the best previously reachable state was 10.08/9.10 and required knowing the answer. The bistability documented on the long-wavelength set is gone. Sweeping the header distance over 1.0 mm, the two-step commits 60.49-60.54 however it starts (or refuses outright, losing the dataset: R_meas 0.276, ISa 2.9); the joint fit commits 60.256-60.313, tracking the truth, and lands in the correct cell basin every time. Same from two starting beam centres. CORPUS. 115 datasets, paired against the tip, 113 processed in both arms: * cell deviation from the reference: 58 better, 38 worse, 16 unchanged (sign test p = 0.052); median 0.171 % -> 0.128 %. The asymmetry is in the magnitudes: the largest improvement is 2.4 percentage points, the largest regression 0.125. The 24.4-point figure at the head of that list is not a cell change at all - it is a hexagonal lattice the base arm described in its C-centred orthorhombic setting, which a sorted-axis comparison reports as a large difference; the scorer's own cause for that dataset is an under-called point group, and the primitive volume ratio passed in both arms. * POINT GROUP == reference 105 -> 106. Exact space group 82 -> 81: one screw-axis call (P2 -> P2_1) on a crystal whose R_meas is above 139 %, where the cell itself improved. * V ratio in [0.97, 1.03]: 106 -> 106. Datasets processed: 113 -> 113. * R_meas and ISa are confounded here: of the nine material regressions, seven reached a FINER resolution limit, which raises R_meas and lowers ISa mechanically. The two that did not are small (+0.55 pp R_meas, -0.9 ISa). One unconfounded rescue is large - R_meas 105 % -> 27 % at 0.72 -> 0.51 A on a small-molecule set whose uniform scale step A had refused. Cost: 90-150 ms on runs of minutes. Bit-identical across repeat runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:00:24 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00