Three related changes to the FFT candidate path, batteried together because they touch the same
function.
A COPLANAR CANDIDATE REACHED REFINEMENT. ReduceResults filtered triples on lengths and angles only -
the 30-150 degree bound admits any flat combination - and there was no volume test. On one dataset 41
of 5535 candidates had |V|/abc below 0.05, with a clean decade gap to the next, and three of them
reached the optimizer. UnitCell is float, and for a cell that flat the metric determinant is around
1.5e-7, so float32 gets its sign wrong 19% of the time where float64 never does. The guard against a
negative argument to sqrt then CREATES the singularity it was meant to prevent: it puts c in the a-b
plane, the reciprocal volume is 1/0, and the residual is 0 times infinity. Ceres reported a
not-a-number Jacobian and wrote several hundred lines of solver output per failed solve.
VolumeFraction() is |V|/(|a||b||c|), rejected below 0.02 - about 1.1 degrees off flat, ten times below
the flattest real candidate observed and a thousand times above where float loses the sign. It is
enforced at the producer and at the two optimizer entry points. Note the existing sanity checks use
ABSOLUTE volume, which a 320 cubic-angstrom flat cell passes. The same reciprocal-volume division is
now guarded at the two remaining sites that share the pattern.
A SHORTLIST CONFINED TO ONE PLANE cannot close a cell, and the row it is missing is the plane normal.
That is detected from the scatter-matrix eigenvalue ratio - measured, degenerate clouds score 2e-5 to
3.3e-4 against 0.026 or more for every non-degenerate one, a factor of eighty - and one further
transform is spent with the same direction count inside a three-degree cap about the normal, so the
plan and buffers are untouched. More directions cannot substitute: at the exact true direction the
long axis ranks 1422 of 16384 by prominence while the shortlist cut is four times higher. Ranking, not
sampling, is the obstacle. A four-fold denser grid was measured and rejected - it reaches the same
answer to three decimal places and takes a run from 2.5 to 8 GB of device memory.
fft_min_unit_cell_A is reachable as --fft-min-unit-cell and is lowered automatically by -C, mirroring
how the maximum is already raised. The default of 10 is unchanged: a lower floor admits spurious
sub-cells on protein data, and over 73 protein runs the floor was never lowered while the sibling
maximum did fire twice, so the path is live and correctly inert.
Corpus of 93 datasets, both arms, one build: 72 bit-identical on report content and p.hkl checksum, 13
failing identically, and the count of working datasets rises by one. The volume guard fires on 58 of
93 and 47 of those stay bit-identical - it fires constantly and almost never changes an answer, which
is what it should do. Solver chatter falls from 919 lines across three datasets to none. The cap
fires on 4 of 93, none of them in the in-house or private arms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW
fft_max_unit_cell_A was both the default and an enforced check_max, so 500 A was the longest
basis vector the FFT could ever return: FFTIndexer sizes its projected histogram from that value
and the transform's last usable bin IS that length. Of the PDB's 206950 X-ray entries, 1091
(0.53%) have an axis longer than that and were unindexable by construction. The accepted range
now goes to 1200 A, which leaves 8. The DEFAULT is unchanged at 500 - the histogram is sized from
the value in use, so nothing pays for the wider range unless a caller asks for it.
The peak picker's running-mean background was truncated at the ends of the spectrum rather than
slid inward, so a peak within bg_half (~15 A) of either end - which is exactly where the longest
cells sit - was judged on a one-sided background, biasing its prominence by however much the
spectrum sloped there. Keep the window a constant width and slide it. Both bounds stay
monotonically non-decreasing in j, so the GPU kernel's running sum is still valid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.
**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.
**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.
**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**
Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress).
Reviewed-on: #45
This is an UNSTABLE release and not recommended for production use (please use rc.11 instead).
* jfjoch_broker: Experimental rotation (3D) indexing
* jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values
Reviewed-on: #18
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5