v1.0.0-rc.164 #74

Merged

leonarski_f merged 44 commits from rc164 into main 2026-08-26 22:47:00 +02:00

Conversation 0 Commits 44 Files Changed 208

+7144 -748

leonarski_f commented 2026-08-26 22:35:47 +02:00

Owner

Copy Link

Copy Source

• rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new 9. DIFFRACTION ANISOTROPY section in <prefix>_report.txt and matching _reflns.pdbx_aniso_B_tensor_* / _reflns.jfjoch_aniso_* items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a NOT DETECTED / DETECTED / CANNOT DETERMINE verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction.
• rugnux can hand its integrated observations to another scaling program: --export-unmerged writes <prefix>_unmerged.mtz, an unmerged MTZ readable by aimless, pointless, careless and iotbx.merging_statistics, in --mode mx and --mode scale alike. Each rotation reflection's partials are summed into one full; --export-unmerged-partials writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are.
• rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. --adaptive-integration-radius=off restores the fixed radius and an explicit --integration-radius still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log.
• rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4.
• rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning.
• rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked UNTESTED and warned about where it is adopted, as coming from the lattice metric rather than from the intensities.
• rugnux --mode scale re-merges a _process.h5 in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under /entry/MX/reindexMatrix where the lattice was re-seated, and --mode scale also reports the Wilson B-factor estimate instead of WILSON_B= nan. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a _process.h5 must apply reindexMatrix where it is present.
• rugnux installs on its own, as a package called rugnux - dnf install rugnux or apt install rugnux - instead of arriving inside jfjoch-viewer. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a jfjoch-viewer from rc.163 or earlier, which still owns /usr/bin/rugnux, upgrades cleanly rather than failing on the duplicate file.
• rugnux is also a standalone download, built for arm64 as well as x86_64: rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz and rugnux-<version>-win64-cuda<major>.zip on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware.
• Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so rugnux and jfjoch_viewer need nothing but an NVIDIA driver, and only to use the GPU. The .rpm/.deb continue to take cuFFT from the distribution. The developer utilities jfjoch_extract_hkl and jfjoch_recompress are no longer packaged anywhere.
• Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged.
• The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
• The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the vm_flags kernel interface was backported into the 5.14 kernel.
• A data collection started with async_start that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by /wait_until_running and /wait_till_done instead of as a timeout and a successful collection respectively. The error message is the one the writer gave.
• A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to Inactive with an error message and has to be initialized again, instead of sitting in Idle looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted.
• A failed /initialize is reported to /wait_until_running and /wait_till_done as soon as it happens, instead of when their timeout expires.
• space_group_number accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent.
• The results report's REPORT_VERSION is 3, two sections having been added. Existing key names and table columns are unchanged.
• The merged statistics table has 9 resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. --resolution-shells sets a different count.
• rugnux --model now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the .mtz/.cif/.hkl come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible.
• rugnux --model can resolve the indexing ambiguity of a serial stills run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (-C / -S). Without one or the other, a merohedral serial run still merges both hands together and says so.
• The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of -z / --model resolves it in each case. The long reference pages now carry a table of contents.

* rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents.

leonarski_f added 44 commits 2026-08-26 22:35:47 +02:00

rugnux: export the integrated observations as an unmerged MTZ ... 5248b1bc00

Adds --export-unmerged <file.mtz>, in --mode mx and --mode scale alike.
rugnux could only write merged reflections, which closed it off from the
programs that consume unmerged observations - aimless, pointless, careless
and iotbx.merging_statistics all read this file.

The column layout follows what pointless itself writes (H K L M/ISYM BATCH
I SIGI FRACTIONCALC XDET YDET ROT LP FLAG), with DELPHI, ZETA, BGMEAN and
BGVAR added because we have them and a scale model can use them.

Three details decide whether the file is usable:

Lorentz-polarization is applied and recorded in LP. It is per-observation
geometry spanning a factor of ~120 across a sweep and no reading program
can recover it; merging with no scale model at all gives R_meas 0.372 /
0.346 / 0.324 in the low shells without it against 0.183 / 0.218 / 0.257
with it. The partiality is not divided out - it stays in FRACTIONCALC -
and neither is the per-image scale, since these programs fit their own.

The observations are partials and are flagged as such, packed as
256*M + ISYM with LDTYPE=1. Without the flag pointless reads every part as
a whole observation and mis-assigns the symmetry; with it, a tetragonal
case assembles 8.29M parts into 1.15M observations and comes back as its
own space group at confidence 0.92.

The scan axis is written as the negation of the stored goniometer axis,
which is the Cambridge convention: against pointless's own orientation
matrix that agrees to 0.6 degrees, where the axis as stored disagrees by 40.

Merging the exported file with aimless reproduces rugnux's own merge to
CC 0.9997 on a tetragonal case and 0.9978 on a cubic one, and the
anomalous signal survives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Drop Ceres' LAPACK, METIS and SuiteSparse back-ends: nothing calls them ... 86b7e07131

Every Ceres solve in the project selects a dense linear solver explicitly --
DENSE_QR in PostRefine, RingOptimizer and StillsPartialityRefine,
DENSE_NORMAL_CHOLESKY in XtalOptimizer, DENSE_SCHUR in GeometryRefiner -- and
none of the seven Solver::Options sites sets dense_linear_algebra_library_type,
which defaults to EIGEN. ceres::Covariance, the other SuiteSparse consumer, is
never instantiated. So no sparse factorization and no LAPACK call is reachable.

Measured rather than argued: an LD_PRELOAD shim exporting dgeqrf_, dpotrf_,
dtrtrs_, spotrf_ and METIS_NodeND, each aborting on entry, let a full mx run
(1800 frames, index -> refine -> integrate -> scale -> merge) finish with every
counter at zero.

The change is a no-op on results. Baseline and modified binaries produce
byte-identical run.mtz, run.hkl and run.cif on a rotation dataset at -N 1, and
two baseline runs give that same signature, so the comparison is meaningful.
Only the DATE= stamp in the report differs.

What it buys is six fewer shared libraries: libopenblas and libmetis, plus
libgfortran, libquadmath, libgomp and libz behind them. rugnux links its own
zlib statically -- zero undefined zlib symbols before and after -- so the
dynamic libz only ever arrived through OpenBLAS.

It also makes the build reproducible across hosts. SUITESPARSE defaults ON and
self-disables only when its probe fails, so a machine where SuiteSparse_DIR is
not found already compiled CERES_NO_SUITESPARSE while one carrying the
SuiteSparse CMake config would not have. Pinning it removes that divergence,
and shortens what a cross-compiled aarch64 build has to supply.

EIGENSPARSE stays ON, so a sparse solver remains available if one is ever
selected, without an external library.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h

rugnux: sum each rotation reflection's partials into one full on export ... bb5aaca68e

--export-unmerged now writes one row per reflection with its partials
summed, which is how a rotation dataset is normally handed to a scaling
program. The previous behaviour is still available as
--export-unmerged-partials.

Partials made the file's usefulness depend on the reading program's
partial-assembly settings. On an orthorhombic case only 31% of partial
sets passed aimless's default PARTIALS test, so it merged 2% of the file:
multiplicity 4.7 against our own 12.4, CC1/2 0.788, and CC against our
merged intensities 0.806. Summed, aimless uses 89% of the file and rejects
nothing: multiplicity 12.5, CC1/2 0.931, completeness 96.1 to 98.0,
pointless Laue confidence 0.669 to 0.792, CC 0.848. The other two test
crystals are ties overall and win their outer shells, and the anomalous
signal comes out stronger from the fulls. iotbx.merging_statistics, which
never assembled partials at all, now reports real merging statistics
rather than statistics of parts.

Events are grouped by Combine()'s own rule so the file and the merge
assemble exactly the same reflections; the sum is plain with variances in
quadrature and no partiality division, which reproduces the merge's full
times FRACTIONCALC to a few percent.

Per-part metadata is combined on a partiality-weighted centroid rather
than an intensity-weighted one: 8-41% of events have parts summing to zero
or less, where intensity weights are undefined or sign-flipped, and 10-21%
of intensity-weighted centroids fall outside the event's own frame range.

Events below --min-partiality are not written. Declaring a reflection whose
rocking curve caught a thousandth of itself as a complete measurement gives
a scaling program noise with a small sigma and it weights it heavily; there
is no partial test to catch it once it is a full. This is the same constant
already applied to the same quantity in the merge, and it is what takes the
orthorhombic case from CC 0.219 to 0.848. --min-partiality 0 writes every
event.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

rugnux: name the unmerged export from the prefix, and drop centring absences ... ef07770c5f

--export-unmerged and --export-unmerged-partials no longer take a filename;
they write <prefix>_unmerged.mtz and <prefix>_unmerged_partials.mtz, as
--write-process-h5 already does for <prefix>_process.h5.

Lattice-centring absences are no longer written. They are integrated on
purpose - prediction runs in P so the space-group search can confirm or
disprove the centring - but in an exported file they cost the reader the
answer: pointless reads the interstitial nodes, takes the lattice for
primitive, and demotes I23 to P2_1 3, while aimless roughly halves its
<I/sigma> merging them. They were also only present when the indexer had
settled on the conventional centred cell, so the file's content depended on
which of two equivalent cells it picked.

Screw and glide absences are still written. They are the evidence a space
group is chosen on, so a program reading this file can check that choice
independently; deleting them turns pointless's test into an assumption
("we cannot determine if the reflections are truly systematically absent").
Deleting them changes no merging statistic. XDS and DIALS draw the line in
the same place - XDS ships screw absences and never measures a centring one,
and DIALS's absence machinery has only screw-axis classes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

PCIe driver: guard vm_flags_set for RHEL, which backported it into 5.14 ... 59ebe8d9ae

RHEL and its derivatives report LINUX_VERSION_CODE 5.14.0 for the whole of el9
while backporting newer kernel APIs into it, so the version check alone picked
the pre-6.3 branch on every el9 >= 9.5 and the module failed to build with
"assignment of read-only member 'vm_flags'". HAVE_VM_FLAGS_SET could override
it, but nothing sets it and dkms.conf does not pass it, so a build from the RPM
had no way to reach it.

vm_flags_set() and the const vm_flags member appear between kernel-5.14.0-440
and -450.el9, during the 9.5 development stream. RHEL_MINOR tracks the released
minor exactly, so RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9,5) is the test:
9.0-9.4 (GA -427) do not have them, 9.5 (-503), 9.6 (-570), 9.7 (-611) and
9.8 (-655) do. el10 is 6.12-based and LINUX_VERSION_CODE already covers it, as
it does the removal of no_llseek that the other guard in this driver handles.

Spelled through a JFJOCH_RHEL_GE() helper rather than inline, because the
obvious form

  #if defined(RHEL_RELEASE_CODE) && RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9,5)

is a hard error on both gcc and clang for every non-RHEL kernel: && does not
short-circuit past a parse error, and the undefined function-like macro leaves
"0 (9,5)" behind.

Also corrects > to >= on the upstream check. The wrappers landed in 6.3, so a
vanilla 6.3.0 kernel was taking the pre-6.3 branch.

Generated code on el8 is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PLMZgfMEdtPBZt1F2rNbRA

PCIe driver: build DKMS modules for the target kernel, not the running one ... 8db8267ff8

dkms.conf quoted the make binary: MAKE="'make' -C src/ all". dkms passes the
kernel it is building for by rewriting the leading "make" of that string into
"make -jN KERNELRELEASE=$kernelver" (dkms 3.2.1, line 1446) - an anchored
prefix substitution that the quotes defeat, so nothing was ever injected. The
Makefile then hardcoded /lib/modules/$(shell uname -r)/build and would have
ignored it in any case.

With AUTOINSTALL=yes the module was therefore always compiled against the
running kernel and installed into the tree of whichever kernel dkms was
building for. A dnf update that pulls in a new kernel builds against the old
one and drops the result in the new kernel's /extra, where it fails to load on
the next boot. Crossing RHEL 9.4 to 9.5 it would also compile the wrong side of
the vm_flags guard, which is how this was noticed.

The Makefile now takes KDIR, defaulting to the running kernel exactly as
before, and dkms.conf passes ${kernel_source_dir} - which dkms resolves for the
target kernel before sourcing the conf, and which honours --kernelsourcedir.
Dropping the quotes also lets dkms's -jN through, so the recursive invocations
become $(MAKE) to keep the jobserver.

Verified by building against a kernel other than the running one: the module
now comes out carrying the target kernel's vermagic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PLMZgfMEdtPBZt1F2rNbRA

rugnux: report the Wilson B in --mode scale, and correct what the docs claim about the fixed integration radius ... 0f5a6e41d6

--mode scale called AnalyzeTwinning but not CalcGlobalWilsonB, so WILSON_B=
read nan there. Mirrors what --mode mx does; on one crystal scale mode now
reports 72.03 against mx mode's 71.91 on the same data, and where the fitted
B is not positive it still reads nan, which is the same verdict --mode mx
reaches. The merged .mtz and .hkl are byte-identical before and after.

docs/CPU_DATA_ANALYSIS.md and BraggStencil.h both said the flux the fixed r1
disk does not capture "is a function of resolution alone, which the per-shell
scale absorbs". Neither half is true. The loss carries a directional component
of several A^2 on top of the isotropic part. There is no per-shell scale, and
there cannot usefully be one: every scale is fitted against a reference built
from a reflection's own symmetry equivalents, equivalents share s^2 exactly,
so any function of s^2 lies in the exact null space of the scaling model. What
actually absorbs the isotropic part is the Wilson B, which is degenerate with
it - so WILSON_B and _reflns.B_iso_Wilson_estimate carry an r1-dependent bias,
measured at 18 A^2 of spread across four radius triplets on one crystal.

The text also corrected a third error: in the default gaussian mode r1 is not
the integration domain at all - the intensity is a profile-fit amplitude over
a grid set by r2 - and a note now sits at the section a reader looking up the
Wilson B actually lands on.

Comments and documentation only for that part; verified by rebuilding and
confirming byte-identical output with no flags.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

rugnux: measure and report diffraction anisotropy ... 80e10f51b3

rugnux now says whether a dataset's fall-off is direction-dependent, and by
how much. It corrects nothing and truncates nothing: no intensity is changed,
no reflection is dropped on a directional criterion, and the written files do
not depend on direction at all.

Two quantities, because they are not the same thing. The anisotropic deltaB is
the range of the principal components of the anisotropy tensor - a rate of
fall-off. The diffraction limit along each principal direction is where
<I/sigma(I)> in a 20 degree cone falls through 2 - where signal actually runs
out. One battery case has only 0.28 A between its directional limits and a 58x
ratio in cone <I/sigma>, so reporting either alone would miss it.

The tensor is a Laue-constrained deviatoric ADP tensor fitted on INTENSITIES
with no positivity cut, by weighted Gauss-Newton over 12 shells x 60
directions with a free constant per shell. Fitting amplitudes after a
positivity cut, which is what xtriage and ctruncate do, destroys about 40% of
the measured anisotropy - the cut keeps only the positive noise excursions in
whichever direction has died, and that is the direction carrying the signal.
Against the same 38 merged files rugnux reads 1.24x xtriage's eigenvalue
spread and 1.61x ctruncate's; on strong near-isotropic data all three agree to
a few percent, and they diverge exactly where a direction has died.

The verdict is gated three ways - not detected, detected, or cannot determine
- against the dataset's own systematic floor, measured in the tensor
directions its Laue symmetry forbids. The floor cannot be measured on merged
reflections, which have exact Laue symmetry by construction, so the floor is
taken from the unmerged observations and the verdict is "cannot determine"
without them. Triclinic has no forbidden subspace and always returns cannot
determine. A cubic crystal returns exactly zero, because that is its symmetry
and not a measurement.

A second axis reports the resolution signature: a genuine Debye-Waller
fall-off is linear through the origin in s^2, and a deficit that is flat is
something else. Magnitude alone had promoted a crystal that is 68% not a
Debye-Waller B into the top five of this battery; it now reads not detected
with the caution attached.

Following Sheriff & Hendrickson (1987) Acta Cryst. A43, 118-121 for the tensor
and Popov & Bourenkov (2003) Acta Cryst. D59, 1145-1153 for the estimator.

The directional limits are written as jfjoch_ local mmCIF items rather than
_reflns.pdbx_aniso_diffraction_limit_*, whose dictionary definition is
explicitly the ellipsoid fitted to a diffraction cut-off surface - a
construction rugnux does not perform. The generic anisotropic B tensor items
are written.

Changes no existing number; only REPORT_VERSION moves, 1 to 2.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Space-group search: raise the twin-law H bound from 1.70 to 1.85 ... 3ea1206775

Re-measured over ~780 reports - three batteries and four de-novo
integration-radius arms - H is the only Stage-A guard that ever fires: 14
refusals on three crystals, and not one chi-squared-ratio refusal or b veto
anywhere.

Two of the three are the genuine trigonal twins, reading 1.90 and 1.91 on
every arm and 3.61-4.80. The third is a weak tetragonal crystal reading
1.73-1.80 that is not twinned: its 422 is confirmed by every operator
correlation, and three independent tests agree - our own L test on the
P2(1)2(1)2(1) merge it was demoted into, xtriage on that same merge ("no
twinning is suspected... most likely due to an NCS axis parallel to the twin
axis"), and the arithmetic that a twin at the implied fraction caps twin-pair
correlation at 0.55 while the four added operators score 0.84-0.90.

Its excess is the coverage limit the comment above the bound already names:
the added 422 operators exchange h and k, so they carry a systematic floor
the parent's sign-flipping 2-folds do not. The two populations therefore
collide at 1.80 and 1.90, not at 1.70, and the bound now sits at the midpoint
of that window.

Degrading the data - 4x sigma inflation, or an eighth of the sweep - drives
the ratio towards 1 and never up, and the estimator is precise (bootstrap
standard error 0.2-3.7% of the ratio), so the collision is a genuine
population overlap rather than sampling noise. That also rules out scaling
the bound by its own error, which at any useful multiple refuses half the
battery.

Inert on the shipped defaults: no crystal in the battery reads between 1.70
and 1.85 there. It changes one assignment on an experimental integration
radius, from wrong to right. The committed twin test is unaffected - genuine
promotions in that harness read 0.959-1.016 against 4.23-5.36 for its twin,
so nothing lies in the interval that moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

libjpeg-turbo: follow the parent build's toolchain when cross-compiling ... 46ed703eca

libjpeg-turbo is brought in with ExternalProject_Add rather than FetchContent,
because upstream does not support inclusion via add_subdirectory. An
ExternalProject is a separate cmake invocation that inherits nothing from its
parent, and the CMAKE_CACHE_ARGS here named only the feature options, so it
always configured for the build host.

That is invisible in a native build and fatal in a cross build: the library
compiles for x86_64 inside an aarch64 build and the mistake only surfaces at
the final link of whatever consumes it, as an architecture mismatch on
libjpeg.a rather than anything pointing at this file. rugnux reaches it through
JFJochWriter -> JFJochPreview, so it is on the path of every build, not just the
viewer's.

Forward the toolchain file and the compiler. CMAKE_TOOLCHAIN_FILE is empty in a
native build and cmake ignores it there, so nothing changes off the cross path.
libjpeg-turbo selects its own SIMD from the target architecture, so this also
gets the NEON kernels for free on aarch64 and stops nasm from being consulted at
all; a cross-compiled libjpeg.a comes out ELF aarch64 with 14 NEON objects.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h

Ship jfjoch_viewer and rugnux as two packages, and cross-build rugnux for arm64 ... 8d32024dd1

The viewer tarball also carried rugnux, jfjoch_extract_hkl and jfjoch_recompress.
A GUI download that contains a batch processor is confusing on its own, and it
forced anyone who only wanted rugnux to take Qt with it. JFJOCH_RUGNUX_ONLY is
the viewer-only subtree minus viewer/ and tools/, so it needs no Qt at all, and
each of the two options now yields one artifact holding one program:

  JFJOCH_VIEWER_ONLY   jfjoch_viewer   .tar.gz / installer / .dmg
  JFJOCH_RUGNUX_ONLY   rugnux          .tar.gz / .zip

The two CLI tools are still built and no longer installed anywhere. rugnux gets
a CPack component of its own, so its notices land in share/doc/jfjoch_rugnux.

What makes the split possible is linking cuFFT statically in both products. It
was the only CUDA component still dynamic - cudart and the fast-feedback indexer
were already static - and shipping it as a .so meant each self-contained archive
had to carry the library beside its executables and find it again through an
$ORIGIN rpath. That rpath was set on jfjoch_viewer alone, while the same tarball
installed three more executables, so those shipped with no runpath at all (cmake
strips the build-tree one, which points at the build host's CUDA directory) and
could not load the libcufft lying next to them. Static, an artifact is one file
that runs: the x86_64 rugnux tarball needs only libc, libstdc++, libgcc_s, libm,
libpthread, librt and libdl. The bundling code and the rpath both go away.

libcufft_static.a carries a relocatable-device-code object, so a consuming
executable needs a CUDA device link; CUDA 13 no longer ships the
libcufft_static_nocallback.a that used to avoid this. CUDA_RESOLVE_DEVICE_SYMBOLS
emits that step while leaving the host link driver alone, so the -march and
-flto flags CI passes still apply. The server build keeps the shared library:
its .deb/.rpm take CUDA from the distro, and every executable there would
otherwise need the same device link.

The guards that exclude server-side targets keyed on NOT JFJOCH_VIEWER_ONLY,
which a rugnux-only build does not satisfy - it failed on jfjoch_writer pulling
in JFJochImagePuller. They now key on JFJOCH_PORTABLE_ONLY, true for either
product, rather than on one product's option in five places.

docker/ubuntu2404 additionally cross-compiles rugnux for arm64 SBSA, which is
both DGX Spark (GB10, sm_121) and Grace Hopper (GH200, sm_90); one binary with
both fatbins runs on either. It needed three things beyond a cross gcc. arm64 is
published on ports.ubuntu.com and Noble's deb822 sources carry no Architectures:
field, so adding the architecture without splitting the entries 404s on every
index. The CUDA cross packages are in neither the x86_64 repo nor the sbsa one
but in cross-linux-sbsa, whose fatbinary_section.h ships only in the x86_64
target tree even though it is architecture-neutral. And HDF5 has two try_run
probes, so qemu-user-static is needed as CMAKE_CROSSCOMPILING_EMULATOR; every
other fetched dependency either guards its run-checks or has none.

The cross-built tarball is one ELF aarch64 binary carrying all 17 GPU modules
for both sm_90 and sm_121, needing no cuFFT and no CUDA runtime at run time.
It is untested on hardware: XDS is x86-64 only, so CI can show it links but only
a Spark or a GH200 can show it works.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h

CI: upload the rugnux artifacts the way every other job does ... 6a7404f03f

The three new rugnux jobs reached for akkuman/gitea-release-action@v1, which
does not exist on this Gitea: actions resolve against gitea.psi.ch/<owner>/<repo>,
so the runner tried to clone gitea.psi.ch/akkuman/gitea-release-action and failed.
That happens while preparing the workflow, before any step runs, so all three
jobs died immediately without building anything.

Every other job uploads with gitea_upload_file.py and the release token, so use
that: bash on the Linux jobs, powershell on the Windows one, matching the viewer
tarball and installer steps they sit next to. The Linux archives are renamed
.tar.gz -> .tgz first, as the viewer archive already is. actions/checkout is now
the only external action the workflow references.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h

CI: drive the Windows rugnux build from cmd, as the viewer build already does ... fd9e7bce1f

The job used shell: bash, and the Windows runner has no bash on PATH, so it
failed before running cmake. It also skipped the MSVC environment, which every
cmake invocation on that runner needs: build-windows calls vcvars64.bat, located
through vswhere, at the start of each step. Follow it exactly - cmd, the same
vcvars preamble, -G Ninja, and cpack from inside the build directory - and keep
powershell only for the upload, which is what build-windows uses there too.

CMAKE_PREFIX_PATH drops the Qt entry and keeps C:/deps: JFJOCH_RUGNUX_ONLY is
the portable subtree minus viewer/, so it needs Eigen and ZLIB and no Qt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h

Fix the packaging split where it was only exercised on one configuration ... 491312f263

Three failures, all from validating the rugnux path alone and assuming the rest
followed.

viewer/CMakeLists.txt had an unbalanced ENDIF: removing the cuFFT bundling block
took its opening IF(JFJOCH_CUDA_AVAILABLE) with it and left the closer behind, so
every configuration that adds viewer/ died on "flow control statements are not
properly nested" -- both Windows jobs and the viewer tarball.

The cross toolchain could not find Eigen. CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY
confines find_package to the target roots, and the ubuntu2404 image installs Eigen
from source in /opt/eigen-3.4 rather than as libeigen3-dev, so it sat outside them.
It is header-only, so serving it from outside the target roots is safe; the root is
now listed explicitly.

Static cuFFT is not available everywhere. The Windows toolkit ships no
cufft_static.lib, so CUDA::cufft_static does not exist there and linking it fails
at generate time. Condition on the target rather than the platform, so a toolkit
that gains or loses the static library needs no further change, and restore the
Windows DLL bundling for both products -- now in the top-level CMakeLists, since
rugnux needs it too and does not add viewer/. rugnux.exe therefore ships beside
cufft64_*.dll rather than alone; the Linux and cross artifacts are still one file.

That condition also has to keep the JFJOCH_PORTABLE_ONLY half. Testing only for
the target quietly moved the SERVER build onto static cuFFT as well, where every
executable -- broker, tests, tools -- would then need the CUDA device-link step and
would fail on an undefined __cudaRegisterLinkedBinary_* at link. Configure still
succeeds, so nothing catches that until the link; the server build keeps libcufft.so
exactly as before.

Checked by configuring all four flavours in the image CI actually uses -- rugnux,
viewer and server on x86_64 plus the aarch64 cross -- and confirming each links the
cuFFT it should.

The rugnux archives now also carry their target architecture, since that build
produces more than one: rugnux-<version>-linux-x86_64-cuda13.tar.gz beside the
aarch64 one. CMAKE_SYSTEM_PROCESSOR is the target's, never the build host's. The
viewer is x86_64-only and keeps its existing name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h

tests: follow REPORT_VERSION to 2 ... a4228b9c1d

80e10f51b added an anisotropy section to the result report and moved
REPORT_VERSION from 1 to 2, but left ResultReportTest asserting 1, so the
[Diagnostics] tag has been failing on this branch since that commit. The
assertion is the point of the test - a consumer greps that key - so it
follows the constant.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Space-group search: say what it tested, and what it could not ... 3175e65400

Three reporting changes. None of them touches a decision: the merged .mtz,
.hkl, .cif and _image.dat are byte-identical before and after on two test
crystals, and the space group and cell are unchanged. REPORT_VERSION moves to
3 and one warning appears where the second change is working.

The alternatives line was misleading. "Best space group: C2 or P21 or P2
(indistinguishable from these data)" sat forty lines below the run's single
cell, which is C2's. P2 and P21 are primitive: their lattice is a sub-lattice
of that one, their cell has half its volume, and their indices are not the
ones in the written files. A user acting on "P21" got a group name with no
cell, beside a cell belonging to a different group, with nothing saying so.
The report now names the alternatives whose centring differs from the chosen
group's, gives the volume ratio of the cell each implies, and says that
adopting one means reindexing. It stays byte-identical when every alternative
shares the chosen centring, which is every other case that occurs - the
enantiomorphic and origin-ambiguous pairs, where one cell really does describe
them all. It does not print derived cell constants: the primitive cell of a
C-centred monoclinic is not P2's conventional cell, and deriving that is a
reduction plus a per-group setting choice, which is more than a report line
should carry.

A centring the data could not test now says so. Prediction runs in P so the
search can confirm a centring from the reflections it extinguishes, but when
the indexer returns the primitive sub-cell those reflections are never
predicted, and an absent count of 0 reads as "predicts no absences" when it
means "none was measured". The search abstains correctly and the centring is
then taken from the metric, but none of that reached the user. The candidate
table gains a three-valued centring column and the report raises a warning
where the metric decides it. No decision logic changed - the flag is two
counts the search had already made, and nothing reads it back.

The twin-law H is now reported for every operator, and the adopted promotion's
ratio against its bound. It was previously visible only inside a refusal
message, which is why recalibrating that bound recently required rebuilding
the statistic from reference data rather than reading it out of runs we
already had. The ratio is reported at the Stage A promotion where it is
computed, not in the Stage B table, whose rows all share one point group and
would have printed the same number on every line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

CI: make the artifact checks capable of failing, and of passing ... 8fa1ab7b05

The rugnux tarball built and packaged correctly; the step that verifies it did
not. Two shell mistakes, both mine, both of the same family.

`! find /tmp/rgx -name 'libcufft*'` can never pass: find exits 0 whether or not
it matched anything, so the negation always fails. That is what broke the x86_64
job after a successful build.

The aarch64 checks were worse, in the way that matters. Piping a large producer
into `grep -q` kills the producer with SIGPIPE as soon as grep exits on its first
match, and `set -o pipefail` promotes that to the pipeline's status: on
`cuobjdump --list-elf | grep -q sm_90` over a 225 MB binary it fails a check that
should pass, and on `! ... | grep -q "x86-64"` the leading `!` inverts it into a
PASS -- so an x86 file leaking into an aarch64 tarball, the exact thing that check
exists to catch, would have been reported as clean.

Both now capture each producer's output to a file and grep the file, with explicit
if/exit rather than exit-status negation.

Verified by running the checks verbatim against the real artifacts rather than
only reading them: both tarballs pass, and the checks were confirmed to FAIL when
they should -- an x86_64 file planted in the extracted tree is caught, and an
architecture absent from the fatbin is reported missing.

The toolchain file now comes from the checkout rather than /opt/cross in the
image. It describes how to build this source, so it belongs with the source:
baked into the image, the Eigen fix in the previous commit could not reach CI
without rebuilding and re-pushing the image, and appeared to have no effect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h

_process.h5: record the change of basis, and the space group ... db908c6ec1

A _process.h5 was internally inconsistent whenever the space group re-seated
the lattice. The per-image reflections and lattices go to file as each image
is processed, but the conventional setting is only chosen afterwards, so the
file kept pre-reindex indices beside a post-reindex cell. Measured on an
affected dataset, max|d_file - d(hkl, stored cell)| was 18.6 A. --mode scale
then compared each frame's lattice against the stored cell, found none of
1800 acceptable, rejected every observation and died in the merge with
"resolution calculation failed".

/entry/MX/reindexMatrix now carries M with hkl_cell = M . hkl_written, and
the reader applies it, so everything it hands out is in the setting of
/entry/sample/unit_cell. Absent means the identity, so a file written before
this reads exactly as before. On the affected dataset M comes out
[[1,1,0],[0,1,1],[1,0,1]], det 2 - the primitive-to-body-centred basis its
volume ratio implied - and the same measure falls from 18.6 A to 1.8e-5.

Writing the reflections in the final setting instead was rejected: the
per-image writer is shared with the broker, which streams and cannot buffer a
run; and h,k,l, predicted_x/y and the per-image lattice are one consistent
statement about one image, which retro-editing the indices would silently
break.

Two things turned up while fixing it. There are three re-seat sites, not one
- the space-group search's own centred-lattice test re-seats too, and logged
nothing - so the matrix composes over all of them. And the space group itself
was never written: it was set only on the arm that searches, while a two-pass
rotation run reuses pass 1's group and takes the other arm, so the canonical
file carried a cell but no group and --mode scale merged in P1.

--mode scale now reproduces --mode mx on the affected dataset: same space
group and cell, 39329 unique reflections both, 100% of reflections common,
CC 0.99985, sum|dI|/sum|I| = 0.0081. The residual is three reflections in
11.09 M crossing an ice-band edge, because mx carries the integrator's d and
scale recomputes it from the cell. An unaffected dataset is byte-identical in
.mtz, .hkl and .cif.

An older affected file still cannot merge - M is not recoverable from it -
but now says so in 51 s, naming both cells and the -S/-C override to use,
instead of failing inside the merge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

rugnux: set the integration radius from the crystal's own spot width ... 48008e1447

On rotation data the signal radius is now r1 = clamp(round(2*r80), 4, 6),
where r80 is the 80% encircled-flux radius of the crystal's own spots. The
background ring keeps its area (r3 = sqrt(r2^2 + 133)), so r1 = 4 is the
shipped default bit for bit and 26 of the 38 battery crystals come out
byte-identical.

The width had to be measured somewhere new. rugnux already has one -
shell_sigma2[].tan - but it is a second moment taken inside the r1 disk it
would be setting, and it saturates at r1/2, so feeding it back measures the
cap and not the crystal. SpotWidth instead measures encircled flux over an
aperture fixed for the whole file (14 px, normalised at 8), in the pre-scan,
from spots the finder already produces on the frames the beam-stop projection
already reads. It touches no integrator output and runs before the first
integration pass, so there is no loop, it costs no extra frame reads, and
both passes - including the space-group search, which runs in pass 1 - see
the same radius. A default run pays a median 1.9 s.

k = 2 is not fitted. For a Gaussian r80 = 1.794 sigma, so r1 = 2*r80 is
3.59 sigma, where the truncated second moment recovers 0.990 of sigma^2. The
new test checks the estimator returns 1.794 sigma on a known Gaussian.

Battery, 38 crystals, both arms run twice: the space group is identical on
all 38 and 35 agree with the reference in both arms. Per shell on the 12
crystals the rule moves, 6 win and 4 tie, with mean per-shell <I/sigma> up
30.6, 24.6, 15.8, 9.1, 8.0 and 5.1 per cent and R_meas down as much as 23.8.
Runtime is neutral - 19m13s against 22m00s warm.

One crystal is a real cost and is named in docs/RUGNUX.md with its
workaround: an I222 case that is simultaneously the widest-spot and among the
highest-mosaicity in the set loses 28.5% of its observations at unchanged
completeness, because at r1 = 6 its predicted reflection density leaves the
background ring too few clean pixels. No cheap guard separates it - its
predicted spacing is mid-table, larger than five crystals that survive r1 =
12 - and the guard that would, on the measured drop rate out of pass 1, needs
a diagnostic channel out of both integration engines and is not yet
validated.

This depends on 3ea120677: at the previous twin-law bound of 1.70 the
wider radius costs one crystal its 422, refused on an H ratio of 1.73 even
though every operator correlation confirms the point group.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Anisotropy: say what was measured, and stop crying wolf ... 0da593b32a

Four reporting changes and a restored changelog line. The gate boundaries are
unchanged - see below - and no merged number moves: .mtz and .hkl are
byte-identical on two crystals, and REPORT_VERSION stays 3 because everything
here is an added key or prose.

A directional diffraction limit that is really the edge of the measured data
was printed as if it were the crystal's. One crystal reported 1.09 / 0.99 /
0.99 A against a 0.994 A cutoff, so two of its three limits were the cutoff.
A censored limit now prints with a "<" and sets ANISOTROPY_D_MIN_CENSORED. It
fires on 4 of 28 crystals, and on two of those all three limits are censored,
which means their reported directional spread was binning noise.

The caution about a too-high symmetry assignment fired on every crystal with
one deviatoric degree of freedom - every tetragonal, trigonal and hexagonal
case, 17 of 38. A caution that fires on 45% of runs is noise. It is now
conditional on the signature that actually indicates it, a large
symmetry-forbidden signal with a gate that established nothing, and fires on
2 of 38: the crystal already on record as space-group unstable, and one
other. The forbidden-direction z is now reported so the reader can see it;
its battery median is 6.5.

The verdict line quoted deltaB_linear while the headline reported deltaB, and
the two differ - deltaB_linear is the larger on 12 of 31 crystals. It now
writes out the arithmetic, says which number the gate tests, and says which
to act on. The observation count the floor was measured on is reported, and
the mmCIF verdict item carries its vocabulary like its neighbours.

On the boundaries: they were calibrated against XDS-derived unmerged data,
while production measures the floor from rugnux's own observations, and those
disagree by 0.12x to 16.2x with 7 of 29 verdicts flipping. Re-measured on
production input - by merging each cubic crystal in proper subgroups of its
own Laue class, where the true anisotropy is exactly zero and every deltaB
that comes back is manufactured - the false-positive rate is 24% at 2.0, 10%
at 3.5 and 5% at 5.0. Each sits inside the published band read as its
worst-case figure rather than its average. 3.5 maximises power minus
false-positive rate across the grid; moving to 4.0 costs a real detection and
reduces the false-positive rate by nothing. So the boundaries stay, and the
"strong" band at 5.0 is a 5% statement on real data rather than the 0.5% its
derivation suggested.

The changelog entry for the diagnostic itself was committed with only its
first line, having been extracted by a script that took one line of a
four-line bullet. Restored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Integration: fall back to the fixed radius on a pattern too dense for it ... 4e6eb18d93

48008e144 widens the signal radius on crystals with wide spots. On one
battery crystal - simultaneously the widest-spot and among the highest in
mosaicity - the wider radius left its neighbours' background rings with too
few clean pixels and cost 28.5% of its observations. Pass 1 now measures how
often that happens and, above a bound, hands pass 2 the settings from before
the pre-scan widened them.

The obvious quantity does not work. On the total rate of reflections dropped
for a starved ring, the losing crystal reads 4.08% and the rule's four
biggest winners read 1.89-2.04% - and a crystal at the shipped radius reads
2.37%, above all of them. Re-running the winners at r1 = 4 shows why: they
read 2.20-2.32% there too, and widening moves them down. That floor is module
gaps, the beam stop and the resolution mask, which are properties of the
detector and do not move with the radius.

So the counter separates the two. A ring is neighbour-starved when it would
have kept more than five pixels but for the pixels a neighbouring
reflection's signal region occupies. That is exact rather than estimated: the
reflection mask marks the disk inside r2 and the ring is everything outside
it, so a masked ring pixel always belongs to some other reflection's core.
The separation goes from a factor of 2 to a factor of 13 - over the twelve
crystals the radius moves, the rate is 0.000 five times, 0.001 three times,
then 0.004, 0.235, 0.315 and 4.082 - and the bound is the log-space midpoint
of that one gap, 0.0113, a factor 3.6 clear of the nearest measurement on
either side.

Predicted reflection spacing does not separate them at all: the losing
crystal is 19th of 38, a winner sits at 21.9 px, and the loosest pattern in
the battery starves 1.93% of its rings.

Battery: the space group is identical on all 38 and the merged .hkl is
byte-identical on 37, so it is inert wherever it does not fire. On the one
crystal it fires on, <I/sigma> is up 18.3%, R_meas down 29.4%, observations
up 6.3%, CC1/2 0.944 to 0.974, and its two empty top shells come back as
numbers. Its indexing rate, refined distance, beam centre and cell are
bit-identical between the two arms, so this is the guard and not the two-pass
gate.

The counters are a shared channel through both engines, summed across
workers and logged once per pass; on the GPU it is one atomic add per dropped
reflection. The profile-fit runaway guard reports on the same channel, which
is the first measurement of its trip rate.

This does not recover that crystal fully. With the adaptive radius on, pass 1
reaches a different lattice and pass 2 indexes 21% fewer frames - which
happens before the measurement this guard reads exists, and is unaffected by
it. At matched indexing rate the guard recovers 96% of the baseline's
observations against 90.5% without it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Space-group search: score operators on normalised intensities ... f6bfa54624

Symmetry operators were scored by Pearson correlation on raw merged
intensities. Both members of a symmetry pair sit at the same |s|, so the
resolution fall-off is variance the two arms share exactly, and it inflates
the correlation of true and false operators alike.

The clearest demonstration is the test this commit adds: a synthetic data set
with no symmetry at all - a radial fall-off times an independent per-
reflection factor - is assigned point group 432 by the shipped code, with all
23 rotations confirmed at 0.632 to 0.656. The existing suite passes
identically before and after, because nothing covered this. The new case
fails 64 of its 100 assertions on the old scoring and passes on the new.

On real data the same effect had the gate leaking: on one cubic crystal three
pseudo-symmetric operators scored 0.506 to 0.517, above the 0.5 bound, so
they were confirmed and 432 had to be refused further downstream by the
twin-law and systematic-b guards. Scored on E-squared they read 0.283 to
0.298 and exactly the eleven genuine rotations of 23 are confirmed.

The normalisation has to be over the reflections the correlation actually
pairs. Reusing the existing normalised array is worse than doing nothing: it
is normalised over the set the absence tests use, whose surviving fraction is
itself resolution-dependent, and the coupling to the resolution cut rises
from 0.086 to 0.262 against a raw baseline of 0.086. Normalised over the
paired set it falls to 0.023.

The twin-law H statistic keeps its own vectors on raw intensities. It shares
the pair arrays with the correlation, and normalising in place moves it by up
to 12% against a bound whose window is 5.5% wide. Verified rather than
assumed: two instrumented binaries print the same H to twelve significant
figures while the correlation differs.

min_operator_cc goes 0.5 to 0.30. Normalised correlations run lower, and the
observed window on rugnux's own search merges is 0.298 to 0.351; 0.35 is too
high, because one crystal's weakest genuine operator reads 0.351. The
headroom between a crystal's weakest true operator and its own measured false
-operator floor widens on 14 of 14 crystals, median 0.430 to 0.619, and the
worst operational margin goes from 0.031 to 0.051.

Battery, twice, against a baseline reproducible to zero: the space group is
identical on all 38 crystals, per-shell merging is 0 better and 0 worse
across all 380 shells, and the merged mmCIF is byte-identical on 38 of 38 -
on the 12 crystals whose integration radius now adapts as well as the 25 that
do not. The arm is live rather than inert: all 313 operator correlations move
while every pair count and every H value stays bit-identical, and on one
cubic crystal three operators that raw intensities confirmed are rejected,
with the space group unchanged.

Following Padilla and Yeates (2003) Acta Cryst. D59, 1124-1130 for why a
resolution-normalised statistic is the right one for a symmetry test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Space-group search: record what min_pairs_per_operator is actually worth ... 64957d5a5f

Comment only; the constant stays at 20.

f6bfa5462 moved min_operator_cc to 0.30, which is close enough to the null of
a sparse operator that the pair minimum deserved measuring rather than
assuming. Measured, over 38 crystals and 142 (crystal, pair-count) points,
the null of the normalised operator correlation has mean -0.02 to +0.05 and
standard deviation 1.08/sqrt(n), flat from 20 pairs to 5000. Its tail is the
part worth writing down:

    log10 P(null CC >= 0.30) = -1.03 z + 0.69,  residual sd 0.14 decades

One decade per standard deviation, where normal theory gives about five. So a
false operator at 20 pairs clears 0.30 on 12.4% of draws, and any calibration
of this gate that reasons from 1/sqrt(n-3) with a normal or Fisher-z tail is
optimistic by one to two orders of magnitude.

It does not bite where the decision is made. Over 314 candidate operators on
36 crystals the sparsest has 520 pairs and none is below 500; the weakest
confirmed operator sits 9.1 sigma above its own null and the strongest
rejected one 42.7 sigma above. Nothing passes on noise. The search's own
<I/sigma> cut is what keeps that null flat - reconstructing the same crystals
from an unfiltered merge reproduces a null floor of 0.10 to 0.20, which is
where the "false CC rises at low pair count" impression came from. That
describes a shrinking merge, not a sparse operator.

A minimum of 251 pairs would buy a per-operator false rate of 1e-3. It was
built and rejected on measurement: SearchSpaceGroup runs twice, once on all
observations, which decides, and once on a Lorentz-filtered merge, which only
reports - and no log prints the second arm's pair counts. Genuine operators
fall to 98 pairs there, so a floor of 251 empties a conjugacy class on one
crystal and makes the report say its filtered merge supports triclinic, about
a merge whose own correlations put every one of those operators 9 sigma clear
at 0.91 to 0.97. Any floor above about 150 does that; any below about 103 is
barely better than 20. A count floor rejects on pair count however decisive
the correlation is, which is why the shape is wrong rather than the value.

The comment records the law, the rejected value, and the form that would work
if this is ever revisited - a significance floor rather than a count floor.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Integration: widen the radius for the final pass only ... 7e3ab8150d

48008e144 set the integration radius from measured spot width, and on one
crystal that cost 21% of the indexed frames. The cause is not the indexer.

The geometry pre-pass integrates, post-refines the detector distance and beam
from what it integrated, and discards the intensities. PostRefine takes its
positions, its weights and its reflection set from the integrator, and the
observed position is a raw first moment over the signal disk with the
background still in it. A flat background contributes nothing to the
numerator, because the disk is centred on the prediction, but it adds n*b to
the denominator - so every measured offset is pulled toward its prediction by
I/(I+n*b), and n goes from 50 pixels at r1 = 4 to 113 at r1 = 6. A wider disk
under-corrects the geometry. Measured: 225.978 mm against 225.944, and a
slightly worse held-out residual.

That is the entire difference the pre-pass hands on: 33 microns of distance
and 0.02 pixels of beam. It is enough because that crystal's metric is 0.54
degrees off orthorhombic, so the de-novo Bravais choice sits on a knife edge -
scanning the beam in hundredth-pixel steps flips it back and forth in islands,
and both sides are bit-reproducible. Snapping the cell to 90 degrees costs a
contiguous 66-degree stretch of the sweep, which is where the frames go.

So the pre-pass now keeps the radius the run started with and only the final
pass integrates at the measured width. Pass 1 becomes bit-identical to a
fixed-radius run, so the geometry it hands on is too, by construction rather
than by tuning: on all twelve crystals the rule moves, the post-refined
distance and the pass-2 validation frame count match the fixed-radius values
digit for digit. On the crystal that regressed, the indexing rate goes 0.7900
to 1.0000, observations up 30.1%, multiplicity 5.22 to 6.68, and low-
resolution R_meas 13.1% to 10.7% against XDS's 10.8%.

Against the right reference - the rule off, same lattice, same geometry,
matched indexing rate - the widened radius on that crystal is better in 10 of
10 shells on R_meas and 10 of 10 on CC1/2. It looked worse against the shipped
arm only because that arm merges a fifth fewer frames, and dropping the
hardest part of a sweep flatters a pooled statistic.

Taking the position centroid over a fixed aperture regardless of r1 was tried
and does not work: the intensities, the sigmas and the surviving reflection
set carry the perturbation too, and the geometry moved further from the
baseline, not closer.

Battery: space group identical on all 38 and 35 agreeing with the reference in
every arm, merged intensities byte-identical on the 26 crystals the rule does
not move, over a baseline reproducible to zero. No runtime cost.

The density guard moves with it and now reads the final pass's own count.
Exercising its re-run branch with a lowered bound turned up two reporting
bugs, fixed here.

Not fixed, and worth its own task: that crystal is still one hundredth of a
pixel from losing a fifth of its data, because the indexer computes the
unconstrained cell for exactly this comparison but only scores it once the
constrained one has already failed outright.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Spot width: measure on a growing sample and stop once it settles ... 1bb4cecbd1

The pre-scan measured the spot width on all 60 frames of its sample every
time. The radius that measurement feeds is a three-way choice - R1ForWidth
moves only where 2*r80 crosses 4.5 and 5.5 - and 29 of 37 measurable
crystals sit more than 0.29 px clear of both switches, so most of the
sample is spent confirming a bucket the first few frames already picked.

Measure instead on a growing share of it - every eighth frame, then every
fourth, every second, all of it - and stop at the first tier whose answer
has settled: within 0.40 px of what the smaller sample said AND 0.25 px
clear of both switches. Both conditions are load-bearing. Clearance alone
loses the crystal whose r80 is 0.07 px from a switch, because its small
samples read across it; the step test alone lets a sample settle on a
switch and stop there. Both bounds sit interior to a two-dimensional
region that is right on 38 of 38, and every candidate was re-scored at all
eight phase offsets of the tier ladder.

The tiers are sized in worker-rounds rather than frames: a first tier of
five frames occupies eight workers as long as one of eight does. A {12,4,
2,1} ladder measures 11% fewer frames than {8,4,2,1} and is 11% slower.

Over the rotation battery this reads 1215 of 2280 frames - 21 crystals
stop at 15, 4 at 30, 13 still run all 60 - and the chosen r1 is identical
on 38 of 38, as is the beam-stop mask the reordered loop also touches.
The pass feeds nothing else into the run, so every merged intensity is
unchanged by construction and no battery is required. Cost over the
battery goes 15.6 s to 10.1 s, median 0.18 s to 0.14 s per data set.

Note for anyone optimising this further: the measurement itself is 0.54%
of the pre-scan's cycles. The cost is the decode, preprocessing and spot
finding each frame needs before it - 64% of the pass - because the
beam-stop projection decodes on the GPU and never materialises the frame
on the host. Cutting frames is the only lever short of harvesting the
width from the two-pass run's first GPU pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Spot width: stop doing obviously redundant work in the pixel loops ... 00c27584cb

None of this is worth its speedup - the routine is 0.5-0.8% of the
pre-scan's cycles on two independent profiles, and the whole change is
0.9 ms per data set. It is worth having because the code was doing work
that has no reason to exist.

The isolation grid was a vector<vector<uint32_t>> over ISOLATION_PX
cells: on a 4148x4362 detector that is 23244 std::vector objects
constructed, heap-allocated and destroyed per image for a structure that
is read once. It is now a counting sort - one offset array, one index
array. This was the largest single item and it is not a pixel loop.

The encircled-flux curve added every pixel into every bin at or beyond
its own radius, about 7.5 adds per pixel, which sums the same aperture
R_MAX/2 times over. Each pixel now lands in the one bin its radius falls
in and the curve is the running total. The bins are double where the
running totals were float, so the result is more accurate, not merely
faster.

No square roots remain in the pixel loops. The isolation and beam gates
compare squared distances, and the radial bin is a table lookup over the
197 squared distances the aperture can produce. That is the same bin, not
an approximation: floor(sqrt(floor(y))) == floor(sqrt(y)) for every real
y >= 0, because k*k is an integer, so the table recovers ceil(sqrt(rc2))
exactly once the perfect-square case is separated out. The disks are also
walked as disks rather than as their bounding boxes - constexpr per-row
half-widths - which takes the background pass from 1681 to 1257 pixel
visits per spot.

Verified over the rotation battery: r80 identical to nine significant
figures on 37 of 37 measurable crystals, the 38th unmeasurable in both
arms, and the chosen r1 identical on 38 of 38. The pass feeds nothing
but r1 into the run, so merged output is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Space-group search: record what the E^2 shell count does on a small merge ... d987bd6178

The shell count is order.size() / 100 in integer arithmetic, so a merge of
fewer than two hundred reflections gets a single shell and E^2 becomes I
divided by one global mean.

That is harmless for the thing E^2 was introduced for - the operator
correlation is a Pearson coefficient and is invariant to a common scale,
so it lands on exactly the value raw I would have given. It is not
harmless for the overlap cap, which then rejects reflections by their
intensity against the whole merge rather than against their own
resolution shell, and so cuts the low-resolution end and spares the high.

No behaviour change; a merge that small has already failed for other
reasons. This is here so the next reader does not have to rediscover that
the normalisation quietly switches off at the bottom of the range.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Add a stills validation harness, the analogue of rugnux_vs_xds.py ... 39a0b067a4

rugnux_vs_xds.py gates every rotation change in this project. Nothing
gates a stills change, so until now one could not be validated at all -
which is why the background-ring width below had to be measured before it
could be argued about.

The rotation harness does not transfer. A stills dataset is one file of
100k-500k images with no per-crystal reference, the space group is known
in advance and never in question, and what a change moves is the merge.
So this asks a different question - did this make stills data better or
worse - by running rugnux over the same fixed image subset under two or
more argument sets and tabulating the merge per resolution shell.

Three things it does on purpose. It pins the resolution range per dataset
so both arms share shell edges: without that the automatic cut-off moves
between arms and the columns are not the same shells, which happened on
two of seven datasets. It fixes and prints the image subset, contiguous
rather than strided because a stride turns one sequential read into ten
thousand seeks. And --repeat measures the control floor, which on six of
seven datasets is bit-identical in every column of every shell.

It reads the stable interfaces - the report's KEY= value lines and the
mmCIF _reflns block and _reflns_shell. loop. The one console-log number,
how many predicted reflections lost their background ring, is labelled as
such; it is there because it is the only place the price of a wider ring
is counted. A CrystFEL stream can be scored alongside as an optional
reference, keyed on the global image serial number - Event: //N restarts
at zero in every file and silently multiplies the count.

No dataset list is shipped. A dataset directory name identifies a
sample, and so does a run label or a pump-probe parameter in a filename,
so the table committed here is a template carrying the reasoning and no
paths. A real one belongs in a config kept outside the repository, which
is why --config is required and has no default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Stills: run the background ring to 14 px instead of 12 ... c4f4f7e28f

The reported sigma carries the background ring mean's own error as
(dI/dbkg)^2 * bkg/n_ring, and the stills ring holds only 244 pixels at
r3 = 12 against the rotation default's 408 - so that term is a larger
share of the variance here than anywhere else in the program. r3 = 14
takes the ring to 416.

What normally stops a ring being widened is crowding, since a neighbour
inside the annulus is excluded from it. Stills barely crowd: neighbour
occlusion is 0.000% at both radii on six of the seven datasets scored and
0.721% -> 0.678% on the densest, against most reflections having a
neighbour within 13 px on rotation. Rotation is left alone at 4,6,13,
where the ring already wins by a factor of several.

Measured with the new stills harness over six datasets and 72 shells,
10000 images each, resolution range pinned so both arms share shell
edges, against a control floor that is bit-identical in every column:

  <I/sigma>  +1.23% median over the 56 signal-bearing shells (44/56)
  R_meas     -0.37 points (39/56)
  CC1/2      flat (29/56)
  obs        flat

R_meas is the number to read here. <I/sigma> improving when the ring is
widened is half mechanical - a better background estimate lowers sigma by
construction - whereas R_meas has no sigma in it, so it moves only if the
intensities themselves got better. A prediction from ring pixel counts
alone, made before the run, was 1.6-2.7%; landing at 1.2% with the
profile-fit weighting in the way is the mechanism behaving as expected.

The larger and more consistent effect is the opposite of the risk this
was checked against: reflections discarded for want of a background ring
fall from 2.53-2.88% to 1.87-2.25% on all seven datasets, because a wider
ring reaches past the module gap that starved the narrow one.

Against it: CC1/2 and observation count are flat, and the gain reverses
below <I/sigma> of about 4 - median -0.4% and -0.6% in the two weakest
bands. Roughly a fifth of that is background curvature, which
--background-radial recovers; the rest is not explained.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Anisotropy: carry the sort key with the part, as the merge's ingest does ... 278f8e3d73

ScaledObservations assembles every integrated partial into rocking events
before the tensor is fitted, and sorted them through a pointer. On the
largest sweep that is 29.1 million parts spread over about 2.5 GB of
88-byte records against a 32 MB L3, and the comparator dereferenced one
of those pointers on every one of roughly 660 million comparisons.

Single-threaded, once per run, at the end of a run whose image loop uses
sixteen threads and a GPU - and for a diagnostic that corrects no
intensity and removes no reflection. It was 8% of one run, 12%, 15% and
16% of others, and 19% of the largest.

The fix is the idiom this repository already uses for this exact sort:
RotationScaleMerge's ingest sorts the same partials on the same key
through a narrow record and its header says why. ScaledObservations was
the one place that read the key back through the pointer instead. The
key now travels with the part. Twelve lines, no bit-packing, and a
reserve of the upper bound so the growth chain stops copying a few
hundred megabytes on a large sweep.

Ties are real and had to be handled rather than argued away: the
predictor emits both Ewald intersections and near the blind region both
land on the same frame, so one crystal has 7424 tied adjacent pairs, 3778
of them in reverse arrival order. The comparator keeps the same four
values in the same order, so introsort - whose every branch is decided by
comparator outcomes at positions and never by element size - makes the
same comparisons and the same swaps and leaves the same permutation.
Checked rather than asserted: a hash of the sorted permutation of all
29.1 million parts, a hash of the returned vector's raw bytes, and the
md5 of the whole .cif all match on eight crystals, each first run twice
on the baseline as a determinism control.

The sort is 1.7-2.1x faster and the function 1.5-2.0x, which is 6.7 s of
single-threaded critical path over those eight crystals and 10.8% of
total wall clock over six timed warm and interleaved.

A wider element costs 7% in the grouping pass that follows. A 24-byte
variant that indexes back instead was built and is marginally faster to
sort and slower overall, so the plainest version is also the fastest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Space-group search: read the presence cut on counting significance ... e332aaf28c

present_i_over_sigma is meant to say "this reflection is really there".
The merged I/sigma is not that quantity. The merge carries the error
model's intensity-proportional term, sigma^2 = a*sigma0^2 + (b*I)^2, so
I/sigma saturates - at ISa*sqrt(n) for a reflection observed n times, and
at ISa exactly for one observed once. Above that knee it stops rising
with the intensity.

Measured over the battery's 76 search merges: the median I/sigma of the
top E^2 decile sits at 0.81 of ISa, and on the weakest merge the decile
profile reads 1.58 1.61 1.63 1.64 1.64 1.65 1.65 against an ISa of 1.70 -
flat to three digits, log-log slope against E^2 of 0.008. Reflections an
order of magnitude apart in real intensity read the same number. So one
constant demands between 2.16 and 11.91 in counting significance
depending on the crystal, and the fraction of a merge clearing it runs
from 3.1% to 98.8%.

Convert it once instead. For a reflection observed once
sigma_counting^2 = sigma^2 - (b*I)^2, so I/sigma_counting >= T is exactly
t >= T/sqrt(1+(T/ISa)^2). That cut lies strictly below ISa for every ISa,
so it is always reachable by the reflection the ceiling binds hardest,
and it is within 1% of T on any merge with ISa >= 21, so a healthy merge
is left where it was. Multiplicity is taken as 1 deliberately rather than
estimated, for the same reason.

The quantile fallback goes with it. Its comment claimed to be inert on
the battery; it fires on 4 of 76 arms, and on one of them it relaxes the
cut BELOW the nominal value because that merge is bimodal. A cap on a
statistic is not a substitute for measuring the right statistic.

Two comments in the tree still described a merged sigma floored at b|I|,
removed in 72efb75a8 before rc-161. Both corrected; that stale claim is
what made this look like a different bug.

Battery, both arms, against a control floor of exactly zero: space groups
35/38 before and after, the same three crystals differing from the
reference, all 380 shells identical and all 38 merged mmCIFs
byte-identical. It is live rather than dead code - the pass_cc population
moves on 45 of 47 arms and 212 of 314 operator correlations move, with 0
of 314 verdicts changed. The weakest crystal's single 2-fold goes from CC
0.489 on 1131 pairs to 0.597 on 4822, widening its margin over the
threshold by 57%.

Two things this does not fix, both quantified in the work behind it. The
conversion is monotone, so it corrects the threshold but cannot restore
the ranking the (b*I)^2 term already destroyed - that needs a
counting-only merged sigma and touches the CUDA merge kernel. And one
crystal's gate still admits 98.8% of its merge with a slope of 0.037,
discriminating nothing; the instrument for that is E^2, which needs
min_operator_cc re-derived first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Changelog: say what the anisotropy sort actually cost ... 62f0f6472d

It was about eleven seconds on the largest sweep, not minutes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Anisotropy: assemble only the reflections the floor actually reads ... 0e6ece3c38

BuildObservationCells strides whole ASU clusters down to MAX_CLUSTERS =
20000 and keeps 1.4% of what it is handed - so the assembly before it
sorted and summed every integrated partial in the sweep to throw 98.6% of
the result away on the next line. The comment there says the striding
"costs nothing and bounds the work"; it bounded it after the expensive
part.

Sample first instead: keep a partial iff a splitmix64 hash of its ASU key
falls in the low one part in 2^n. Whole ASU clusters are mandatory rather
than convenient - the cluster-robust sandwich sums one score per cluster,
and the forbidden-direction signal IS the contrast between mates, so a
raw-hkl sample would collapse that onto the independent covariance and
inflate the floor. The hash rather than the key matters too: HKLKey::pack
puts h in the low bits, so masking the key directly would select lattice
planes instead of a sample.

The rate comes from the partial count, not from merged.size(): the merged
list is resolution-cut, and taking the rate from it made the first
version never fire at all on the crystal it was written for. One part in
2^n of the partials is one unique reflection in 2^n, so the count already
computed for the reserve is the right one.

DELTA_B is unchanged by construction, not by luck. AnalyzeAnisotropy
touches the unmerged observations in exactly two places - an empty test
and MeasureSystematicFloor - so the tensor, the shape verdict and the
directional limits come from the merged list alone. Over the battery:
0 verdict flips of 38; DELTA_B, DELTA_B_LINEAR, SHAPE, FREE_DIRECTIONS
and the D_MIN keys differ on none of the 38, byte-identical rather than
within noise; every reported key identical on 26; the mmCIF diff is
exactly _reflns.jfjoch_aniso_floor and _significance; no caution appears
or disappears anywhere.

The spread those two keys move within was measured rather than assumed,
and re-running cannot measure it because the code is deterministic. The
shipped code already samples - by striding cluster order - so moving the
phase of its own stride gives the sampling spread it has always carried.
Against that control the change moves the floor less: ratio range
0.803-1.481 against 0.497-1.343, geomean 1.031 against 0.941. The gate
therefore did not get easier to trip - the floor's geomean moves +3.1%
here against -5.9% for the phase control, the crystal nearest the gate is
not sampled at all and is bit-identical, and the two sampled crystals
nearest the gate both move away from it. N_OBSERVATIONS goes up, not
down, because a smaller pool lands the downstream integer stride slightly
above the cluster cap.

The ASU reduction is 14-45 ns a call in isolation, which nearly sank the
design; in place it is free, because it overlaps the memory stalls of the
Reflection stream it walks. The 24 crystals below the sampling cap run
the shipped algorithm plus the reduction and cost +1.79 s in total,
median +0.04 s.

The block that introduced this measures 8.07 s before and 1.97 s after,
which is 5.74 s off the largest crystal (-12.3%) and -11.9 s of
anisotropy time over the battery.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Unmerged export: carry the sort key with the part here too ... abb61e2708

SumRockingEvents assembles rocking events for the unmerged MTZ through
the same indirect comparator the anisotropy diagnostic had, on a larger
array - it has neither the scaled-image test nor the usable() filter in
front of it. It is latent, since --export-unmerged is off by default, but
it is the same defect and it is a mechanical fix.

Same treatment: the key travels with the part, plus an exact reserve.
p_unmerged.mtz, p_unmerged_partials.mtz and p.hkl are md5-identical
across eight runs on two crystals. With the export on it is worth about
0.9 s on a seventeen-million-partial run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P

Package rugnux as its own rpm and deb ... a67c52d8fa

rugnux was a component of the jfjoch-viewer package, so getting the
offline analysis CLI meant installing the desktop viewer, Qt and the XDS
plugin with it. It links none of the hardware libraries and talks to no
service, so a machine that only processes data should be able to install
it alone.

It is packaged as "rugnux", not "jfjoch-rugnux": the binary, the release
tarball and docs/NAMING.md all call it that, and it is what a user types.
Overriding the component's package name is the only way to drop the
CPACK_PACKAGE_NAME prefix.

/usr/bin/rugnux moves between two packages that both continue to exist,
so the declaration has to be a bound on the OLD viewer rather than an
Obsoletes - obsoleting jfjoch-viewer would uninstall the viewer. rpm gets
Conflicts, deb gets Breaks+Replaces so the two stay co-installable from
here on. rpm rewrites "-" to "_" in a package version (an installed
release reads 1.0.0_rc.160), so the bound is built with the same
substitution; spelled 1.0.0-rc.164 the comparison would silently never
match, which is the failure mode this kind of guard usually has.

Verified by building rpms from a project mirroring these settings: the
package comes out named rugnux at 1.0.0_rc.164 carrying /usr/bin/rugnux
and its notices, the jfjoch package no longer carries rugnux, and
"rpm -i --test" against the rc.160 viewer installed on this workstation
reports the intended dependency conflict rather than a raw file
collision - which is what lets dnf resolve it by upgrading the viewer in
the same transaction. The deb side is checked against CPackDeb.cmake,
which supports NAME, BREAKS and REPLACES per component, but no .deb was
built: this host is RHEL.

CI needs no change; the upload step globs *.rpm and *.deb.

Docs: bring the release notes and the rugnux page up to rc.164 ... dd67cffde6

The changelog section had grown to 22 entries written from the commits
rather than for a user. Collapsed to 13, each saying what is gained
before what changed, with related work merged - the six anisotropy
entries into one, five space-group entries into two, five
integration-radius entries into one - and four things added that had no
entry at all: the packaging split, the statically linked cuFFT, the Ceres
back-end drop and REPORT_VERSION reaching 3.

CPU_DATA_ANALYSIS gains 9.5, the adaptive integration radius: the r80
measurement, the clamp, the constant-ring-area r3, the convergence test,
why it applies to the final pass only and the density fallback. Three
passages there had gone stale against the code: 13.1 still described the
I/sigma quantile fallback that b90fcfb98 deleted, and 13.5 said the gate
tests delta_B when it tests delta_B_linear and that the high-symmetry
caution fires wherever the Laue class leaves one free direction, which
0da593b32 narrowed.

RUGNUX gains an Installation section - there was no page anywhere saying
where rugnux comes from - a synopsis, a worked first run naming the files
that actually appear and the report keys to grep, and sections on the
unmerged export and on diffraction anisotropy. Its report section list
was one section behind: anisotropy is 9, warnings is 10.

The packaging change reached further than the changelog implied, so the
pages describing what a release ships were corrected too: the viewer
tarball no longer carries rugnux or the two developer utilities, the
Linux archives link cuFFT statically rather than shipping it beside the
binary under an $ORIGIN rpath, the aarch64 archive has a higher glibc
floor than the RHEL 8 ones and is untested on hardware, the archives
unpack with no top-level directory, and the notices live per package
under share/doc/jfjoch_<component>. The RHEL 9.5 vm_flags workaround in
FPGA_PCIE_DRIVER is obsolete and now says so.

Broker: report a failed asynchronous start instead of a timeout ... a6a715703f

An acquisition started with async_start returns from /start before the
measurement thread has run, so a failure there had no caller to raise it
to. MeasurementThread stored it in start_exception, but only the
synchronous branch of Start() ever read it.

The ordinary-failure path - a writer refusing to overwrite an existing
file, say - sets the state to Idle rather than Error, and Idle is what
wait_until_running_post maps to 504 "timeout, need to restart". So the
one case the async workflow exists to report came back as a timeout with
an empty body, and the writer's message was dropped. /wait_till_done was
worse: Idle is its success case, so it answered 200 for a run that never
started. Only a critical detector fault, which leaves the state at Error,
was reported at all.

The wait functions now rethrow start_exception, so both endpoints produce
the same 500 and the same message as a synchronous start.
rethrow_exception does not consume the exception_ptr, so repeated calls
all report the same failure. It is cleared by every entry point that
begins new work - Start (before ImportDatasetSettings, which can throw),
Initialize, Pedestal, LoadDetectorSettings, SetDarkMaskSettings - so a
pending failure is never attributed to the operation after it. The
synchronous branch no longer clears it, so a wait call made after a
failed /start reports the failure rather than an apparent timeout.

wait_until_running_post and wait_till_done_post drop their timeout == 0
special case, which called GetStatus() directly and so bypassed the
rethrow: ?timeout=0 returned 200 from /wait_till_done where ?timeout=1
returned the error. wait_for evaluates the predicate before expiring, so
the state reported is unchanged.

JFJochReceiverService::Start clears the receiver status when the start
fails. JFJochReceiver's constructor sets progress to 0 and a zeroed
status before the throw, and nothing cleared it, so /status reported
Idle with progress 0 - a stalled acquisition, to the frontend - and
/statistics reported an all-zero run that never happened, until the next
start overwrote it. Its catch widens to std::exception, so a
non-JFJochException gets the cleanup and the logging it skipped before.

Nothing else is left dirty by a refused start: the receiver keeps its
null receiver and Idle state, PrepareAction only resets counters (the
FPGA is armed in StartAction, in a thread launched after the start
message goes out), and SendStartMessage precedes every std::async in the
receiver constructor. The test starts four more times afterwards, one of
them a complete acquisition with no re-initialisation in between.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uwv9ScHtDH6g8tYgfSuApo

Broker: stop reporting a failed calibration as a successful one ... c29dd67b5f

Every calibration step signals failure by calling SetState and returning
normally - none of them throws, so none reached the catch in
CalibrateDetector. The unconditional SetState(Idle, "Calibration sequence
done", Success) after the try block then overwrote all of them.

/cancel during a JUNGFRAU pedestal therefore left the broker Idle and
apparently ready to measure while holding a truncated G0 and
default-constructed zeros for G1/G2, and every subsequent run was
silently mis-converted with nothing in /status to show it. The genuine
failures - "Pedestal not collected properly", "Mask not collected
properly" - were hidden the same way.

The steps now return whether they succeeded, and the sequence reports
success only if they all did. A cancellation or a failure leaves the
state Inactive with Error severity rather than Idle or Error: the
calibration is undefined, so the detector has to be initialized again,
which is what Inactive means everywhere else in the machine. The
exception path joins them, since a throw mid-sequence leaves the
calibration no better defined. Cancelled pedestals were already Inactive
but carried Warning severity, which reads as an advisory.

CalibrateJUNGFRAU now abandons the sequence at the first failure instead
of collecting G1 and G2 on top of a G0 that was never measured - the
cancel path already behaved that way - and ConfigureDetector is skipped
when there is no calibration to operate with, a cancelled sequence having
left the detector mid-abort.

Both error paths that end an Initialize now notify the condition
variable. The state has left Busy, but without the notification a client
in /wait_until_running slept out its whole timeout - up to an hour, if it
asked for one - before noticing a failure that had already happened.

Separately, dataset_settings.space_group_number allowed 1..194 in the
OpenAPI schema while the broker accepts 1..230, so every generated
client's validate() rejected all 36 cubic space groups before the request
left. The regenerated clients follow in the version bump.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uwv9ScHtDH6g8tYgfSuApo

Changelog: the calibration outcome, the initialize wait and cubic space groups ... 03dd28f40b

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uwv9ScHtDH6g8tYgfSuApo

Version 1.0.0-rc.164 ... c8b05295c6

Runs update_version.sh, which regenerates the C++ server model, the
TypeScript frontend client and the python client from the OpenAPI spec.
That also propagates the space_group_number bound raised to 230 in the
previous commit, so the generated clients stop rejecting cubic space
groups before the request is sent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uwv9ScHtDH6g8tYgfSuApo

rugnux: let --model settle the indexing of the reflections it writes ... f7cb701f43

ValidateAgainstModel was called after WriteReflections, so the two
relabelings it makes - the change of hand where the data were merged in
the model's enantiomorph, and the alternative indexing it picks by R-free
where no reference MTZ had already fixed it - reached the R-factors, the
maps and _maps.mtz and nothing else. The .mtz/.cif/.hkl beside them kept
whichever indexing the merge happened to land in, so a file whose maps
came from a model could not be refined against that model without being
reindexed first.

Validation now runs before the reflection files are written and returns
the operators it applied. AdoptModelFrame puts the merged reflections
through them with a proper ASU reduction - ReindexReflections only
relabels, which is all its scoring callers need - and the Bijvoet halves
follow the Friedel sign of that reduction. The integrated observations
are relabelled too, so --export-unmerged describes the same indexing as
the merged files rather than a second one.

change_of_hand_op is the inversion, so adopting a model's hand exchanges
I(+) and I(-). On a tetragonal crystal whose enantiomorphic pair share
their whole absence pattern, the search picks one of the two arbitrarily,
and the exported anomalous differences were the wrong way round whenever
it picked the other member from the model's. The written space group
follows the hand, and so does the report that describes those files.

The _process.h5 is deliberately left alone: its per-image reflections
went to disk as they were integrated, and recording a group that did not
match them would mis-merge on a later --mode scale. Nothing is lost -
an enantiomorphic pair has the same Laue class and the same absences, so
it merges identically either way.

Serial stills could not use a model for this at all: each crystal is
indexed in its own hand, so by the time there is a merge to fit a model
to, the hands have already been averaged together, and reindexing the
merged result as a whole cannot separate them again. ModelReferenceIntensities
computes |F_model|^2 from the structure - Fcalc plus a flat solvent term
at the standard constants, which are not fitted because there are no
observations yet - and hands it to the per-image resolver that a
reference MTZ already drove. It needs the cell and the group up front,
which serial indexing wants anyway. Measured on a serial dataset in a
merohedral trigonal group, at a fixed resolution limit so the shells
match: CC1/2 better in 8 of 10 shells (79.7 to 82.5 overall), R_meas
better in 9 of 10 (111.1 to 104.7), ISa 1.11 to 1.20, and R-free against
a structure the merge never saw 0.393 to 0.375.

The ambiguity probe now logs the runner-up R-free as well as the winner.
On the same data the margin is 0.016 when the hands are mixed, where the
two global indexings are tied because both are already inside every
reflection, and 0.193 once they are not - the difference between a
decision and a coin toss, which the old single number hid. The warning
about an unresolved ambiguity no longer fires when a model will resolve
it, and names the flags that would; and the summary line no longer offers
the adopted group as its own alternative, which it did once --model had
moved the run onto the member the search had listed as the alternative.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vi1gV6Z45aZL5wLwe85Ksn

Docs: a quick start at the top of the rugnux page, and the indexing ambiguity written down ... e576184e2f

The rugnux page opened with installation and reached its quick start
around line 500 of 840, which is not where someone who wants to process a
dataset looks. It now opens with four commands - the defaults, a
reference MTZ, a model, and a pinned cell and space group, which is what
people actually ask for - followed by the five files a run leaves and a
few pointers. The old quick start keeps the detailed walkthrough under
"Running rugnux", moved up to sit after the orientation sections.

The indexing ambiguity had no prose anywhere, and neither did -z, which
had one row of a table for four different jobs. A new section says what a
reference MTZ does, what the ambiguity is, that it costs a rotation run
an arbitrary frame and a serial run its CC1/2 outright, how to resolve it
in each of the four cases, and that it is neither the enantiomorph nor
twinning - the two things it is most often taken for.

The three long reference pages carry a table of contents, from the MyST
contents directive rather than a hand-written list, so it cannot go
stale. The theme's sidebar already lists each page's H2 headings, so this
adds the level below that and an in-page map; splitting the pages was the
alternative and would have broken every anchor other documents link to.

Two things the code says that the docs did not. --mode scale does not
accept a reference MTZ on rotation data - the rotation scaler refuses it
and the run stops - where the page offered it as one of the things to
re-merge with. And CPU_DATA_ANALYSIS 10.9 still said stills use the
reference as their per-image scale target, which stopped being true when
stills were changed to scale against their own merge; the reference is
their per-image ambiguity test and nothing else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vi1gV6Z45aZL5wLwe85Ksn

rugnux: report nine resolution shells, as XDS does ... daac3c2da1

The binning rule was already XDS's: equal steps in 1/d^2 between the
lowest- and the highest-resolution reflection the merge kept, anchored on
the data rather than on the nominal low-resolution cut. Only the count
differed - ten shells against XDS's nine - so at the same resolution
limits neither the boundaries nor the reflection populations matched, and
a per-shell number could not be read across. Forced to nine on a
tetragonal rotation dataset, rugnux now gives 4.27 3.03 2.47 2.14 1.92
1.75 1.62 1.52 1.43 against CORRECT.LP's 4.26 3.02 2.47 2.14 1.92 1.75
1.62 1.52 1.43; the hundredth of an angstrom in the first two shells is
each program anchoring on its own lowest-resolution reflection, the two
sets of survivors not being identical.

This matters most to the comparisons that are made shell by shell.
rugnux_vs_xds.py hands XDS's resolution range to rugnux but not its shell
count, so its low-resolution R_meas column compared a shell running to
4.49 A against one running to 4.26 A - 814 unique reflections against
943. The outer shell was nearly immune, its inner edge being
d_min*sqrt(n/(n-1)), which is 1.51 at ten shells and 1.52 at nine.

--resolution-shells still sets any count. rugnux_stills_ab.py pins the
count for both its arms and now pins nine, so its tables read the same
way as everything else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vi1gV6Z45aZL5wLwe85Ksn

Docs: stop the rugnux examples implying that parallelism has to be asked for ... 5dafb9d21c

Every example command carried -N 32, which reads as though a run is
single-threaded until told otherwise. It is not: the thread count follows
the machine, and -N is there to limit it, or to lift the per-image loop's
default ceiling of 16 workers per GPU. The examples drop the flag and the
quick start says what -N is actually for.

Two things the quick start left to be discovered elsewhere: -S takes a
Hermann-Mauguin symbol or a space-group number, either spelling; and
--model, where the model is isomorphous, pins the enantiomorph - which no
measurement can decide - so the merged reflections come out in the
model's hand and its space group, and on anomalous data that is what puts
I(+) and I(-) the right way round.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vi1gV6Z45aZL5wLwe85Ksn

leonarski_f merged commit 749db470ca into main 2026-08-26 22:47:00 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) mx-gitea perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#74