Docs: bring the release notes and the rugnux page up to rc.164

The changelog section had grown to 22 entries written from the commits
rather than for a user. Collapsed to 13, each saying what is gained
before what changed, with related work merged - the six anisotropy
entries into one, five space-group entries into two, five
integration-radius entries into one - and four things added that had no
entry at all: the packaging split, the statically linked cuFFT, the Ceres
back-end drop and REPORT_VERSION reaching 3.

CPU_DATA_ANALYSIS gains 9.5, the adaptive integration radius: the r80
measurement, the clamp, the constant-ring-area r3, the convergence test,
why it applies to the final pass only and the density fallback. Three
passages there had gone stale against the code: 13.1 still described the
I/sigma quantile fallback that b90fcfb98 deleted, and 13.5 said the gate
tests delta_B when it tests delta_B_linear and that the high-symmetry
caution fires wherever the Laue class leaves one free direction, which
0da593b32 narrowed.

RUGNUX gains an Installation section - there was no page anywhere saying
where rugnux comes from - a synopsis, a worked first run naming the files
that actually appear and the report keys to grep, and sections on the
unmerged export and on diffraction anisotropy. Its report section list
was one section behind: anisotropy is 9, warnings is 10.

The packaging change reached further than the changelog implied, so the
pages describing what a release ships were corrected too: the viewer
tarball no longer carries rugnux or the two developer utilities, the
Linux archives link cuFFT statically rather than shipping it beside the
binary under an $ORIGIN rpath, the aarch64 archive has a higher glibc
floor than the RHEL 8 ones and is untested on hardware, the archives
unpack with no top-level directory, and the notices live per package
under share/doc/jfjoch_<component>. The RHEL 9.5 vm_flags workaround in
FPGA_PCIE_DRIVER is obsolete and now says so.
This commit is contained in:
2026-08-26 15:16:02 +02:00
parent a67c52d8fa
commit dd67cffde6
10 changed files with 448 additions and 81 deletions
+12 -23
View File
@@ -3,31 +3,20 @@
### 1.0.0-rc.164
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction.
* rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are.
* rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log.
* rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4.
* rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning.
* rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities.
* rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present.
* rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file.
* rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware.
* Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere.
* Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged.
* The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
* The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel.
* rugnux: where the space-group search reports alternatives that are not on the same lattice as the group it chose (`C2 or P21 or P2`), it now names the ones the reported unit cell does not describe and says how their cell differs from it.
* rugnux: a lattice centring the data could not test - the crystal was integrated on the primitive sub-cell, so the reflections the centring extinguishes were never measured - is marked `UNTESTED` in the space-group candidate table and, where it is adopted, is warned about as coming from the lattice metric rather than from the intensities.
* rugnux: the space-group search prints the twin-law disagreement H for every operator it tested and the H ratio of the point group it adopted, on every run, instead of only when that ratio refuses a promotion.
* rugnux: `--mode scale` works on a `_process.h5` whose space group came from re-seating the lattice; a file written before this stops with a message naming the two cells instead of failing inside the merge.
* rugnux: de-novo space-group determination scores each symmetry operator on resolution-normalised intensities (E²) rather than on raw merged intensities, so the operator threshold means the same thing on every crystal and no longer moves with the search resolution cut.
* rugnux: the widened integration radius applies to the final integration pass only - the two-pass geometry pre-pass keeps the radius the run started with, so the post-refined detector geometry, and the lattice the second pass indexes on, are the ones the fixed radius gives.
* rugnux: on a pattern too dense for the widened radius - where neighbouring reflections leave more than 1.1% of the reflections without a background ring - the final pass is re-integrated at the fixed 4 px radius, and says so.
* rugnux: on rotation data the integration signal radius is set from how wide the crystal's own spots are, measured in the pre-scan, instead of the fixed 4 px; `--adaptive-integration-radius=off` restores the fixed radius, and an explicit `--integration-radius` still overrides both.
* rugnux: the spot-width pre-scan stops once the integration radius it is measuring has settled, instead of always working through the whole sample; the radius it chooses is unchanged.
* rugnux: on stills the integration background ring runs to 14 px instead of 12, so fewer reflections are discarded for want of a background ring and per-shell R_meas improves over most of the signal-bearing range; the gain reverses in shells below <I/sigma> of about 4.
* rugnux: the anisotropy diagnostic assembles only the unique reflections its systematic-error floor reads, instead of sorting every integrated partial, so a large rotation run ends several seconds sooner. `ANISOTROPY_DELTA_B`, the shape verdict and the directional diffraction limits are unchanged; `ANISOTROPY_FLOOR` and `ANISOTROPY_SIGNIFICANCE` are measured on a different sample of reflections and move within the spread that sample already carried.
* rugnux: the space-group search decides whether a reflection is genuinely present from its counting significance rather than from the merged I/sigma, which saturates at the merge's own ISa; the cut now means the same thing on a weak crystal as on a strong one, and the quantile fallback that used to paper over it is gone.
* rugnux: a directional diffraction limit that is the edge of the measured data rather than the crystal's own limit is marked as such - with a `<` in the report and in `ANISOTROPY_D_MIN_CENSORED`, and in the mmCIF - so `ANISOTROPY_D_MIN_SPREAD` is not read as a measurement when it is a lower bound.
* rugnux: the anisotropy verdict line names which of `ANISOTROPY_DELTA_B` and `ANISOTROPY_DELTA_B_LINEAR` it is quoting, says which of the two to act on, and says why the second can be the larger.
* rugnux: the anisotropy caution about a too-high symmetry assignment now fires only where that is actually indicated - the symmetry-forbidden tensor directions far above their own counting noise together with a gate that established nothing - instead of on every tetragonal, trigonal and hexagonal data set.
* rugnux: the anisotropy section reports `ANISOTROPY_N_OBSERVATIONS`, the unmerged observations the systematic-error floor was measured on, and `ANISOTROPY_FORBIDDEN_Z`, that floor against its own counting noise.
* rugnux: the mmCIF item `_reflns.jfjoch_aniso_verdict` carries its vocabulary as a comment, as the other `jfjoch_aniso_` items already did.
* rugnux reports diffraction anisotropy: the anisotropic deltaB and the diffraction limit along each
principal direction, with a NOT DETECTED / DETECTED / CANNOT DETERMINE verdict against the data
set's own systematic error, in `<prefix>_report.txt` section 9 and in the mmCIF. No intensity is
corrected and no reflection is removed.
* rugnux: `--mode scale` reports the Wilson B-factor estimate instead of `WILSON_B= nan`.
* rugnux: `--export-unmerged` writes the integrated observations as `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike; each rotation reflection's partials are summed into one full, and `--export-unmerged-partials` writes one row per image instead. Lattice-centring absences are not written; screw and glide absences are.
* The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged.
### 1.0.0-rc.163
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
+41 -5
View File
@@ -646,6 +646,10 @@ For each predicted reflection at $(x_p,y_p)$, define three radii:
- $r_2$: inner background radius,
- $r_3$: outer background radius.
The defaults are $4,6,13$ px for rotation data and $6,8,14$ px for stills, which have a sparser
pattern and can afford the wider ring. `--integration-radius` sets them by hand; on rotation data
$r_1$ is otherwise measured from the crystal's own spots (§9.5).
Pixels are classified by their squared distance $r^2=(x-x_p)^2+(y-y_p)^2$:
- **signal region:** $r^2 < r_1^2$,
@@ -653,7 +657,7 @@ Pixels are classified by their squared distance $r^2=(x-x_p)^2+(y-y_p)^2$:
Invalid pixels (masked/bad/saturated) are excluded from both sums. In addition, pixels lying inside the signal disk ($r<r_2$) of any *other* predicted reflection are removed from this reflection's background annulus, so a neighbouring spot cannot leak into the background estimate. (Both the annulus and that exclusion become ellipses when the option below is used; with it off, which is the default, they are the circles just described.)
**Radially elongated background ring (opt-in, `--integration-stencil <k>`, default 0).** The three radii above are fixed pixel counts, identical for every reflection at every resolution. A reflection is not round, though: a finite bandwidth streaks it radially by $\sigma_\mathrm{bw}=\text{bandwidth}\cdot R_\mathrm{px}$. On a radially smeared spot the fixed $6\ldots13$ px ring therefore sits only $\approx1.3$$2.2$ radial $\sigma$ from the centre — on the reflection's own tails, which it then measures as background.
**Radially elongated background ring (opt-in, `--integration-stencil <k>`, default 0).** The three radii above are one triple for the whole run, identical for every reflection at every resolution. A reflection is not round, though: a finite bandwidth streaks it radially by $\sigma_\mathrm{bw}=\text{bandwidth}\cdot R_\mathrm{px}$. On a radially smeared spot the fixed $6\ldots13$ px ring therefore sits only $\approx1.3$$2.2$ radial $\sigma$ from the centre — on the reflection's own tails, which it then measures as background.
With $k>0$ the **background ring becomes an ellipse**, elongated along the beam→reflection direction by $k\sigma_\mathrm{bw}$. The **radial** semi-axes become $r_2+k\sigma_\mathrm{bw}$ and $r_3+k\sigma_\mathrm{bw}$; the **tangential** half-widths stay $r_2$ and $r_3$; and the growth is capped at $2r_3$, which bounds what a mis-declared bandwidth can do to the bounding box. Pixels are then classified as
@@ -729,6 +733,32 @@ For integrated reflections, polarization correction can be applied as a multipli
---
### 9.5 Choosing the signal radius from the crystal's own spots (rotation)
The three radii are one triple for the whole run, but on rotation data they are no longer a fixed constant: $r_1$ is measured from how wide *this* crystal's spots actually are (`--adaptive-integration-radius`, on by default for rotation, off for stills, ignored when `--integration-radius` is given).
**Why $r_1$ matters even though it does not set the intensity.** In the default `gaussian` mode the intensity is a profile-fit amplitude over the grid $\lceil r_2\rceil$ (§9.3), so $r_1$ is not the integration domain. It *is* the aperture the profile **width** is learned over, and a second moment taken over a disk of radius $a$ saturates at $a^2/4$. At $r_1 = 4$ the learned $\sigma$ can therefore never exceed 2 px, and a crystal whose spots are broader than that is fitted with a profile the model cannot represent.
**The measurement is independent of the integrator.** It is made in the pre-scan, on the frames the beam-stop projection already reads, so it costs no extra frame reads and there is no feedback loop. On the spots the spot finder has already found, a spot is used only if it is clear of the detector edge and of the direct beam, has no neighbouring spot within 28 px, is one of the 40 strongest in its resolution band, sits on a fully readable disk, and reaches a signal-to-noise of 15 with its centroid within 2 px of the found position. For each surviving spot the background-subtracted **encircled-flux curve** is accumulated in 1-px annuli out to a fixed 14 px aperture and normalised at 8 px — an aperture that owes nothing to $r_1$, $r_2$ or $r_3$.
**Pooling.** Spots are stratified into five resolution bands (23, 34.5, 4.57, 712, 1230 Å), because a weak crystal's strongest spots sit at high angle and a strong one's at low angle. Each band with enough members contributes the radius at which its **median** curve reaches 0.80 of its normalised flux — $r_{80}$ — at the band's median $d$. Those points are fitted by weighted least squares against $1/d$ (the mosaic contribution to the detector footprint grows as $1/d$) and evaluated at a common 5 Å, then clamped to the range the bands actually measured so the fit never extrapolates.
**The radius.**
$$r_1 = \mathrm{clamp}\!\left(\mathrm{round}(2\,r_{80}),\ 4,\ 6\right),\qquad r_2 = r_1 + 2,\qquad r_3 = \sqrt{r_2^2 + 133}$$
The factor 2 is not fitted: for a Gaussian $r_{80} = 1.794\,\sigma$, so $r_1 = 2r_{80} = 3.59\,\sigma$, where the truncated second moment recovers 0.990 of $\sigma^2$. The expression for $r_3$ holds the **background-ring area constant** at its value for the shipped $4,6,13$ ($13^2 - 6^2 = 133$) — a ring that shrank with the disk is what makes a bare `--integration-radius` worse than the default it replaces. The floor of 4 is that shipped default; the ceiling of 6 is pattern density, since $r_2$ also drives the neighbour-ownership radius and the ring's inner edge. At $r_1 = 4$ the triple is bit-for-bit the shipped default, so a crystal with ordinary spots is left exactly where it was.
**The sample grows until the answer settles.** The frames are measured in tiers of stride 8, 4, 2, 1, each tier's sample strictly containing the previous one, and the pooling is redone after each. Measuring stops when the new $r_{80}$ is within 0.40 px of what the smaller sample said **and** is at least 0.25 px clear of both radii at which the rounding in $r_1$ changes answer. Both conditions are load-bearing: clearance alone lets a small sample settle across a switch, and the step test alone lets it settle *on* one. Every frame of the sample is still read — the beam-stop mask and the beam centre are unchanged; what the tiers save is the decompression, preprocessing and spot finding the width measurement adds on top of the read.
**It applies to the final pass only.** A rotation run integrates twice (§7.5), and the widened radius is handed to the canonical second pass, not to the geometry pre-pass. The reason is that post-refinement takes its observed positions from the integrator, and an observed position is a first moment over the signal disk with the background still in it: a flat background adds nothing to the numerator but adds $n\,b$ to the denominator, so every measured offset is pulled toward its prediction by $I/(I + n b)$, and $n$ nearly doubles between $r_1 = 4$ and $r_1 = 6$. A wider disk therefore *under*-corrects the geometry — enough, on a crystal whose metric is half a degree off orthorhombic, to flip the second pass's de-novo Bravais choice.
**The density guard.** Widening $r_1$ pushes $r_2$, the ring's inner edge, into the neighbours; a reflection whose ring is left with five or fewer clean pixels has no background and is dropped whole. The integrator counts these, and separates the ones lost to **neighbouring reflections** from the ones lost to the detector itself (module gaps, the beam stop, the resolution mask) — a floor that reaches a couple of percent on some geometries and does not move with $r_1$. Where the neighbour-driven loss exceeds 1.13 % of the predicted reflections, the pattern is too dense for the widened radius and the final pass is integrated again at the fixed $4,6,13$, reported as pass 3 of 3 with the reason in `PASS_DECISION`.
Every integration pass, adaptive or not, now logs the radii it used together with the fraction of predicted reflections that lost their background ring, split into the neighbour and detector parts, and the profile-fit fallback rate.
---
## 10. Scaling and merging
After per-image integration, Jungfraujoch scales observations and merges them into unique reflections. The design is intentionally compatible with XDS/XSCALE concepts, and handles both still and rotation data.
@@ -943,7 +973,13 @@ When no space group is supplied, a POINTLESS-like search scores Laue-group symme
The correlation is on **resolution-normalised** intensity $E^2 = I/\langle I\rangle(\text{shell})$, normalised over exactly the reflections the correlation pairs. Both members of a symmetry pair lie at the same $|s|$, so on raw $I$ the resolution fall-off is variance shared perfectly between the two arms and appears as a positive correlation for *any* pairing at all: a shell-matched random pairing — the exact null for a metrically-allowed false operator — scores a median 0.31 across the rotation battery, and on one crystal 0.53 — above the bound the correlation is tested against. That floor varies more from crystal to crystal (spread 0.46) than the whole true/false gap is wide (0.38), so an absolute bound on the raw statistic is a different test on every crystal; and it moves with the search resolution cut, which is what made that cut a symmetry-deciding parameter. Normalised, the floor has a median of 0.015, never exceeds 0.06, and barely moves with the cut.
The operator correlations are taken on reflections above an $I/\sigma$ cut, and that cut is **capped at the merge's own $I/\sigma$ quantile** rather than applied as a fixed number. On a search merge whose ISa is below 3, a fixed cut of 3 selects nothing at all, leaving every operator correlation undefined and collapsing the point group to 1. The cap keeps at least the strongest quarter and is inert — the cut stays exactly 3.0 — on a healthy merge.
Both the correlation stage and the absence tests need to know whether a reflection is **genuinely present**, and that question is asked of its *counting* significance, not of the merged $I/\sigma$. A merged $\sigma$ carries the error model's intensity-proportional term, $\sigma^2 = a\,\sigma_0^2 + (b\,I)^2$, so merged $I/\sigma$ saturates — at $\mathrm{ISa}\sqrt{n}$ for a reflection observed $n$ times, and at $\mathrm{ISa}$ exactly for one observed once. Above that knee it stops rising with the intensity: on the weakest search merge of the rotation battery the $I/\sigma$ of every decile of $E^2$ reads 1.581.65 against an $\mathrm{ISa}$ of 1.70, so reflections an order of magnitude apart in real intensity report the same number. A single constant applied there demands anywhere between 2.2 and 11.9 in counting significance depending on the crystal.
The nominal cut $T$ (default 3.0) is therefore converted once, using the ISa of the merge being searched. For a reflection observed once $\sigma_\text{counting}^2 = \sigma^2 - (b\,I)^2$, so $I/\sigma_\text{counting} \ge T$ is exactly
$$\frac{I}{\sigma} \;\ge\; \frac{T}{\sqrt{1 + (T/\mathrm{ISa})^2}}$$
The converted cut lies strictly below $\mathrm{ISa}$ for every $\mathrm{ISa}$, so it is always reachable by the reflection the ceiling binds hardest, and it is within 1 % of $T$ on any merge with $\mathrm{ISa} \ge 21$ — a healthy merge is left exactly where it was. Multiplicity is taken as 1 deliberately rather than estimated, for the same reason. Where the merge reports no ISa (or $b = 0$) the cut is used as it stands.
Several space groups may share an absence pattern exactly. Where they do, the search scores them identically and **all of them are named** in the result rather than one being reported as the answer: some are enantiomorph pairs, which merged intensities cannot distinguish in principle, and others differ only by a screw condition that the centering condition already implies, so the screw has no observable signature at all. The representative reported first is the lowest space-group number, which is a convention and not a measurement.
@@ -974,15 +1010,15 @@ with one free constant per resolution shell, so every isotropic feature — the
The fit is on **intensities, with no positivity cut**. Fitting amplitudes, or dropping non-positive intensities as an amplitude-based tool must, loses roughly 40% of the signal: in a direction that has died half the merged intensities are negative, so a positivity cut keeps only the positive noise excursions and flattens the fall-off exactly where the anisotropy is largest.
**Two different quantities are reported, and they are not interchangeable.** $\Delta B$ (the range of the principal components) is a *rate*; the diffraction limit along each principal direction — where $\langle I/\sigma(I)\rangle$ in a 20° cone about that direction falls through 2 — is where the signal actually runs out. A crystal can have a large $\Delta B$ and almost no spread in directional limit, or the reverse.
**Two different quantities are reported, and they are not interchangeable.** $\Delta B$ (the range of the principal components) is a *rate*; the diffraction limit along each principal direction — where $\langle I/\sigma(I)\rangle$ in a 20° cone about that direction falls through 2, read by interpolation in $s^2$ over equal-count shells — is where the signal actually runs out. A crystal can have a large $\Delta B$ and almost no spread in directional limit, or the reverse. Where $\langle I/\sigma(I)\rangle$ never falls through 2 in a direction, the limit returned is the **edge of the measured data** rather than the crystal's own; such a direction is marked — with a `<` in the report, a 1 in `ANISOTROPY_D_MIN_CENSORED`, and a note in the mmCIF — so the spread is not read as a measurement when it is a lower bound. Both constants are AIMLESS's (cone half-angle 20°, the $\langle I/\sigma\rangle$ level this project states resolution at).
**The resolution signature.** A genuine DebyeWaller $B$ makes the directional deficit a straight line through the origin in $s^2$. The per-shell $\ell = 2$ amplitude is therefore fitted against $s^2$ and the curve is classified: *linear* (a real $B$), *flat* (a deficit that does not follow $\exp(-\tfrac12 \mathbf{s}^\mathsf{T} B \mathbf{s})$ at all, so the fitted $\Delta B$ describes the data with the wrong functional form and may be an **under**-estimate), or *convex* (a deficit that grows faster than $s^2$, which a $B$ cannot do). The verdict is re-derived at 8 and at 16 shells, and reported as undetermined if it moves.
**The verdict, and what it is measured against.** Whether an anisotropy is real is not decided against a counting-statistics error bar. Real data carry systematic error far larger than counting error, and gating on the latter reports anisotropy on datasets that have none. Instead the data set measures its own systematic error: in the tensor directions the Laue class *forbids*, the true tensor is exactly zero whatever the crystal is, so whatever is measured there is systematic. That measurement needs the unmerged observations — a merge has exact Laue symmetry by construction, and the forbidden directions are identically zero in it — so it is made on the scaled, rocking-curve-assembled observations. The counting part is subtracted, the counting error of the directions actually being tested is added back, and the ratio of $\Delta B$ to the result is banded: below 2 not established, 23.5 marginal, above 3.5 established, above 5 strong.
**The verdict, and what it is measured against.** Whether an anisotropy is real is not decided against a counting-statistics error bar. Real data carry systematic error far larger than counting error, and gating on the latter reports anisotropy on datasets that have none. Instead the data set measures its own systematic error: in the tensor directions the Laue class *forbids*, the true tensor is exactly zero whatever the crystal is, so whatever is measured there is systematic. That measurement needs the unmerged observations — a merge has exact Laue symmetry by construction, and the forbidden directions are identically zero in it — so it is made on the scaled, rocking-curve-assembled observations. The counting part is subtracted, the counting error of the directions actually being tested is added back, and the result is the **floor**. What is tested against it is $\Delta B_\text{linear}$ — the part of the fall-off that actually follows $\exp(-\tfrac12\mathbf{s}^\mathsf{T}B\mathbf{s})$, clamped at zero — and **not** the headline $\Delta B$; the report names which of the two it is quoting. The ratio is banded: below 2 not established, 23.5 marginal, above 3.5 established, above 5 strong. The bands are calibrated against known ground truth — merging cubic crystals in proper subgroups of their own Laue class, where the true anisotropy is exactly zero — which puts the false-positive rate at 24 % at 2.0, 10 % at 3.5 and 5 % at 5.0.
The report says **NOT DETECTED**, **DETECTED**, or **CANNOT DETERMINE**, and the third is a real answer rather than an evasion. It is returned when the Laue class is triclinic (no forbidden direction exists, so there is no internal measurement of the systematic error and no substitute for it), when the observed rotation range is under about 90° (a lab-fixed systematic then reaches several tensor directions instead of one), when the merged data are at the noise floor, when the scale model carried no dose term (an uncorrected dose ramp manufactures anisotropy that no significance test can see through), or when no unmerged observations were available. The smallest $\Delta B$ that could have been established on the data set is reported with the verdict; it is set by the systematic error rather than by counting, so it does **not** improve with more reflections or a longer exposure.
A too-high space-group assignment is the one failure mode that is silent: real anisotropy is then pushed into the directions used to measure the systematic error, which inflates the floor and biases the answer towards reporting none. A caution says so wherever the Laue class leaves a single free direction.
A too-high space-group assignment is the one failure mode that is silent: real anisotropy is then pushed into the directions used to measure the systematic error, which inflates the floor and biases the answer towards reporting none. A caution says so, but only where that is actually indicated — a single free direction, no detection, **and** a forbidden-direction measurement far above its own counting noise — rather than on every tetragonal, trigonal and hexagonal data set.
Everything lands in `<prefix>_report.txt` section 9 (`ANISOTROPY_*` keys), in the printed statistics, and in the merged mmCIF: the eigen-decomposition of the tensor as the standard `_reflns.pdbx_aniso_B_tensor_*` items (relative to the weakest direction, since only the deviatoric part is determined), and the directional limits, the shape and the verdict under the `_reflns.jfjoch_aniso_*` local prefix.
+35 -4
View File
@@ -7,6 +7,9 @@ To deploy Jungfraujoch, one needs to follow four steps:
3. Install Jungfraujoch writer
4. Install Python OpenAPI client
[`rugnux`](RUGNUX.md), the offline analysis tool, is installed separately and independently of all
four — see [Install rugnux](#install-rugnux-offline-analysis) at the end of this page.
Installation procedure depend a lot on the operating system. For RedHat Enterprise Linux 8/9, Rocky 8/9,
Ubuntu 22.04/24.04 or compatible, installation can be done with prebuilt packages from the
[package repositories](REPOSITORIES.md) and is relatively straightforward. For other systems one needs
@@ -90,11 +93,14 @@ $ dmesg |grep jfjoch
```
If things work, it is recommended to install the driver with DKMS, so it is rebuilt for kernel updates.
On RHEL 8 you can install the prebuilt `jfjoch-driver-dkms` package from the
[Gitea package registry](REPOSITORIES.md). On other systems follow procedure in
Install the prebuilt `jfjoch-driver-dkms` package from the
[Gitea package registry](REPOSITORIES.md); on other systems follow the procedure in
[PCIe driver](FPGA_PCIE_DRIVER.md).
NOTE: Driver installation procedure on non-RHEL 8 systems is not well understood/optimized at the moment.
DKMS builds the module for the kernel it is being installed for rather than the running one, so a
module built during a kernel update loads correctly after the reboot. RHEL 9.5 and later — and their
CentOS Stream, Rocky and AlmaLinux equivalents — build unaided; the `HAVE_VM_FLAGS_SET` workaround
earlier releases needed is obsolete.
NOTE: In case driver is included in the init RAM-disk image, it is necessary to rebuild the RAM-disk if driver is updated:
```
@@ -165,4 +171,29 @@ make jfjoch
Use pip:
```shell
pip install jfjoch-client
```
```
## Install rugnux (offline analysis)
`rugnux` is not part of the server stack and is installed independently of all of the above. It
needs neither the broker, the writer, Qt nor a CUDA toolkit — only an NVIDIA driver if you want to
use the GPU — and it does not have to run on the acquisition machine at all.
From the [package repositories](REPOSITORIES.md):
```
sudo dnf install rugnux # RHEL / Rocky
sudo apt install rugnux # Ubuntu
```
Or, on a machine no repository covers, from the standalone archive:
```
mkdir -p /opt/rugnux-<version>
tar xzf rugnux-<version>-linux-x86_64-cuda12.tgz -C /opt/rugnux-<version>
/opt/rugnux-<version>/bin/rugnux
```
The archive has no top-level directory, so the `-C` is required. See
[rugnux ▸ Installation](RUGNUX.md#installation) for the Arm and Windows archives, the driver
versions and building from source.
+8 -5
View File
@@ -93,8 +93,11 @@ Interfacing should be done through the JungfraujochDevice class in `fpga/host_li
Certain performance counters can be read through sysfs mechanism in the kernel.
One needs to `cat` files in `/sys/class/misc/jfjoch<number of device>/` directory.
## RHEL 9.5+ issue
RedHat Enterprise Linux 9.5 backported modification to settings virtual memory flags from Linux kernel 6.3, while still operating kernel version 5.14.
It is complicated to come up with a single rule to select when newer functions should be used, so it works with RHEL 9.5+,
while still being compatible with other Linux distributions. It is even more complex given not all RHEL compatible distributions adopted the change at the same version.
For the moment the quick fix is to define an environment variable `HAVE_VM_FLAGS_SET` before making the kernel.
## RHEL 9.5+ virtual memory flags
RedHat Enterprise Linux 9.5 backported the `vm_flags_set` interface from Linux kernel 6.3 while still reporting kernel version 5.14, so a plain kernel-version test picks the wrong branch and the build fails.
This is now detected automatically from `RHEL_RELEASE_CODE`, so the module builds unaided on RHEL 9.5 and later and on the CentOS Stream, Rocky and AlmaLinux equivalents, as well as on distributions that have not backported it.
**No user action is needed.** The `HAVE_VM_FLAGS_SET` environment variable that earlier releases required is obsolete; it is still honoured if set, but setting it is no longer necessary and the DKMS packaging never passed it anyway.
## Which kernel DKMS builds for
The DKMS package builds the module for the kernel it is being **installed for**, not the one currently running, so a module built while a kernel update is being applied loads correctly after the reboot.
Building by hand in `fpga/pcie_driver/` still defaults to the running kernel; pass `KDIR=/lib/modules/<version>/build` (or `KVER=<version>`) to target another one.
+3 -3
View File
@@ -147,9 +147,9 @@ Notes:
- For a machine without an NVIDIA GPU, add `-DJFJOCH_USE_CUDA=OFF`: the viewer then runs the same
pipeline on the CPU (FFTW indexer) at lower throughput.
To produce a self-contained installer (bundles the Qt runtime via `windeployqt`, the analysis CLIs,
and — on the CUDA build — the cuFFT runtime DLL, so the target host needs neither Qt nor a CUDA
toolkit), with NSIS installed:
To produce a self-contained installer (bundles the Qt runtime via `windeployqt` and — on the CUDA
build — the cuFFT runtime DLL, so the target host needs neither Qt nor a CUDA toolkit), with NSIS
installed:
```
cd build-win
+41 -26
View File
@@ -13,9 +13,12 @@ result see [Deployment](DEPLOYMENT.md); for the package-repository URLs see
| Artefact | Distributed via | Contains |
| --- | --- | --- |
| `.rpm` / `.deb` packages | [package repositories](REPOSITORIES.md) | The full server stack: `jfjoch` (broker, frontend, FPGA and detector tools), `jfjoch-writer`, `jfjoch-viewer` (incl. the XDS plugin), `jfjoch-driver-dkms` |
| `jfjoch_viewer-<version>-linux-cuda<major>.tgz`, `...-linux-cpu.tgz` | Gitea release page | Portable Linux viewer package: `jfjoch_viewer`, `rugnux`, `jfjoch_extract_hkl`, `jfjoch_recompress` and the license notices |
| `jfjoch-viewer-<version>-win64-cuda<major>.exe`, `...-win64-cpu.exe` | Gitea release page | Windows installer with the same four programs, plus the Qt runtime |
| `.rpm` / `.deb` packages | [package repositories](REPOSITORIES.md) | The full server stack: `jfjoch` (broker, frontend, FPGA and detector tools), `jfjoch-writer`, `jfjoch-viewer` (incl. the XDS plugin), `jfjoch-driver-dkms`, and `rugnux` (offline analysis, independent of the rest) |
| `jfjoch_viewer-<version>-linux-cuda<major>.tgz`, `...-linux-cpu.tgz` | Gitea release page | Portable Linux viewer: `jfjoch_viewer`, its desktop entry, icon and D-Bus service, and the license notices |
| `jfjoch-viewer-<version>-win64-cuda<major>.exe`, `...-win64-cpu.exe` | Gitea release page | Windows installer for `jfjoch_viewer`, plus the Qt runtime |
| `rugnux-<version>-linux-x86_64-cuda<major>.tgz` | Gitea release page | Portable Linux [`rugnux`](RUGNUX.md), the offline analysis CLI, and the license notices. One executable |
| `rugnux-<version>-linux-aarch64-cuda<major>.tgz` | Gitea release page | The same, cross-built for 64-bit Arm — NVIDIA GH200 and DGX Spark |
| `rugnux-<version>-win64-cuda<major>.zip` | Gitea release page | The same for Windows, plus the cuFFT DLL |
| `jfjoch-writer` `.rpm` / `.deb` | Gitea release page | The writer alone, for a file-writing machine without the rest of the stack |
| `libjfjoch_xds_plugin.so.<version>` | Gitea release page | XDS HDF5 read plugin (built on RHEL 8); see [Integration with MX software](SOFTWARE_INTEGRATION.md) |
| `jfjoch-client` | [PyPI](https://pypi.org/project/jfjoch-client/) and the Gitea PyPI index | Generated Python OpenAPI client |
@@ -48,10 +51,16 @@ hardware, build from source without the flags.
## Operating-system floor
The `.rpm` / `.deb` packages are built per distribution (RHEL/Rocky 8 and 9, Ubuntu 22.04 and 24.04)
and are tied to it. The portable viewer `.tgz` is built on RHEL 8, the oldest supported
distribution, so its glibc floor is low enough to run on any newer Linux — that is what it is for,
and why it replaces the per-distro packaging of the viewer on the release page. The Windows
installer is built and verified on Windows 11.
and are tied to it. The portable viewer `.tgz` and the **x86_64** `rugnux` `.tgz` are built on
RHEL 8, the oldest supported distribution, so their glibc floor is low enough to run on any newer
Linux — that is what they are for, and why they replace the per-distro packaging of those programs
on the release page. The **aarch64** `rugnux` `.tgz` is the exception: it is cross-built against
Ubuntu 24.04, so it needs glibc 2.39 or newer (which DGX OS 7 and any current Arm server distribution
have). The Windows installer is built and verified on Windows 11.
**The portable archives have no top-level directory.** They unpack straight into `bin/` and
`share/`, so always extract them into a directory of their own (`tar xzf … -C /opt/rugnux-<version>`)
rather than into a working directory.
## CUDA and non-CUDA builds
@@ -63,22 +72,22 @@ The CUDA toolkit used is the one on the corresponding build machine: **CUDA 12**
packages, **CUDA 13** for RHEL 9, Ubuntu and Windows. The major version is part of the artefact and
repository name, so a download is self-identifying. Building from source needs CUDA 12.8 or newer.
**A CUDA build does not require a CUDA machine.** Of the CUDA components only **cuFFT** is linked
dynamically — the CUDA runtime and the fast-feedback indexer are linked statically — and cuFFT
itself has no link-time dependency on the NVIDIA driver library. Jungfraujoch asks how many CUDA
devices are present at start-up and treats "none" (including "no driver installed") as zero GPUs,
falling back to the CPU path. So a CUDA build starts and runs correctly on a machine with no NVIDIA
GPU at all, provided the cuFFT runtime can be loaded:
**A CUDA build does not require a CUDA machine.** Jungfraujoch asks how many CUDA devices are
present at start-up and treats "none" (including "no driver installed") as zero GPUs, falling back
to the CPU path. So a CUDA build starts and runs correctly on a machine with no NVIDIA GPU at all.
What each artefact has to find at run time differs:
- **Portable `.tgz` and Windows installer** — cuFFT is **part of the distribution**, shipped next to
the executable (on Linux found through an `$ORIGIN` rpath). Nothing else is needed: no CUDA
toolkit, and on a GPU machine only the NVIDIA driver.
- **`.rpm` / `.deb`** — cuFFT comes from the distribution's own CUDA packages, so that one
dependency is managed centrally with the rest of CUDA. Install the cuFFT package alongside, or use
the `nocuda` repositories on a machine where CUDA is not wanted.
- **Portable Linux `.tgz`** — nothing. The CUDA runtime, the fast-feedback indexer **and cuFFT** are
all linked statically, so each archive is a single executable that depends on nothing but the C
and C++ runtimes. On a GPU machine the NVIDIA driver is the only NVIDIA component needed.
- **Windows installer and `.zip`** — the CUDA toolkit ships no static cuFFT for Windows, so the
cuFFT DLL is **part of the distribution**, next to the executable. No CUDA toolkit is needed.
- **`.rpm` / `.deb`** — these deliberately keep cuFFT dynamic, so that one dependency is managed
centrally with the rest of CUDA. Install the cuFFT package alongside, or use the `nocuda`
repositories on a machine where CUDA is not wanted.
The cuFFT runtime is large (the Windows DLL is ~256 MB), so the CUDA artefacts are correspondingly
bigger than the CPU ones — the other reason for shipping both.
Static CUDA linkage makes the Linux CUDA artefacts substantially bigger than the CPU ones, and the
Windows cuFFT DLL is ~256 MB — which is the other reason for shipping both variants.
On a machine with an NVIDIA GPU, take the CUDA variant: only that one uses the GPU.
@@ -91,10 +100,14 @@ is no CUDA-toolkit version requirement on the host.
| Artefact | CUDA toolkit | GPU generations | Minimum driver |
| --- | --- | --- | --- |
| RHEL 8 packages, portable Linux `.tgz` | 12.9 | Volta (V100) through Blackwell: `sm_70`, `75`, `80`, `86`, `89`, `90`, `100`, `120`, `121` | 525.60.13 |
| RHEL 9 and Ubuntu packages, Windows installer | 13.x | Turing (T4) through Blackwell: the same list **without** `sm_70` | 580.65.06 (Linux), R580 (Windows) |
| RHEL 8 packages, portable viewer `.tgz`, x86_64 `rugnux` `.tgz` | 12.9 | Volta (V100) through Blackwell: `sm_70`, `75`, `80`, `86`, `89`, `90`, `100`, `120`, `121` | 525.60.13 |
| RHEL 9 and Ubuntu packages, Windows installer, Windows `rugnux` `.zip` | 13.x | Turing (T4) through Blackwell: the same list **without** `sm_70` | 580.65.06 (Linux), R580 (Windows) |
| aarch64 `rugnux` `.tgz` | 13.x | `sm_90` (GH200) and `sm_121` (DGX Spark) only | 580.65.06 |
| any `cpu` / `nocuda` variant | — | — | none |
The aarch64 build is cross-compiled and verified in CI to be Arm, self-contained and to carry both
GPU targets, but it is **not exercised on hardware** — CI has no GH200 or Spark runner.
**A V100 needs the CUDA 12 build.** CUDA 13 dropped offline compilation for Volta, and the PTX a
fatbin also carries only ever JIT-compiles *forwards*, so a CUDA 13 artefact contains nothing a V100
can execute: every kernel launch fails with *no kernel image is available for execution on the
@@ -114,8 +127,8 @@ installed driver, which fails with `cudaErrorCallRequiresNewerDriver`.
## Windows installer
The Windows artefact covers `jfjoch_viewer` and the portable analysis CLIs only; the rest of
Jungfraujoch (broker, receiver, FPGA host, detector control) is Linux-only.
The Windows artefacts are the `jfjoch_viewer` installer and the separate `rugnux` `.zip`; the rest
of Jungfraujoch (broker, receiver, FPGA host, detector control) is Linux-only.
The toolchain bounds of the released installer are:
@@ -141,5 +154,7 @@ To build the viewer yourself on Windows, see
## Licenses
Every package variant carries the project license, the third-party manifest and the verbatim
license texts of the bundled dependencies under `share/doc/jfjoch`. See
license texts of the bundled dependencies, each package under a directory of its own —
`share/doc/jfjoch_broker`, `jfjoch_writer`, `jfjoch_viewer`, `jfjoch_rugnux`, `jfjoch_driver_dkms`
so that no two packages claim the same path and they can be upgraded independently. See
[Third-party software notices](THIRD_PARTY_NOTICES.md).
+8 -3
View File
@@ -32,7 +32,12 @@ We provide the following packages in the repository:
* jfjoch — broker, web frontend, FPGA and detector command-line tools
* jfjoch-driver-dkms — PCIe kernel-module source, built by DKMS
* jfjoch-writer — HDF5 writer service
* jfjoch-viewer — desktop viewer, the offline analysis tools and the XDS plugin
* jfjoch-viewer — desktop viewer and the XDS plugin
* rugnux — [offline analysis CLI](RUGNUX.md), independent of the acquisition stack
Note that `rugnux` is named without the `jfjoch-` prefix, matching the program and the release
archive. Up to 1.0.0-rc.163 it was part of `jfjoch-viewer`; the package declares that move, so
installing it upgrades an older viewer rather than colliding with it.
## Ubuntu based systems
@@ -46,8 +51,8 @@ sudo apt update
`$distribution` uses Ubuntu names `jammy` (22.04) and `noble` (24.04). `$component` can be set to `cuda13` and `nocuda`.
Only slsDetectorPackage 8.0.2 is built for Ubuntu.
The same four packages as above are provided: `jfjoch`, `jfjoch-driver-dkms`, `jfjoch-writer` and
`jfjoch-viewer`. Up to 1.0.0-rc.160 the first of them was misnamed `jfjoch-jfjoch`; the current
The same five packages as above are provided: `jfjoch`, `jfjoch-driver-dkms`, `jfjoch-writer`,
`jfjoch-viewer` and `rugnux`. Up to 1.0.0-rc.160 the first of them was misnamed `jfjoch-jfjoch`; the current
package replaces it, so `apt upgrade` handles the rename.
Ubuntu packages are currently only going through a very limited testing.
+284 -4
View File
@@ -10,10 +10,103 @@ requested.
It runs the *same* analysis code as the online and interactive tools, just driven from the
command line over a file rather than a live detector stream.
```
rugnux {<options>} <input.h5>
```
Run it with no arguments to print the usage.
> **Note.** `rugnux` is under very active development. This page describes the tool and
> its options at a high level; the authoritative, always-current list of options is the program's
> own usage message — run `rugnux` with no arguments.
## Installation
`rugnux` is a **single self-contained executable**. It needs no CUDA toolkit, no Qt, and no
Jungfraujoch service running anywhere; on a machine with an NVIDIA GPU it needs the NVIDIA
**driver**, and without one it still runs on the CPU.
### From the package repositories (RHEL / Rocky / Ubuntu)
On a distribution covered by the [package repositories](REPOSITORIES.md), `rugnux` is a package of
its own:
```
sudo dnf install rugnux # RHEL / Rocky 8 and 9
sudo apt install rugnux # Ubuntu 22.04 / 24.04
```
It installs `/usr/bin/rugnux` and depends on nothing from the acquisition side — no broker, no
detector libraries, no Qt — so it can go on a machine that only processes data.
> **Upgrading from rc.163 or earlier.** `/usr/bin/rugnux` used to belong to the `jfjoch-viewer`
> package. The `rugnux` package declares that the file has moved, so installing it upgrades an old
> `jfjoch-viewer` in the same transaction instead of failing on the duplicate path. If your
> `jfjoch-viewer` is pinned to an old version, unpin it or remove it first.
### From the release archive
For a machine no package manager covers — or for Windows and Arm, which have no repository — take
the archive for your architecture from the Gitea release page:
| Archive | For |
| --- | --- |
| `rugnux-<version>-linux-x86_64-cuda12.tgz` | 64-bit Intel/AMD Linux. Built on RHEL 8, so it runs on any newer Linux |
| `rugnux-<version>-linux-aarch64-cuda13.tgz` | 64-bit Arm Linux — NVIDIA GH200 and DGX Spark. Built on Ubuntu 24.04, so it needs glibc 2.39 or newer. Cross-compiled and **not yet exercised on Arm hardware** |
| `rugnux-<version>-win64-cuda13.zip` | 64-bit Windows |
**The archive has no top-level directory** — it unpacks straight into `bin/` and `share/`. Always
give `tar` a destination of its own, or it will scatter those into whatever directory you are in:
```
mkdir -p /opt/rugnux-1.0.0
tar xzf rugnux-1.0.0-linux-x86_64-cuda12.tgz -C /opt/rugnux-1.0.0
/opt/rugnux-1.0.0/bin/rugnux # prints the usage
```
What you get is:
```
bin/rugnux the program
share/doc/jfjoch_rugnux/LICENSE GPLv3
share/doc/jfjoch_rugnux/THIRD_PARTY_NOTICES.md
share/doc/jfjoch_rugnux/licenses/ verbatim licence texts of the bundled dependencies
```
Nothing is written outside that directory, nothing needs root, and several versions can sit side by
side. To remove it, delete the directory. Put `bin/` on your `PATH` if you want to type `rugnux`
rather than the full path.
> **Mixing the two.** If a `rugnux` package is also installed, `/usr/bin/rugnux` will normally win
> on `PATH`. Put the archive's `bin/` first, or call it by its full path, to be sure which one you
> are running — `rugnux` prints its version on every run.
### GPU support
The released archives are CUDA builds. They need only an NVIDIA **driver** on the host — 525.60.13
or newer for the CUDA 12 archive, 580.65.06 or newer for the CUDA 13 ones — and no CUDA toolkit,
because everything CUDA is linked statically. With no GPU or no driver, `rugnux` reports zero CUDA
devices and falls back to the CPU path, which works but is far slower and offers only the `fftw`
indexer. A **V100 needs the CUDA 12 archive**; which generations each build covers is in
[Release contents ▸ GPU generations and the NVIDIA driver](RELEASE_CONTENTS.md#gpu-generations-and-the-nvidia-driver).
### Building from source
`rugnux` alone, without the server stack or Qt:
```
cmake -S . -B build -DJFJOCH_RUGNUX_ONLY=ON -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_FLAGS="-march=x86-64-v3" -DCMAKE_C_FLAGS="-march=x86-64-v3"
cmake --build build -j$(nproc) --target rugnux
```
The binary lands in `build/rugnux/rugnux`. Two dependencies must come from the system — **zlib** and
**Eigen ≥ 3.4** (`zlib-devel` and `eigen3-devel`, or their Debian equivalents); everything else is
downloaded during the first configure, which therefore needs network access. `cmake --build build
--target package` produces the same `.tgz` the release ships. The `-march` flag is not set by the
build system on purpose, so a plain build is slower than the released one on the CPU-bound stages —
see the note in `CMakeLists.txt`.
## Where it fits among the three analysis tools
| Tool | Mode | Driven by | Output |
@@ -35,9 +128,9 @@ prediction and Bragg integration — as does rotation scaling and merging, with
the fallback where there is no GPU. The thread count (`-N`) governs the CPU side of all of it.
The released CUDA builds need only an NVIDIA **driver** on the host, no CUDA toolkit: 525.60.13 or
newer for the CUDA 12 artefacts (RHEL 8 packages, portable Linux `.tgz`) and 580.65.06 or newer for
the CUDA 13 ones (RHEL 9, Ubuntu, Windows). Which GPU generations each artefact supports — a V100
in particular works only with the CUDA 12 build — is in
newer for the CUDA 12 artefacts (RHEL 8 packages, the x86_64 `rugnux` archive) and 580.65.06 or
newer for the CUDA 13 ones (RHEL 9, Ubuntu, the aarch64 and Windows `rugnux` archives). Which GPU
generations each artefact supports — a V100 in particular works only with the CUDA 12 build — is in
[Release contents ▸ GPU generations and the NVIDIA driver](RELEASE_CONTENTS.md#gpu-generations-and-the-nvidia-driver).
## Input and output
@@ -71,6 +164,11 @@ reusing them would hide the spot-finding settings from the lattice search.
All three carry the **refined unit cell** (from rotation indexing) and the **space group determined
from systematic absences** (constrained to the indexed lattice symmetry).
- `<prefix>_unmerged.mtz`**opt-in** (`--export-unmerged`): the integrated observations *before*
merging, as an unmerged MTZ in POINTLESS's column layout, so the data can be scaled and merged by
**aimless**, **pointless**, **careless** or `iotbx.merging_statistics` instead of by rugnux. See
[The unmerged export](#the-unmerged-export) below. `--export-unmerged-partials` writes
`<prefix>_unmerged_partials.mtz`, one row per image, instead of summing.
- `<prefix>_report.txt` — the **results report**: what the run determined, in a form both a person and
a beamline script can read. Always written, next to the files above. See
[The results report](#the-results-report) below.
@@ -101,6 +199,10 @@ ignores them, and one that does can find them without guessing.
| `_reflns.jfjoch_radiation_damage_relative_B` | Relative *B* from the first to the last rotation batch (Ų); positive is the usual direction, high-resolution intensity fading with dose |
| `_jfjoch_radiation_damage_batch.*` | Per-batch loop: `id`, `rotation_start_deg`, `relative_B` |
| `_diffrn_detector.jfjoch_distance_mm`, `_jfjoch_beam_center_x_pxl`, `_jfjoch_beam_center_y_pxl` | The refined detector geometry actually used, which is not otherwise recoverable from the reflection file |
| `_reflns.pdbx_aniso_B_tensor_eigenvalue_1..3`, `_pdbx_aniso_B_tensor_eigenvector_*` | The anisotropy tensor, eigen-decomposed. Eigenvalues are **relative to the weakest direction** (so the third is 0 and the first is the anisotropic Δ*B*), because only the deviatoric part is determined; eigenvectors are in the PDB orthogonalisation convention. Not written for a cubic Laue class, where symmetry forces Δ*B* to be zero |
| `_reflns.jfjoch_aniso_delta_B`, `_jfjoch_aniso_delta_B_linear` | The anisotropic Δ*B*, and the part of it that actually follows exp(−½ **s**ᵀ*B***s**). The second is what the verdict is gated on |
| `_reflns.jfjoch_aniso_d_min_1..3` | Diffraction limit (Å) along each principal direction. A comment marks a value that is the edge of the measured data rather than the crystal's own limit |
| `_reflns.jfjoch_aniso_shape`, `_jfjoch_aniso_floor`, `_jfjoch_aniso_significance`, `_jfjoch_aniso_verdict` | The resolution signature of the deficit, the data set's own systematic-error floor, Δ*B*<sub>linear</sub> over that floor, and the resulting verdict. Each carries its vocabulary as a comment |
> **Compatibility note.** Before rc.161, `_reflns.jfjoch_diffrn_ISa` carried the *asymptote*, not the
> whole-range value. There is no version marker inside the file, so a number taken from an older
@@ -117,6 +219,56 @@ expect. Two properties worth knowing before using it:
untouched, but the absolute scale is not meaningful. This matters only if you intend to compare
magnitudes with another file; SHELXC and ANODE use ratios alone.
## The unmerged export
`--export-unmerged` writes `<prefix>_unmerged.mtz`: every integrated observation, before scaling and
merging, in the column layout POINTLESS writes and **aimless**, **pointless**, **careless** and
`iotbx.merging_statistics` read. It is written in `--mode mx` and `--mode scale` alike, off by
default, and does not replace anything — rugnux still writes its own merged files in the same run.
Use it to scale the data with a different program, to have pointless give an independent opinion on
the space group, or to compare rugnux's merge against another one on identical input.
**Columns.** `H K L M/ISYM BATCH I SIGI FRACTIONCALC XDET YDET ROT LP FLAG` — POINTLESS's own set —
plus four rugnux extras, `DELPHI` (offset from the centre of the rocking curve), `ZETA` (the Lorentz
geometry of that curve), `BGMEAN` and `BGVAR` (the background that was subtracted, and its variance).
`BATCH` is the image ordinal plus one, and a batch header is written for every batch that carries an
observation. `M/ISYM` records both the symmetry operation and the Friedel hand, so the index as
measured is recoverable from the index as stored.
**What has been applied to the intensities, and what has not.** `I` and `SIGI` carry the
**Lorentz-polarization factor and nothing else**; the factor itself is in the `LP` column, so raw
counts are `I/LP`. LP is applied because it is per-observation geometry that varies by more than two
orders of magnitude across a sweep and no reader can reconstruct it. Deliberately *not* applied:
the **partiality is not divided out** (it is reported in `FRACTIONCALC`), and the **per-image scale is
not applied at all** — those programs fit their own scale model, and handing them pre-scaled data
would have them fit a correction to a correction. No resolution cut, outlier rejection or ice-ring
filtering is applied either.
**Partials.** On a rotation run the partials of each reflection are summed into one full, using the
same rule rugnux's own 3D combine uses — consecutive frames no more than two apart — and the full is
written at the batch its rocking curve is centred on, with the summed rocking-curve fraction in
`FRACTIONCALC`. An event that caught less of its rocking curve than `--min-partiality` is not
written, exactly as in the merge. Summing is the default because a downstream program's own partial
handling is far more conservative than rugnux's: given raw partials, aimless accepted a small
fraction of the file and merged at a fraction of the multiplicity; given summed fulls it uses
essentially all of it. `--export-unmerged-partials` writes the unsummed form to
`<prefix>_unmerged_partials.mtz` for a program that would rather sum them itself. Stills have no
rocking events and are the same either way.
**Systematic absences.** Lattice-**centring** absences are not written; **screw and glide** absences
are. Prediction runs in a primitive setting so that the space-group search can test the centring,
but the interstitial reflections that leaves make a reading program take the lattice for primitive
and demote the group. Screw and glide absences are kept because they are the evidence the space
group was chosen on — deleting them would turn a reading program's test into an assumption. XDS and
DIALS draw the line in the same place.
**Scan axis.** The batch headers carry the goniometer axis **negated** relative to the one in the
input file. This is not a correction to the file: rugnux brings an observation made at angle φ back
to zero by rotating it by +φ, so the crystal itself turns by −φ, and an MTZ batch header records the
axis a batch's own increasing `PHI` turns the crystal about. With the sign as exported, pointless's
independently determined orientation matrix agrees with rugnux's to well under a degree.
## The results report
`<prefix>_report.txt` records **what the run determined**, next to the reflection files. It is
@@ -150,7 +302,7 @@ below are an interface other software may depend on: they do not change without
Sections, in order: `1. DATA SET`, `2. INDEXING`, `3. GEOMETRY POST-REFINEMENT` (rotation only),
`4. SPACE GROUP DETERMINATION`, `5. SCALING AND MERGING`, `6. TWINNING`, `7. RADIATION DAMAGE`,
`8. SWEEP QUALITY`, `9. WARNINGS`.
`8. SWEEP QUALITY`, `9. DIFFRACTION ANISOTROPY`, `10. WARNINGS`.
**`SPOT_RESOLUTION_ESTIMATE=`** in section 1 is how far the merged data are expected to reach, read
off the found spots alone — no lattice, no integration, no merge — so it is there on a run that never
@@ -215,6 +367,50 @@ scaled at all. Every range also appears as a `WARNING:` sentence in section 9.
The same finding is written **per image** into the `_process.h5` as `/entry/MX/sweepQuality`, when
one is written — see [HDF5](HDF5.md#41-entrymx--spot-finding-and-indexing-cxi-style).
### Diffraction anisotropy
Section 9 reports how much the fall-off with resolution depends on **direction**, and whether that is
established above the data set's own systematic error. It runs automatically on every merging run —
there is no flag — and it is a **description only**: no intensity is corrected, no reflection is
removed on a directional criterion, and the merged data and the written reflection files do not
depend on direction at all. The algorithm is in
[CPU/GPU data analysis ▸ Diffraction anisotropy](CPU_DATA_ANALYSIS.md).
Two different quantities are reported and they are not interchangeable. `ANISOTROPY_DELTA_B` is a
*rate* — the range of the principal components of the anisotropy tensor, on the ordinary
crystallographic *B* scale, so it is directly comparable with phenix.xtriage's `B_cart`, ctruncate's
anisotropic *B* and AIMLESS's anisotropic Δ*B*. `ANISOTROPY_D_MIN_PRINCIPAL` is where the signal
actually *runs out* along each principal direction. A crystal can have a large Δ*B* and almost no
spread in directional limit, or the reverse.
| key | meaning |
|---|---|
| `ANISOTROPY_VERDICT` | `DETECTED` \| `NOT_DETECTED` \| `CANNOT_DETERMINE` |
| `ANISOTROPY_FREE_DIRECTIONS` | Deviatoric directions the Laue class allows — 5 triclinic, 3 monoclinic, 2 orthorhombic, 1 tetragonal/trigonal/hexagonal, **0 cubic** |
| `ANISOTROPY_DELTA_B` | The anisotropic Δ*B* (Ų), fitted on intensities with nothing dropped |
| `ANISOTROPY_DELTA_B_LINEAR` | The part of it that follows exp(−½ **s**ᵀ*B***s**). **This is the number the verdict is gated on**, and the report says which of the two it is quoting |
| `ANISOTROPY_PRINCIPAL_B` | The three principal components, relative to the weakest |
| `ANISOTROPY_D_MIN_PRINCIPAL` | Diffraction limit (Å) along each principal direction — where ⟨I/σ(I)⟩ in a 20° cone about it falls through 2 |
| `ANISOTROPY_D_MIN_CENSORED` | One flag per direction. `1` means ⟨I/σ(I)⟩ never fell through 2, so the limit is the **edge of the measured data**, a bound and not a measurement. The prose marks it with a `<` |
| `ANISOTROPY_D_MIN_SPREAD` | Range of the three limits — itself a lower bound if any is censored |
| `ANISOTROPY_SHAPE` | `LINEAR` (a real DebyeWaller *B*) \| `FLAT` (the deficit does not follow a *B* at all, so Δ*B* may be an **under**-estimate) \| `CONVEX` (grows faster than a *B* can) \| `UNDETERMINED` (the verdict moved on rebinning) |
| `ANISOTROPY_FLOOR`, `ANISOTROPY_SIGNIFICANCE` | The data set's own systematic-error floor (Ų) and Δ*B*<sub>linear</sub> over it. Banded: below 2 not established, 23.5 marginal, above 3.5 established, above 5 strong |
| `ANISOTROPY_DETECTION_LIMIT` | The smallest Δ*B* that could have been established on these data. It is set by systematic error, not by counting, so it does **not** improve with more reflections or a longer exposure |
| `ANISOTROPY_N_OBSERVATIONS`, `ANISOTROPY_FORBIDDEN_Z`, `ANISOTROPY_SIGMA_SYSTEMATIC` | The unmerged observations the floor was measured on, that measurement against its own counting noise, and the floor before the counting part is added back |
**`CANNOT_DETERMINE` is a real answer, not an evasion.** The verdict is not measured against counting
statistics — real data carry systematic error far larger than that, and gating on counting error
reports anisotropy on data sets that have none. Instead the data set measures its own systematic
error in the tensor directions its Laue class *forbids*, where the true value is exactly zero
whatever the crystal is. Where that measurement cannot be made, the run says so and gives the
reason: a **triclinic** Laue class (no forbidden direction exists), an observed rotation under about
90°, merged data at the noise floor, a scale model carrying no dose term
(`--no-scaling-corrections`), or no unmerged observations. A **cubic** Laue class is different again
— symmetry forces Δ*B* to be exactly zero, and the run says that rather than reporting a measurement.
Where anisotropy is detected and the directional limits differ by more than 0.5 Å, a `WARNING:` line
says so, since refinement and map interpretation should allow for it.
## Validating against a model (`rugnux --model`)
Given a PDB atomic model of the same structure, `--model model.pdb` scales the model structure
@@ -241,6 +437,13 @@ not run a space-group search: it merges in the space group and unit cell the fil
/ `-C` override them. A `_process.h5` written before the group was stored carries none, and merges
in P1 unless `-S` says otherwise.
Where the full run re-seated the lattice — the space group it settled on is in a different setting
from the one each image was indexed in — the file records the change of basis as
`/entry/MX/reindexMatrix`, and `rugnux` applies it on read, so the reflections and the stored cell
describe the same frame. An older file that was affected by this cannot be repaired (the matrix is
not recoverable after the fact); such a file now stops with a message naming both cells and the
exact `-S`/`-C` override to merge it in its own setting, instead of failing inside the merge.
## Detector calibration from powder rings (`rugnux --mode calibration`)
The `calibration` mode determines the detector geometry — PONI *x*/*y*, the two tilts
@@ -307,6 +510,81 @@ but not the direct beam once the detector is tilted (see above).
## Quick start
### A first run, start to finish
Process a rotation sweep and merge it, with nothing assumed about the crystal:
```
rugnux -o myrun -N 32 /path/to/dataset_master.h5
```
`-o myrun` is the prefix every output file is named from, `-N 32` is the worker-thread count, and the
last argument is the **master** file of a Jungfraujoch dataset. That is the whole command — the
goniometer axis in the file tells rugnux this is a rotation sweep, the unit cell comes from indexing
the data, the space group from its systematic absences, and the resolution limit from where CC1/2
falls off.
Progress, statistics and timing go to the terminal. When it finishes you have five files next to
each other:
```
myrun.mtz merged intensities + French-Wilson amplitudes, for CCP4 / phenix
myrun.cif the same, as mmCIF - the self-describing format, and what to deposit
myrun.hkl the same, as SHELX HKLF 4 - feed this to SHELXC / SHELXD / ANODE
myrun_report.txt what the run determined: cell, space group, statistics, warnings
myrun_image.dat one row per image, for plotting how the crystal behaved over the sweep
```
Read `myrun_report.txt` first — it is written for a person, top to bottom, and it says which space
group was chosen and on what evidence, how far the data go, and anything that needs attention. To
pull one number out of it in a script, every value is a `KEY= value` line:
```
grep '^SPACE_GROUP_NUMBER= ' myrun_report.txt
grep '^UNIT_CELL_CONSTANTS= ' myrun_report.txt
grep '^INCLUDE_RESOLUTION_RANGE= ' myrun_report.txt
grep '^ISA= ' myrun_report.txt
grep '^WARNING:' myrun_report.txt
```
**Useful variations**, each independent of the others:
```
# tell it where the data really stop, if you already know - this sharpens the
# space-group search and the error model
rugnux -o myrun -N 32 --scaling-high-resolution 1.4 dataset_master.h5
# keep Friedel pairs apart, for anomalous work
rugnux -o myrun -N 32 -A dataset_master.h5
# a quick look at the first 200 images only
rugnux -o quicklook -N 32 -e 200 dataset_master.h5
# merge as usual, but also keep the per-image file so the data can be re-merged later
rugnux -o myrun -N 32 --write-process-h5 dataset_master.h5
# also write an unmerged MTZ, to scale the data with aimless instead
rugnux -o myrun -N 32 --export-unmerged dataset_master.h5
# check the merged data against a known structure: R-work / R-free and maps
rugnux -o myrun -N 32 --model model.pdb dataset_master.h5
```
Re-merging is cheap and does not re-read the images. Ask the full run to keep its per-image file
with `--write-process-h5`, and `--mode scale` will then re-scale and re-merge the reflections
already integrated in it — seconds rather than minutes:
```
rugnux -o myrun -N 32 --write-process-h5 dataset_master.h5 # integrate and merge once
rugnux --mode scale -o remerged -A myrun_process.h5 # re-merge, here anomalously
```
Use it to try a different resolution limit, anomalous setting, outlier rejection or reference MTZ
without paying for integration again. `--mode scale` merges in the space group and cell the file
records, so the second command needs no `-S`. Note that `--no-merge` also writes a `_process.h5`,
but a run that never merged never determined a space group either, so re-merging that file lands in
P1 unless you pass `-S` yourself — `--write-process-h5` is the one to use.
### Rotation data
Index, integrate, scale and merge a rotation sweep, fully de novo:
@@ -527,6 +805,8 @@ Scaling and merging:
| `--reference-column <label>` | Reference MTZ column to use (default: auto — F-model, else IMEAN/I/…) |
| `--model <file.pdb>` | After merging, validate the merged intensities against this atomic model (see below) |
| `--write-process-h5` | Also write the (large) `_process.h5` when merging (default: only `.mtz`/`.cif`) |
| `--export-unmerged` | Also write the integrated observations as `<prefix>_unmerged.mtz`, an unmerged MTZ (POINTLESS column layout) for aimless / pointless / careless. Rotation partials are summed into one full per reflection. Intensities carry the Lorentz-polarization factor and nothing else — the partiality is not divided out and the per-image scale is not applied. Works in `--mode mx` and `--mode scale`. See [The unmerged export](#the-unmerged-export) |
| `--export-unmerged-partials` | As above, but writes `<prefix>_unmerged_partials.mtz` with each partial as its own row (one batch per image) for the reading program to sum |
Integration:
+11 -7
View File
@@ -7,13 +7,17 @@ On experimental basis we also build repositories for Ubuntu 22.04 and 24.04.
Running Jungfraujoch on Red Hat Enterprise Linux 7 is currently not tested and not recommended,
but likely possible with providing some packages from external repositories.
The desktop viewer `jfjoch_viewer` (only) additionally runs on **Windows 11**, where it is shipped as
a pre-built installer; it can also be built from source with Visual Studio 2026 (MSVC), CUDA 13.3 and
Qt 6.11 — see [jfjoch_viewer ▸ Building from source on Windows](JFJOCH_VIEWER.md#building-from-source-on-windows).
The Windows installer bundles the Qt runtime, and on the CUDA build the CUDA runtime (cuFFT) as
well, so end users need neither Qt nor a CUDA toolkit installed — only an NVIDIA GPU driver for the
GPU path. The rest of Jungfraujoch is Linux-only. See [Release contents](RELEASE_CONTENTS.md) for the
CPU baseline and CUDA requirements of each released package.
Two programs additionally run on **Windows 11**: the desktop viewer `jfjoch_viewer`, shipped as a
pre-built installer, and [`rugnux`](RUGNUX.md), the offline analysis CLI, shipped as a `.zip`. Both
can be built from source with Visual Studio 2026 (MSVC) and CUDA 13.3 — the viewer additionally
needs Qt 6.11; see
[jfjoch_viewer ▸ Building from source on Windows](JFJOCH_VIEWER.md#building-from-source-on-windows).
The Windows artefacts bundle the Qt runtime (viewer only) and, on the CUDA builds, the cuFFT DLL, so
end users need neither Qt nor a CUDA toolkit installed — only an NVIDIA GPU driver for the GPU path.
On Linux the portable archives link CUDA entirely statically and so need nothing but the driver.
`rugnux` is also built for **64-bit Arm Linux** (GH200, DGX Spark). The rest of Jungfraujoch is
Linux-only and x86-64-only. See [Release contents](RELEASE_CONTENTS.md) for the CPU baseline and
CUDA requirements of each released package.
## Software dependencies
+5 -1
View File
@@ -15,9 +15,13 @@ finding/indexing), `scale` (re-scale/merge the already-integrated reflections in
without re-integrating) or `calibration` (detector geometry from a calibrant's powder rings, written
as a `.poni` file). See [rugnux](RUGNUX.md).
`rugnux` installs on its own, as the `rugnux` package or as a standalone archive — see
[rugnux ▸ Installation](RUGNUX.md#installation).
### jfjoch_extract_hkl
Extracts reflections (HKL list) from a Jungfraujoch master file; can sum the same HKL across
neighbouring images and compare against an XDS `INTEGRATE.HKL` reference.
neighbouring images and compare against an XDS `INTEGRATE.HKL` reference. A developer utility: it is
built from source but, like `jfjoch_recompress`, is not installed into any package.
## FPGA / PCIe card management