20 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 5 8a04773d1d api: record the beam size at the sample, and write it where NXmx puts it
dataset_settings gains beam_size_x_um and beam_size_y_um, the horizontal and
vertical size of the X-ray beam where it meets the sample. They follow the same
route total_flux takes - OpenAPI, DatasetSettings, the CBOR start message, the
HDF5 master, and back out of a stored file - and nothing consumes them; this is
metadata a beamline can state and a downstream program can read.

NXmx puts this in the application definition rather than the base class: not
NXbeam's extent (rank 2, nP x 2, per scan point, always FWHM of a rectangular
aperture) but NXmx's own incident_beam_size, a recommended rank-1 two-element
array in the order x, y. Both are live and neither is deprecated, so the choice
matters; the MX definition wins in an MX file. Written as one array with a
units attribute of "m", like every other length in the master, so the settings
hold micrometres and FillMessage converts once.

The unit table of ReadLength_m becomes LengthUnitFactor so the array read can
share it: a master written elsewhere may state this in millimetres.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
2026-09-02 09:59:34 +02:00
leonarski_fandClaude Opus 5 26fc4b02b3 symmetry: carry the space group as the group, not as its number
The adopted space group travelled the pipeline as a bare int and was rebuilt
downstream with find_spacegroup_by_number, which returns the reference setting.
So every setting a number cannot name was destroyed one line after it was
determined: P 1 1 2 came back as P 1 2 1, I 1 1 2 as C 1 2 1, R 3:R as R 3:H.

DatasetSettings now holds the gemmi::SpaceGroup itself, DiffractionExperiment
exposes it as GetGemmiSpaceGroup() / GetSpaceGroupOrP1(), and everything that
used to take an int - HKLKeyGenerator (its int constructor is gone, so the
compiler finds the callers), the merge, the R-free flags, French-Wilson, the
reindexing ambiguity, the completeness enumeration, the MTZ and mmCIF exports,
the model validation - takes the group. -S keeps the setting the symbol names
rather than reducing it to a number.

The end message carries both spellings and a reader prefers the name, since
only the name keeps the setting while the number is what a reader written
before the name understands. It carries them over CBOR too: the determined
group was never serialised at all, so a group rugnux chose reached the master
file only when the same process wrote it, and an online writer fell back to
whatever the user had supplied at the start. Both keys are optional additions,
so an older reader skips them and a newer one reads an older sender.

On disk the master's /entry/sample/space_group carries the extended
Hermann-Mauguin name and is what the reader takes the group from, so a setting
survives a _process.h5 and the --mode scale that re-reads it; the number stays
beside it and is the fallback for files written before. Every one of the 230
reference settings the old writer could produce reads back as itself, so older
files are unaffected.

Stage A and Stage B of the search still enumerate reference settings only, so
this determines no group differently today - it is what the enumeration needs
before it can be widened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
2026-08-31 07:16:43 +02:00
leonarski_f 6ed4ea541e reader: a link to a file that is not there is not a dataset that exists
Every DECTRIS Eiger master links saturation_value, pixel_mask,
bit_depth_readout and serial_number into a companion <prefix>_meta.h5, and
that file is routinely not kept when a dataset is archived or deposited. The
existence test asked only whether the LINK was written, which it is, so every
optional-field guard in the reader answered yes and the read that followed
threw. A deposited Eiger 16M set could not be opened at all, over values the
reader was perfectly prepared to do without.

Exists() now asks the second question too - whether the object the link names
can be reached - so an orphaned link reads as absent and the fallbacks behind
it do their job.

The saturation value is then allowed to be missing outright, because on such a
file it is: neither the NXmx name nor the DECTRIS one is readable, and there is
no third place to look. Left unset, GetSaturationLimit() falls back to the
container's own overflow. That is the safe direction - it can only fail to call
a pixel saturated, where too LOW a value drops the whole reflection and
silently removes the strongest data - and the run says out loud that nothing
will be called saturated.

The set that could not be opened now processes to 2.17 A against a deposited
2.40 A, in the deposited space group, with a cell agreeing to 0.08%. Output is
byte-identical on datasets that already opened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
(cherry picked from commit 0637979f6d95b446406ab70d1f3982841d195b36)
2026-08-30 21:43:54 +02:00
leonarski_fandClaude Opus 5 a27c4cf26f reader: take a miniCBF's mounting from the imgCIF axis table its header states
A miniCBF header states three things about how the instrument is put together
that the reader was assuming instead: which laboratory direction the image's
columns run along, which its rows run along, and which the spindle turns about.
Some beamlines append a CBF template block holding the full imgCIF axis table,
which says all three outright.

Two instruments in the corpus are not what was assumed, in two different ways.
One mounts its detector a quarter turn round, so the image's columns run
vertically. Another turns its spindle about the VERTICAL, with the image mounted
the usual way; its table says so, and its "# Oscillation_axis" line says so a
second way, by naming the image direction the spindle runs along rather than a
vector. Either error leaves the spindle 90 degrees from the image. That is not a
sign, so the run's axis-sign rescue cannot reach it, and no refinement recovers
it: all three affected sweeps indexed nothing usable.

So the table is read. The element axes give the image orientation, matched against
the eight discrete mountings exactly as the NXmx module directions already are -
the match itself moves to DetectorOrientation, so both readers share one
definition rather than two copies. The goniometer axis with no parent gives the
spindle DIRECTION; its sign stays the rescue's business, which is the part a
convention can legitimately differ on. The detector axis with no parent gives the
2theta arm, replacing the assumption that the arm shares the spindle's axis - the
one header stating both states them with the same vector, so this changes no
answer, only what it rests on. imgCIF's frame differs from the internal one by a
half turn about x, a rotation and not a mirror, as writer/HDF5NXmx.cpp already
records from the other side.

Where a header carries no table, a "+SLOW" on the Oscillation_axis line still
says the spindle runs along the image's slow direction. That is the only thing one
of the three affected sets says about it. The axis NAME on that line stays
unusable - the header that carries both says "X.CW" where its own table says Y -
but the direction token is not: where both are present they agree, which is what
makes reading it evidence rather than a guess.

Also: naming a frame with no directory at all now finds its sweep. parent_path()
of a bare filename is empty and iterating an empty path finds nothing, so running
from inside the data directory reported that no images were found.

Measured, with nothing on the command line. The vertical-spindle protein set goes
from no usable lattice to 100% indexed, P 6(3) 2 2 with a cell 0.43% from
deposited, 87846 reflections at 86.3% completeness and CC(1/2) 0.995. Its
companion from the same detector, which has no table and only the +SLOW token,
goes from a spurious monoclinic cell at 2.3% completeness and I/sigma 0.21 to the
right orthorhombic lattice, 97.7% indexed, 59.7% complete, CC(1/2) 0.996. The
quarter-turned set's three sweeps, at three arm positions, now all index without
the hand-passed quarter turn they needed and agree on one cell to 0.03 A. Six
miniCBF sets that state no table and no +SLOW - including one whose
Oscillation_axis line names an axis in a third dialect - are byte-identical in
.hkl, .mtz, .cif and the image statistics, as are two NXmx sets, which is the
shared orientation matcher moving nothing on that path either.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f
2026-08-30 14:03:32 +02:00
leonarski_fandClaude Opus 5 5c44e544dc reader: place a detector swung out on a 2theta arm where the file says it stands
Chemical crystallography reaches high angle by swinging the detector out on a
2theta arm. Both readers had the number and neither used it: the miniCBF header's
Detector_2theta was parsed into a struct member nothing ever read, and on the NXmx
side the rotation was in the depends_on chain, which was not followed at all. A
sweep taken at 30 degrees was therefore processed with its detector plane 30
degrees from where it stood, and nothing indexed.

The geometry could already express it, and needed no change: the arm turns the
detector about the sample, so the distance is still measured along the detector
normal and the beam centre is still the point of normal incidence - which is
exactly the PONI convention, and a swung detector is one PONI rotation. What moves
is the direct beam, by distance*tan(2theta), off the beam centre and often off the
detector.

NXmx is the harder half, because the swing has no field of its own: it is one
rotation in the chain the detector's position depends on, and "two_theta" is only
one beamline's name for that dataset. So the chain is followed and its rotations
composed, rather than a field of one name being looked for - each transformation
states its vector in the frame of the one it depends on, which is why the product
is the whole placement. Translations are skipped; they are the distance and the
beam centre, which the file states separately in the square-on frame. Vectors come
from McStas through the same 180-degree turn about z the module directions already
use, a proper rotation, so an axis carried through it turns the same way.

The three rotations a file this system writes ARE that chain, and are also read as
the PONI angles - so those three paths are skipped, or every tilted file we have
ever written would come back tilted twice. That is the one way this change could
have broken existing data, and the test for it writes a tilted file and reads it
back.

For miniCBF the arm turns about the base spindle axis: on the four-circle geometry
those headers describe the two are one axis, and the imgCIF axis table such a
header carries states them with the same vector. Both now come from one constant,
so a later correction to the frame moves them together.

Measured. On a swung NXmx sweep the chain gives rot2 = -0.34907 rad for the 20
degrees it states, and the sweep goes from "nothing was integrated" to 25000
reflections at 82.2% completeness and CC(1/2) 0.9993, in the same space group and
the same cell to 0.03 A as the square-on sweep of that crystal; the opposite sign
indexes nothing. A miniCBF sweep at 30 degrees goes the same way, to 0.585 A, and
a second sweep of that crystal at 55 degrees reaches 0.476 A and reproduces the
cell again - with a low-resolution limit of 2.36 A rather than 13 A, which is what
a detector swung that far records. On all of them post-refinement recovers the
header's own beam centre and distance, and the beam stop shadow sits within four
pixels of where the swung geometry puts the direct beam, 417 and 537 pixels from
where the unswung one does. Seven sets whose detector is square to the beam, three
of them carrying a chain whose 2theta is zero, are byte-identical in .hkl, .mtz,
.cif and the image statistics.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f
2026-08-30 12:36:46 +02:00
leonarski_fandClaude Opus 5 c9ca7e424a reader: read a third-party NXmx master, and stop a broken one reading as empty
A valid NXmx master written outside the DECTRIS toolchain could not be opened. Measured on a
Diamond-written master of a 360 deg EIGER 16M sweep, where the images and the meta file are pure
DECTRIS and only the master is third-party - which is why the two sides disagree on units at all.
Five independent things, of which two were silent:

* The image size came from detectorSpecific/x_pixels_in_detector, a DECTRIS extension rather than
  NXmx, so a third-party writer has no reason to emit it. It now comes from the image array's own
  shape, as it already did for a VDS master.
* Lengths were assumed to be metres and the units attribute was never read. A pixel size, sensor
  thickness or distance stated in millimetres - correct NXmx - was silently a factor of a thousand
  out. The unit is now read; an undeclared one still means metres, an unknown one is refused.
* The detector distance can sit in NXinstrument rather than in NXdetector; that is now the last
  fallback after the NXmx and the firmware-1.x spellings.
* A pixel mask that is an external link into a file not holding it passed the Exists() check and
  then threw on the open. Whether the array is there is now decided by opening it.
* Each data file was re-opened and searched for /entry/data/data, ignoring the path the master's
  own link names. A master linking to a plain /data therefore found no images at all - and that
  was a warning and exit code 0 over a sweep sitting right there, not an error. The link is now
  taken at its word, and a master that links to data files but yields no images is an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f
2026-08-30 08:27:53 +02:00
leonarski_fandClaude Opus 5 08f57235d6 reader: take the image orientation from the module directions the NXmx file states
NXmx says how the stored image sits in the detector plane, in the NXdetector_module
fast_pixel_direction and slow_pixel_direction vectors. rugnux read neither, and
assumed every detector was mounted the way this system mounts its own. A facility
that bolts its detector a quarter turn round therefore produced a geometry that
was wrong by 90 degrees, which no amount of refinement recovers: the run indexed
nothing and reported that it could find no lattice.

The two vectors are read, taken from McStas into the internal frame, and matched
against the eight discrete image orientations. An exact match is adopted; anything
else is left alone, because an orientation that is not discrete is a continuous
rotation of the detector in its own plane and cannot be told apart from the tilt by
looking at the module. Every file examined here is exactly discrete.

Measured over the 94-dataset battery: 78 masters state no module vectors at all and
12 state the standard ones, so the change can reach exactly 4. Three of those go
from "Nothing was integrated" to a complete merged result with no flags -
43/93/92% indexed, CC(1/2) 0.957/0.984/0.998 - and are the sets that until now
needed a hand-passed --rot3 pi/2. The fourth states a half turn, which the existing
rotation-axis sign rescue already absorbed, and is unchanged to every digit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW
2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5 fed077e683 geometry: hold the detector plane as axis vectors, and give the discrete part its own home
The detector plane was three PONI angles and nothing else, so the two things it
cannot express - an image mirrored in Y, and one mounted at a multiple of 90
degrees - had no home at all. They are now the DetectorOrientation carried by the
detector setup, composed with the PONI rotation into one orthogonal matrix whose
columns ARE the fast axis, the slow axis and the sample->PONI normal:

    lab = R(rot1, rot2, rot3) * Delta * ( (x-bx)*p , (y-by)*p , distance )

GetFastAxis/GetSlowAxis/GetNormalAxis read those columns and DetectorAxes() sets
the plane from them, decomposing back to the angles; PoniRotMatrix and
PoniAnglesFromMatrix are the conversion in both directions, exact on the canonical
branch (rot2 in [-pi/2, pi/2]) and with a stated convention at gimbal lock. The
angles stay stored rather than re-derived, so a geometry given as angles is
written back as the same angles, to the bit.

Delta is never inferred. In particular an arbitrary rot3 is NOT decomposed into a
quarter turn plus a residual: rot3 is a fitted quantity, and a least-squares step
must not be able to turn the stored image. It is set only where something states
it - the detector setup, --detector-mirror-y / --detector-quarter-turns, or the
value a file this system wrote records - and defaults to the identity, which makes
the whole change a no-op for every existing detector and every existing file.

It is a different setting from DetectorSetup::mirror_y, which flips the MODULE
LAYOUT while an image is assembled and so decides what the stored pixels are.
Merging the two would apply the mirror twice for every modular detector, or change
the pixel content of every file written; both are ruled out. The new one earns its
keep exactly where the old one is a no-op: a detector whose image arrives already
assembled has no layout to flip.

Both generators are signed permutations of the in-plane offset, so they preserve
the distance from the PONI. That is why almost nothing downstream changes:
everything needing an azimuth already goes through LabCoord, and everything that
does not needs only a radius. The two hand-written copies of the rotation -
XtalResidual and RingOptimizer - take the discrete part as four constants next to
cos_rot3/sin_rot3, since it acts in the detector frame where rot3 acts in the
laboratory and cannot be folded into it. RingOptimizer needs it despite being a
radial fit: it fits the tilt, and the discrete part changes which way the tilt
tips a ring.

Carried as two optional CBOR keys and two detectorSpecific datasets, both
back-compatible; the NXmx module axis vectors and the translation direction stop
being hardcoded and are computed from it, reproducing today's values exactly at
the identity. GetPoniRotMatrix is renamed GetDetectorMatrix, because it is no
longer only the PONI rotation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW
2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5 60c15d6931 reader: open a master whose beamline writes standards-correct NXmx differently
Four defects, hit in sequence, that between them stopped eight masters from one beamline before any
geometry question was reached. The files are correct NeXus; the reader was assuming one writer's
conventions.

ReadScalar demanded rank 0. NXmx puts no rank on distance, saturation_value, two_theta or det_z, and
these files write them as shape (1,). Any dataspace holding exactly one element is now accepted; a
genuine vector is still refused.

frame_time was read unconditionally and NXmx does not require it.

A virtual-dataset source filename of "." was resolved as a relative path, giving <dir>/. - but "."
is HDF5's spelling for THIS file, and these masters compose /entry/data/data as a virtual dataset
over datasets in themselves that are external links to the data files.

The virtual source's DATASET PATH was parsed and then ignored in favour of a hardcoded
/entry/data/data, while these data files keep their images at the root. Fixing that exposed a fifth:
the positional-read fast path used the master's own path, but a dataset reached through an external
link lives in another file and a chunk address is an offset into THAT file.

An audit of all 38 masters in the non-SLS corpus finds exactly these eight need the change and the
other 30 need nothing. Corpus A/B over 73 dataset pairs, base and patched back to back: 64
byte-identical results, 9 identical failures, none differing. The blast radius is bounded by
construction - JFJochReader is linked only by rugnux and jfjoch_viewer, and the one shared header
this touches only ever accepts more, so nothing that opened before can read differently.

With it the eight files read completely: 12850 images, no decode errors, seven of them spanning two
source datasets through the master's own external links.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW
2026-08-29 19:58:33 +02:00
leonarski_fandClaude Opus 5 1ea6ba25be reader: open a master that states its goniometer axes in one place and their angles in another
A hybrid master carries an NXmx /entry/sample/transformations group holding one EMPTY SUBGROUP per
axis - the direction as a vector attribute, no NX_class, no units, no angles - beside a legacy
/entry/sample/goniometer group holding all the angles. Opening it failed outright with "Cannot open
HDF5 dataset /entry/sample/transformations/omega": the existing legacy fallback keys off
Exists("/entry/sample/transformations"), which is true here, so it never fired, and the axis stub
was then opened as if it were the angle dataset.

Present is not the same as usable. GoniometerGroup now takes transformations only if it holds at
least one DATASET, and ReadAxis asks IsDataSet rather than Exists, so a member that is not a dataset
can no longer be read as one. HDF5Object gains that predicate, in the style of the neighbouring
Exists.

The stub is the load-bearing half, not merely the thing that crashed. In the legacy branch, before
falling back to the assumed (-1,0,0), the reader now looks for the NXmx stub and takes its stated
vector. With it the axis is (0,-1,0) and the run indexes 60/60 validation frames; with the stub
deleted the assumption applies and the same file indexes 0/60 on both schemes and both signs, and
the run stops with no lattice. So without this half the fix would have turned "cannot open" into
"found no lattice" - a differently shaped failure, not a success. The direction stated here is 90
degrees from the assumption, not merely its negation, which the rotation first pass could have
recovered on its own.

The vector size check moved out of the first branch so it now covers every path that produces one.

Verified: the file processes end to end, 100% of frames indexed, cubic cell 105.87 against a
deposited 105.88 (0.009%), space group reported as I23 or I213 - correctly refusing to choose, since
the reflections that separate them are extinguished by the I-centring and were never measured. A
both-layout master reprocesses unchanged (100% indexed, P212121). [HDF5] passes: 2194 assertions in
91 cases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW
2026-08-29 15:47:41 +02:00
leonarski_fandClaude Opus 5 fa4aa30a1e reader: match the legacy companion suffixes exactly, not any underscore
The legacy DECTRIS layout stores five companion scalars beside each goniometer
axis - AXIS_end, _start, _increment, _range_average, _range_total - and only the
bare name is the axis itself. The first cut rejected _end and _range, which left
_start and _increment to be offered to ReadAxis as axes in their own right.

Matching the full suffix set, anchored at the end of the name, also keeps a
genuine two_theta axis readable, which a "contains an underscore" test would not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5 df3696e6e2 reader: fall back to the pre-NXmx key names, so an Eiger 1.x master opens
Firmware 1.x writes the same three values under different names. Try the NXmx spelling first and
the old one only if it is absent:

    detector/distance         <- detector/detector_distance
    detector/saturation_value <- detectorSpecific/countrate_correction_count_cutoff
    sample/transformations    <- sample/goniometer

This cannot change what a current file reads: every modern Eiger master carries BOTH spellings.
Measured on thirteen masters from ten facilities, firmware release-2020.2.1 through
release-2024.1.1 - all of them write detector_distance and countrate_correction_count_cutoff beside
the NXmx names, and a goniometer group beside the transformations one.

The goniometer is the one that matters. A goniometer is only ever set from that one group, so a
file whose axes are somewhere else was not an error - it was read as STILLS, silently, and the run
completed with the wrong answer. The old layout also tags no axis with transformation_type and
gives no vector, both of which ReadAxis required, so absence now means two different things by
layout: in a transformations group it still means "not an axis" (that is how AXISNAME_end and the
width scalars are skipped), while in the legacy group every leaf IS an axis and the companions are
recognised by name instead. A missing direction defaults to the one every DECTRIS master since has
written and says so in a warning rather than assuming it silently; a wrong guess there does not
index, so it is visible, and the rotation first pass tries the opposite sign anyway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 21:56:12 +02:00
leonarski_f 749db470ca v1.0.0-rc.164 (#74)
Build Packages / build:rpm (rocky9) (push) Successful in 19m56s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m57s
Build Packages / build:windows:cuda (push) Successful in 19m18s
Build Packages / build:viewer-tgz:cpu (push) Successful in 14m48s
Build Packages / build:viewer-tgz:cuda (push) Successful in 16m18s
Build Packages / build:rugnux-tgz (x86_64) (push) Successful in 14m19s
Build Packages / build:rugnux:windows (push) Successful in 10m34s
Build Packages / build:rugnux:aarch64 (cross) (push) Successful in 8m49s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 20m55s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 17m4s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 20m48s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 19m15s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 24m26s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 20m32s
Build Packages / build:rpm (rocky8) (push) Successful in 23m39s
Build Packages / Generate python client (push) Successful in 46s
Build Packages / Build documentation (push) Successful in 1m45s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (durin plugin) (push) Successful in 11m3s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 11m30s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 20m10s
Build Packages / XDS test (neggia plugin) (push) Successful in 10m17s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 23m12s
Build Packages / DIALS test (push) Successful in 20m12s
* rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction.
* rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are.
* rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log.
* rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4.
* rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning.
* rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities.
* rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present.
* rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file.
* rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware.
* Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere.
* Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged.
* The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot.
* The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel.
* A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave.
* A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted.
* A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires.
* `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent.
* The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged.
* The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count.
* `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible.
* `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so.
* The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents.

Reviewed-on: #74
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-26 22:47:00 +02:00
leonarski_fandjungfrau 4dc2534dbf v1.0.0.rc-162 (#72)
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-25 08:21:39 +02:00
leonarski_f 538f3504d3 v1.0.0.rc-161 (#71)
Build Packages / build:windows:nocuda (push) Successful in 20m4s
Build Packages / Unit tests (push) Skipped
Build Packages / build:viewer-tgz:cpu (push) Successful in 16m5s
Build Packages / build:viewer-tgz:cuda (push) Successful in 17m26s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 27m46s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m17s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 26m13s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 23m17s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 28m11s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m30s
Build Packages / build:rpm (rocky8) (push) Successful in 24m34s
Build Packages / build:rpm (rocky9) (push) Successful in 21m30s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 23m33s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 20m18s
Build Packages / DIALS test (push) Successful in 18m23s
Build Packages / XDS test (durin plugin) (push) Successful in 11m30s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m2s
Build Packages / Generate python client (push) Successful in 49s
Build Packages / Build documentation (push) Successful in 1m21s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 29m45s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.

**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.

**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.

**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**

Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-13 17:03:10 +02:00
leonarski_f 67dca388bd v1.0.0-rc.160 (#70)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 18m44s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m54s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m40s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m10s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 11m30s
Build Packages / build:rpm (rocky9) (push) Successful in 12m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m8s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 13m22s
Build Packages / XDS test (durin plugin) (push) Successful in 9m2s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m55s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m57s
Build Packages / Generate python client (push) Successful in 23s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 10m24s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-19 09:39:28 +02:00
leonarski_f 451310f43d v1.0.0-rc.158 (#68)
Build Packages / Unit tests (push) Successful in 1h32m35s
Build Packages / build:windows:cuda (push) Successful in 18m0s
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m37s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m55s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m13s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m11s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m35s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m57s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m23s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m45s
Build Packages / build:rpm (rocky8) (push) Successful in 11m39s
Build Packages / build:rpm (rocky9) (push) Successful in 14m0s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m42s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m38s
Build Packages / DIALS test (push) Successful in 14m55s
Build Packages / XDS test (durin plugin) (push) Successful in 7m11s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m7s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m34s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 9m55s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation.
* jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window.
* rugnux: Accept an absolute `-o` output prefix in offline processing.
* Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed).
* Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-12 19:42:29 +02:00
leonarski_f 54c0100e8e v1.0.0-rc.157 (#67)
Build Packages / Unit tests (push) Successful in 1h28m28s
Build Packages / build:windows:nocuda (push) Successful in 14m45s
Build Packages / build:windows:cuda (push) Successful in 13m13s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m47s
Build Packages / build:viewer-tgz:cuda (push) Successful in 7m22s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m52s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m16s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m19s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m50s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m40s
Build Packages / build:rpm (rocky8) (push) Successful in 11m18s
Build Packages / build:rpm (rocky9) (push) Successful in 12m4s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m55s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m22s
Build Packages / DIALS test (push) Successful in 13m37s
Build Packages / XDS test (durin plugin) (push) Successful in 8m47s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m4s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m45s
Build Packages / Generate python client (push) Successful in 34s
Build Packages / Build documentation (push) Successful in 1m4s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 7m16s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`.
* rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling.
* rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics.
* rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1.
* rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group.
* rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix.
* jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation.
* jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning.
* jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint.
* jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed).
* jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types.
* jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails.
* jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak.
* Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation.
* CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-11 07:19:11 +02:00
leonarski_f d6389e12da v1.0.0-rc.156 (#66)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 15m31s
Build Packages / build:viewer-tgz:cpu (push) Successful in 5m46s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m9s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m25s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m21s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m41s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m18s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m26s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m33s
Build Packages / build:rpm (rocky8) (push) Successful in 10m32s
Build Packages / build:rpm (rocky9) (push) Successful in 12m23s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m12s
Build Packages / DIALS test (push) Successful in 12m6s
Build Packages / XDS test (durin plugin) (push) Successful in 8m15s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m12s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m35s
Build Packages / Generate python client (push) Successful in 27s
Build Packages / Build documentation (push) Successful in 54s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 12m37s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-03 19:18:56 +02:00
leonarski_f 75e401f0e5 v1.0.0-rc.153 (#63)
Build Packages / Unit tests (push) Successful in 1h31m59s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 8m43s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m5s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m27s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m24s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 10m27s
Build Packages / build:rpm (rocky8) (push) Successful in 9m20s
Build Packages / build:rpm (rocky9) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m54s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m38s
Build Packages / DIALS test (push) Successful in 12m13s
Build Packages / XDS test (durin plugin) (push) Successful in 7m8s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m8s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m50s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Build documentation (push) Successful in 50s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results

Reviewed-on: #63
2026-06-23 20:29:49 +02:00