The limit is EXCLUSIVE inside Jungfraujoch - the first value that is no longer a count - and NXmx
saturation_value is INCLUSIVE, the highest value that still is one. XDS OVERLOAD and the DIALS
trusted_range read it inclusively too. The write side subtracted the count and no read side added it
back, so the value fell by one on every write-read-write cycle, unbounded: four chained runs over one
dataset gave 32766, 32765, 32764, 32763. It also fed the preprocessor, so one more real count was
called saturated after each cycle.
SaturationValueFromLimit / SaturationLimitFromValue now carry the conversion, used by the writer and
by all three readers (HDF5, the lite receiver, the viewer). JFJochReaderImage's summation test moves
from > to >= in the same commit: it was silently compensating for the missing count, and correcting
one without the other would have shifted it instead.
Two more places said the wrong thing about the same pixels:
error_value was GetUnderflow(), which is -1 for an unsigned image - a value no unsigned pixel can
hold. The marker those images really carry is UINTx_MAX, and GetImageFillValue() already returned it,
so the class held two disagreeing definitions of one marker.
bit_depth_readout is now written for unsigned images only. DIALS remaps the top two codes of
2^bit_depth_readout to -1 and -2 whenever the field is present, without looking at the pixel type.
For an unsigned image those fall below underload_value and are masked, which is what we want. For a
signed one they land INSIDE the trusted range, so a saturated pixel reached DIALS as a trusted count
of -2 - on the strongest reflections. Verified with DIALS 3.27: an int32 file now masks both
sentinels. The field stays where it earns its keep, since dxtbx cannot read unsigned 32-bit without
it.
Neither the values themselves nor the wire format change. Verified against NXmx, DECTRIS SIMPLON,
Durin (Global Phasing fork), XDS and DIALS 3.27; the chained run now holds at 32766.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z
Replaces the start-message TransformationAxis of the previous commit, which was
the wrong shape in two ways.
DetectorTransformation (common/) mirrors a NeXus NXtransformations axis and holds
nothing else: name, type, units, vector, offset, depends_on and the positions
themselves. Deliberately without cleverness - the values are either a single
number for an axis that does not move or one per image, and nothing derives a
position from a start and an increment. That is the point: a producer will later
want to report where a stage actually WENT rather than where it was told to go,
and a structure that stores start+increment cannot express that. A million images
cost 4 MB per axis, which is not a reason to be clever.
Hence also the move to the END message: measured positions are only known once
the run is over.
And hence no metadata version bump, which the previous commit did make. The chain
is optional; when it is absent the writer builds the identical chain from the
start message, exactly as before. Nothing on the wire changes for a producer that
does not send it, so a broker and a writer of different releases still interwork -
the constraint the previous version stated is withdrawn.
The writer transcribes a chain it is given, without recomputing an angle, which
is what makes measured positions possible end to end.
JFJochReader_TransformationChain_SentAndBuilt writes the same run both ways and
checks the two files read back the same, chi/phi included.
CBORSerialize_End_Transformations covers the wire 1:1, asserting the order
survives and that a moving axis keeps one value per image while a stationary one
keeps a single value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chain could not be expressed. A goniometer axis, the Smargon chi/phi and a
grid stage each travelled by a different route - the `goniometer` map, a private
JSON key inside user_data, and `grid_scan` - and nothing said what order they are
mounted in. The order cannot go in the `goniometer` map either: that is a DECTRIS
stream2 field, and RFC 8949 requires deterministic encoders to SORT map keys, so
a map's order is not something a consumer may rely on.
So `transformations` is sent as an ordered ARRAY, base first, each element
carrying name, type, axis vector and angles. The `goniometer` map and `grid_scan`
are still emitted beside it, unchanged, for consumers that only know stream2 -
nothing vendor-defined is mutated, and a stream2 consumer sees exactly what it
saw before.
Smargon chi and phi are ordinary stationary axes that were only ever separate for
historical reasons, and they now appear in the chain like any other. They are
also read back: reader/ had no smargon support at all, so re-opening a file lost
the head position silently. Combined with the earlier change that writes them for
a still rather than only alongside a rotation or a grid scan, the round trip is
now closed.
Metadata version 7. A broker and a writer from different releases must not be
mixed across this: an older writer ignores the chain and reads the unordered map,
so anything whose order matters - a Smargon position, or a grid scan combined
with a rotation - is not reproduced. Said so in docs/CBOR.md and the changelog.
CBORSerialize_Start_Transformations asserts the ORDER survives, not just the
contents, which is the whole point of the array.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Which way the detector's rows run was decided once, in the module assembly, and
never stated again: not on the wire, not in the file, nowhere a consumer could
read it. mirror_y was consumed inside the DetectorGeometryModular constructor and
discarded. It is now a declared property of the detector setup, carried into the
start message, written to HDF5 under detectorSpecific, and read back. Absence
means true, which is the MX convention and the only thing Jungfraujoch has ever
produced.
Deliberately a boolean and not a corner enum: the assembled image can only be
flipped in Y, so a four-corner value would encode states that cannot occur.
DECTRIS stream2 has no field for this - checked against the specification - so
the key is new rather than an extension of theirs, and a consumer that does not
know it skips it and behaves exactly as before.
The .poni file gains pyFAI's orientation. Without it pyFAI applies its own
default, 3 (bottom left), and believes increasing row means physically upwards.
The numbers still agreed - a mirror preserves 2theta, so radial integration was
never affected - but the azimuth came out with the opposite sense, which matters
for cake and sector integration.
Declaring orientation 2 is not a one-line addition: it re-anchors Poni1 to the
top edge and reverses rot2 and rot3, a row flip being improper. Measured against
pyFAI 2026.5.0 by searching all four orientations, both Poni1 anchorings and all
eight sign combinations: exactly two combinations reproduce the lab position
DiffractionGeometry computes to 1.4e-17 m - the unlabelled form written before,
and (orientation 2, Poni1 = height-1-beam_y, +rot1/+rot2/-rot3), which is now
written. Calibration_PoniFileAxisConvention pins it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The value means three different things at once. It is the unit of writer
parallelism - whole files go round-robin to the writers,
(image_number / images_per_file) % socket.size() in ZMQStream2Pusher::SendImage
and TCPStreamPusher - it multiplies writer memory linearly, since every data-file
plugin reserves per file, and it decides whether a legacy master is readable at
all, because dxtbx follows only the first data file of one.
That last point is what makes a flat default wrong. Measured with DIALS on a
2500-image rotation sweep written as legacy: split into five files it reports
2500 images and then raises IndexError beyond image 499, so it half-works
silently; in one file all 2500 read. AutoPROC does not read VDS, so legacy has to
stay the default, which leaves the file count as the only lever.
So make it optional and resolve it from the acquisition. A rotation sweep of at
most 20000 images goes into one data file - rotation datasets are small enough,
and one writer keeps up with them. A grid scan splits on whole fast-axis rows, so
a file is a meaningful piece of the grid. Stills and serial keep 1000, where the
image count far exceeds it and the parallelism and the bounded writer memory are
what matter. An explicit value is always taken literally.
GetImagesPerFile is the single place this is resolved, and it must always return
a fixed non-zero number, because everything downstream - receiver, pusher, puller,
writer - requires one. That was already true of the old 0 = "one file" spelling;
0 is now gone from the API and omitting the field says the same thing better.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.
**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.
**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.
**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**
Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation.
* jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window.
* rugnux: Accept an absolute `-o` output prefix in offline processing.
* Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed).
* Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files)
* jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction
* jfjoch_broker: Image pusher statistics are accessible via the REST interface
* jfjoch_writer: Supports TCP image socket and for these auto-forking option
Reviewed-on: #36
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them,
especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123.
* jfjoch_broker: Improve refection position prediction and Bragg integration code.
* jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation.
* jfjoch_writer: Fix saving mosaicity properly in HDF5 file.
* jfjoch_viewer: Introduce high-dynamic range mode for images
* jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%)
* jfjoch_viewer: Zoom-in numbers have better readability
Reviewed-on: #31
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.96 instead).
* jfjoch_broker: For DECTRIS detectors add dark data collection during initialization for bad pixel mask
* jfjoch_broker: Refactor of calibration logic for more clear code (likely to introduce problems)
* jfjoch_viewer: Add option to handle user pixel mask (experimental)
* jfjoch_viewer: More options for ROI
* jfjoch_viewer: Add window to display calibration
Reviewed-on: #2
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5