Replaces the start-message TransformationAxis of the previous commit, which was
the wrong shape in two ways.
DetectorTransformation (common/) mirrors a NeXus NXtransformations axis and holds
nothing else: name, type, units, vector, offset, depends_on and the positions
themselves. Deliberately without cleverness - the values are either a single
number for an axis that does not move or one per image, and nothing derives a
position from a start and an increment. That is the point: a producer will later
want to report where a stage actually WENT rather than where it was told to go,
and a structure that stores start+increment cannot express that. A million images
cost 4 MB per axis, which is not a reason to be clever.
Hence also the move to the END message: measured positions are only known once
the run is over.
And hence no metadata version bump, which the previous commit did make. The chain
is optional; when it is absent the writer builds the identical chain from the
start message, exactly as before. Nothing on the wire changes for a producer that
does not send it, so a broker and a writer of different releases still interwork -
the constraint the previous version stated is withdrawn.
The writer transcribes a chain it is given, without recomputing an angle, which
is what makes measured positions possible end to end.
JFJochReader_TransformationChain_SentAndBuilt writes the same run both ways and
checks the two files read back the same, chi/phi included.
CBORSerialize_End_Transformations covers the wire 1:1, asserting the order
survives and that a moving axis keeps one value per image while a stationary one
keeps a single value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chain could not be expressed. A goniometer axis, the Smargon chi/phi and a
grid stage each travelled by a different route - the `goniometer` map, a private
JSON key inside user_data, and `grid_scan` - and nothing said what order they are
mounted in. The order cannot go in the `goniometer` map either: that is a DECTRIS
stream2 field, and RFC 8949 requires deterministic encoders to SORT map keys, so
a map's order is not something a consumer may rely on.
So `transformations` is sent as an ordered ARRAY, base first, each element
carrying name, type, axis vector and angles. The `goniometer` map and `grid_scan`
are still emitted beside it, unchanged, for consumers that only know stream2 -
nothing vendor-defined is mutated, and a stream2 consumer sees exactly what it
saw before.
Smargon chi and phi are ordinary stationary axes that were only ever separate for
historical reasons, and they now appear in the chain like any other. They are
also read back: reader/ had no smargon support at all, so re-opening a file lost
the head position silently. Combined with the earlier change that writes them for
a still rather than only alongside a rotation or a grid scan, the round trip is
now closed.
Metadata version 7. A broker and a writer from different releases must not be
mixed across this: an older writer ignores the chain and reads the unordered map,
so anything whose order matters - a Smargon position, or a grid scan combined
with a rotation - is not reproduced. Said so in docs/CBOR.md and the changelog.
CBORSerialize_Start_Transformations asserts the ORDER survives, not just the
contents, which is the whole point of the array.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Which way the detector's rows run was decided once, in the module assembly, and
never stated again: not on the wire, not in the file, nowhere a consumer could
read it. mirror_y was consumed inside the DetectorGeometryModular constructor and
discarded. It is now a declared property of the detector setup, carried into the
start message, written to HDF5 under detectorSpecific, and read back. Absence
means true, which is the MX convention and the only thing Jungfraujoch has ever
produced.
Deliberately a boolean and not a corner enum: the assembled image can only be
flipped in Y, so a four-corner value would encode states that cannot occur.
DECTRIS stream2 has no field for this - checked against the specification - so
the key is new rather than an extension of theirs, and a consumer that does not
know it skips it and behaves exactly as before.
The .poni file gains pyFAI's orientation. Without it pyFAI applies its own
default, 3 (bottom left), and believes increasing row means physically upwards.
The numbers still agreed - a mirror preserves 2theta, so radial integration was
never affected - but the azimuth came out with the opposite sense, which matters
for cake and sector integration.
Declaring orientation 2 is not a one-line addition: it re-anchors Poni1 to the
top edge and reverses rot2 and rot3, a row flip being improper. Measured against
pyFAI 2026.5.0 by searching all four orientations, both Poni1 anchorings and all
eight sign combinations: exactly two combinations reproduce the lab position
DiffractionGeometry computes to 1.4e-17 m - the unlabelled form written before,
and (orientation 2, Poni1 = height-1-beam_y, +rot1/+rot2/-rot3), which is now
written. Calibration_PoniFileAxisConvention pins it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
They are not alternatives: a grid is usually collected at a particular head
position, so an axis and a grid describe different parts of the same setup. The
exclusion was enforced independently in four places - the API converter, the CBOR
serializer, the writer and the reader - and each silently dropped the grid scan
when an axis was present. Nothing warned.
The writer now builds one chain from the base outwards, spindle -> chi -> phi ->
helical -> grid translations, instead of two branches. NXmx applies the deepest
dependency first, so the sample ends up innermost, which is what it physically is:
the grid stage rides on the head and the head rides on the spindle. The grid
translations consequently move inside the rotation - identical to before at
omega = 0, and right rather than wrong when it is not.
A grid scan with no axis at all now writes a stationary omega. NXmx has no way to
say "there is no rotation", and a sample chain of translations alone is not
something readers accept: dxtbx raises outright on it, so every grid-scan master
we have written so far cannot be opened by DIALS. Measured on a file matching the
new chain: dials.import reads it. At 0 degrees the rotation is the identity
whatever the axis points along, so the conventional vector carries no geometric
claim - it only has to be well formed.
The API change is deliberately not breaking: no field changes type or
cardinality, only the prose saying the two were exclusive, and a request that set
both used to lose one silently and now does not.
JFJochReader_GridScan asserted the absence of a goniometer; it now asserts the
axis is present and stationary, which is the contract that matters - a grid scan
must not read back as a sweep.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
NXmx has no field for the depth of the stored image - only bit_depth_readout,
"how many bits the electronics record per pixel". The two diverge exactly when
summation is used: the readout keeps the detector's native width while the
summed image must be wider to hold the sum.
Every NXmx reader nonetheless takes bit_depth_readout as the width of the stored
pixel. dxtbx ignores the non-standard bit_depth_image entirely for a generic
NXmx file, derives its masking markers from bit_depth_readout, and raises
"Unsupported integer dtype uint32" for a 32-bit image when the field is absent.
Reporting the electronic value there would mislead precisely where it differs.
So report the image depth in both fields, and drop the machinery that existed to
carry the electronic one for a DECTRIS detector: the SIMPLON read, the
DetectorSetup setter, and the receiver-side propagation of a key that the DECTRIS
stream2 protocol does not even define. JUNGFRAU and PSI EIGER keep their readout
depth, which the FPGA acquisition genuinely needs.
Also write NXmx underload_value, the lowest valid value. Without it a reader
takes the trusted minimum to be -0x7FFFFFFF, so the error-pixel marker sits
inside the trusted range and is consumed as an intensity. Measured with DIALS
3.27 on a written file: trusted_range goes from (-2147483647, 32766) to
(-32767, 32766), so the INT16_MIN gap pixels are now masked.
Third fix in the same area: JFJochReceiverLite::Configure took the image width
from the incoming stream but not the sign, while the image itself is forwarded
byte-for-byte. A detector sending int32 was re-declared uint32, and the VDS
master was typed unsigned over signed data files. Take pixel_signed from the
stream too - it and the width are both carried by the one image_dtype key.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.
**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.
**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.
**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**
Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Better track time for each operation in the processing stack
* jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress)
* jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress)
* jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0)
Reviewed-on: #46
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Default EIGER readout time is 20 microseconds
* jfjoch_broker: Multiple improvements regarding performance
* jfjoch_broker: Image buffer allows to track frames in preparation and sending
* jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer
* jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets
* jfjoch_writer: Fixes regarding properly closing files in long data collections
* jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code
Reviewed-on: #34
This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis.
If things go wrong with analysis, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Improve logic on switching on/off spot finding
* jfjoch_broker: Increase maximum spot count for FFBIDX to 65536
* jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD)
* jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis
* jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection
* jfjoch_writer: Save more information about each reflection
Reviewed-on: #32
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them,
especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123.
* jfjoch_broker: Improve refection position prediction and Bragg integration code.
* jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation.
* jfjoch_writer: Fix saving mosaicity properly in HDF5 file.
* jfjoch_viewer: Introduce high-dynamic range mode for images
* jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%)
* jfjoch_viewer: Zoom-in numbers have better readability
Reviewed-on: #31
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Add thresholding to prefer shorter vectors after FFT
* jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress)
* jfjoch_viewer: Display file opening errors
* jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available
Reviewed-on: #29
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.111 instead).
* jfjoch_broker: Improve handling of rotation indexing
* jfjoch_broker: More information saved in CBOR end message (WIP)
* jfjoch_writer: Save rotation indexing lattice parameters and Niggli class
* jfjoch_viewer: Remove (for now) primitive cell information
Reviewed-on: #19
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_viewer: Minor polishing of new functionality
* jfjoch_broker: User NaN for empty azimuthal bins
Reviewed-on: #12
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5