Two ways the written files could misdescribe themselves without anyone noticing.
The pixel format is stated twice and the two were never compared: each image
carries its own type as a CBOR tag, which is what the data files are written
with, while the master is typed from the start message. A stream whose header
contradicts its images produced data files of one type under a master declaring
another, and with NXmxVDS, HDF5 then converts silently on every read.
HDF5DataFile::CreateFile now checks the two agree and refuses the run otherwise -
the point where the values first meet, so it covers every path into the writer.
Two test fixtures were relying on exactly that inconsistency. The HDF5 writer
tests wrote uint16 buffers under a JUNGFRAU experiment, which converts to photon
counts by default and so declares int16; they never read the pixels back, so it
went unnoticed. The receiver-lite tests feed frames from
compression_benchmark.h5, which really are signed int16, through a DECTRIS
experiment, which declares unsigned by default - the same class of bug the
pixel_signed propagation fixed on the live path. Both now declare what they send.
Second: a virtual dataset whose source file is absent reads as the fill value,
and HDF5 defaults that to zero, so a data file that was not copied alongside the
master is indistinguishable from frames of genuine zero counts. Measured with
DIALS on a four-file set with one file removed: 25 frames of pure zeros, no error
and no warning. The image VDS is now filled with the error marker instead, which
sits outside underload_value..saturation_value, so a reader masks those frames.
Same measurement after the change: -32768 throughout, which DIALS excludes.
Only the images ask for a fill value; the per-image metadata datasets keep the
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144.
* **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future.
* XDS plugin: Improve performance of VDS reading.
* jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD.
* jfjoch_writer: Clean-up of HDF5 routines to better handle issues.
Reviewed-on: #55
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long
* jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata
* XDS plugin: Add generation of Jungfraujoch plugin for XDS
* CI: Add tests with XDS and DIALS (`xia2.ssx`)
Reviewed-on: #43
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release.
* jfjoch_broker: Use newer version of Google Ceres for (potential) CUDA 13 compatibility
* jfjoch_broker: Improve performance of generating preview images, especially for large detectors (9M-16M)
* jfjoch_viewer: Improve performance of displaying images, especially for large detectors (9M-16M)
* jfjoch_viewer: Add more color schemes for better image readability
* HDF5: Common mutex for reading and writing HDF5 if both operations were to happen in the same executable
* HDF5: suppress warning if path (upstream group) doesn't exists when checking if leaf exists
Reviewed-on: #30
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5