Two ways the written files could misdescribe themselves without anyone noticing.
The pixel format is stated twice and the two were never compared: each image
carries its own type as a CBOR tag, which is what the data files are written
with, while the master is typed from the start message. A stream whose header
contradicts its images produced data files of one type under a master declaring
another, and with NXmxVDS, HDF5 then converts silently on every read.
HDF5DataFile::CreateFile now checks the two agree and refuses the run otherwise -
the point where the values first meet, so it covers every path into the writer.
Two test fixtures were relying on exactly that inconsistency. The HDF5 writer
tests wrote uint16 buffers under a JUNGFRAU experiment, which converts to photon
counts by default and so declares int16; they never read the pixels back, so it
went unnoticed. The receiver-lite tests feed frames from
compression_benchmark.h5, which really are signed int16, through a DECTRIS
experiment, which declares unsigned by default - the same class of bug the
pixel_signed propagation fixed on the live path. Both now declare what they send.
Second: a virtual dataset whose source file is absent reads as the fill value,
and HDF5 defaults that to zero, so a data file that was not copied alongside the
master is indistinguishable from frames of genuine zero counts. Measured with
DIALS on a four-file set with one file removed: 25 frames of pure zeros, no error
and no warning. The image VDS is now filled with the error marker instead, which
sits outside underload_value..saturation_value, so a reader masks those frames.
Same measurement after the change: -32768 throughout, which DIALS excludes.
Only the images ask for a fill value; the per-image metadata datasets keep the
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long
* jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata
* XDS plugin: Add generation of Jungfraujoch plugin for XDS
* CI: Add tests with XDS and DIALS (`xia2.ssx`)
Reviewed-on: #43
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files)
* jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction
* jfjoch_broker: Image pusher statistics are accessible via the REST interface
* jfjoch_writer: Supports TCP image socket and for these auto-forking option
Reviewed-on: #36
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
* Writer can send a PUB message with information on closed file
* Writer uses getopt to format input parameters
* DiffractionExperiment: Add frame number limit
leonarski_fandClaude Opus 5