Every DECTRIS Eiger master links saturation_value, pixel_mask,
bit_depth_readout and serial_number into a companion <prefix>_meta.h5, and
that file is routinely not kept when a dataset is archived or deposited. The
existence test asked only whether the LINK was written, which it is, so every
optional-field guard in the reader answered yes and the read that followed
threw. A deposited Eiger 16M set could not be opened at all, over values the
reader was perfectly prepared to do without.
Exists() now asks the second question too - whether the object the link names
can be reached - so an orphaned link reads as absent and the fallbacks behind
it do their job.
The saturation value is then allowed to be missing outright, because on such a
file it is: neither the NXmx name nor the DECTRIS one is readable, and there is
no third place to look. Left unset, GetSaturationLimit() falls back to the
container's own overflow. That is the safe direction - it can only fail to call
a pixel saturated, where too LOW a value drops the whole reflection and
silently removes the strongest data - and the run says out loud that nothing
will be called saturated.
The set that could not be opened now processes to 2.17 A against a deposited
2.40 A, in the deposited space group, with a cell agreeing to 0.08%. Output is
byte-identical on datasets that already opened.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
(cherry picked from commit 0637979f6d95b446406ab70d1f3982841d195b36)
A valid NXmx master written outside the DECTRIS toolchain could not be opened. Measured on a
Diamond-written master of a 360 deg EIGER 16M sweep, where the images and the meta file are pure
DECTRIS and only the master is third-party - which is why the two sides disagree on units at all.
Five independent things, of which two were silent:
* The image size came from detectorSpecific/x_pixels_in_detector, a DECTRIS extension rather than
NXmx, so a third-party writer has no reason to emit it. It now comes from the image array's own
shape, as it already did for a VDS master.
* Lengths were assumed to be metres and the units attribute was never read. A pixel size, sensor
thickness or distance stated in millimetres - correct NXmx - was silently a factor of a thousand
out. The unit is now read; an undeclared one still means metres, an unknown one is refused.
* The detector distance can sit in NXinstrument rather than in NXdetector; that is now the last
fallback after the NXmx and the firmware-1.x spellings.
* A pixel mask that is an external link into a file not holding it passed the Exists() check and
then threw on the open. Whether the array is there is now decided by opening it.
* Each data file was re-opened and searched for /entry/data/data, ignoring the path the master's
own link names. A master linking to a plain /data therefore found no images at all - and that
was a warning and exit code 0 over a sweep sitting right there, not an error. The link is now
taken at its word, and a master that links to data files but yields no images is an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f
A hybrid master carries an NXmx /entry/sample/transformations group holding one EMPTY SUBGROUP per
axis - the direction as a vector attribute, no NX_class, no units, no angles - beside a legacy
/entry/sample/goniometer group holding all the angles. Opening it failed outright with "Cannot open
HDF5 dataset /entry/sample/transformations/omega": the existing legacy fallback keys off
Exists("/entry/sample/transformations"), which is true here, so it never fired, and the axis stub
was then opened as if it were the angle dataset.
Present is not the same as usable. GoniometerGroup now takes transformations only if it holds at
least one DATASET, and ReadAxis asks IsDataSet rather than Exists, so a member that is not a dataset
can no longer be read as one. HDF5Object gains that predicate, in the style of the neighbouring
Exists.
The stub is the load-bearing half, not merely the thing that crashed. In the legacy branch, before
falling back to the assumed (-1,0,0), the reader now looks for the NXmx stub and takes its stated
vector. With it the axis is (0,-1,0) and the run indexes 60/60 validation frames; with the stub
deleted the assumption applies and the same file indexes 0/60 on both schemes and both signs, and
the run stops with no lattice. So without this half the fix would have turned "cannot open" into
"found no lattice" - a differently shaped failure, not a success. The direction stated here is 90
degrees from the assumption, not merely its negation, which the rotation first pass could have
recovered on its own.
The vector size check moved out of the first branch so it now covers every path that produces one.
Verified: the file processes end to end, 100% of frames indexed, cubic cell 105.87 against a
deposited 105.88 (0.009%), space group reported as I23 or I213 - correctly refusing to choose, since
the reflections that separate them are extinguished by the I-centring and were never measured. A
both-layout master reprocesses unchanged (100% indexed, P212121). [HDF5] passes: 2194 assertions in
91 cases.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW
DECTRIS Eiger firmware 1.x compressed its images with the HDF Group's plain LZ4 filter rather than
bitshuffle, and files from that era are still what a repository hands you. Read-only: nothing here
produces it, and the new enumerator goes last so the existing values do not move, the enum being
part of the CBOR stream.
The framing is the same as bitshuffle's - a 64-bit big-endian total size, a 32-bit big-endian block
size in bytes, then each block prefixed by its 32-bit big-endian compressed size - so only the
unshuffle step differs. Verified on a real chunk: the declared total matched width*height*4 exactly
and the block walk consumed the chunk to the byte. It cannot reuse the bitshuffle path, which
requires the block to be a multiple of BSHUF_BLOCKED_MULT elements: these files put the whole image
in ONE block, which is not.
The unsupported-filter message now names the filter it found and the ones it knows. Before, an
Eiger 1.x file did not report a codec problem at all - it reached spot finding with nothing decoded
and failed as "0 spots from 60 images" then "found no lattice", which reads as a crystallography
failure rather than a format one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.
* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.
**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.
---------
Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144.
* **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future.
* XDS plugin: Improve performance of VDS reading.
* jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD.
* jfjoch_writer: Clean-up of HDF5 routines to better handle issues.
Reviewed-on: #55
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long
* jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata
* XDS plugin: Add generation of Jungfraujoch plugin for XDS
* CI: Add tests with XDS and DIALS (`xia2.ssx`)
Reviewed-on: #43
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release.
* jfjoch_broker: Use newer version of Google Ceres for (potential) CUDA 13 compatibility
* jfjoch_broker: Improve performance of generating preview images, especially for large detectors (9M-16M)
* jfjoch_viewer: Improve performance of displaying images, especially for large detectors (9M-16M)
* jfjoch_viewer: Add more color schemes for better image readability
* HDF5: Common mutex for reading and writing HDF5 if both operations were to happen in the same executable
* HDF5: suppress warning if path (upstream group) doesn't exists when checking if leaf exists
Reviewed-on: #30
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_f
