Completes the rename the C++ identifiers already carried. Producers now emit ice_ring_ratio /
ice_ring_ratio_mean over CBOR and write /entry/MX/iceRingRatio and iceRingRatioMean; both readers
accept the retired ice_ring_score / iceRingScore spellings as well, so nothing that exists on disk
or on the wire stops being readable. The REST plot_type gains ice_ring_ratio and KEEPS
ice_ring_score, both mapping to the same plot, so no client breaks. --ice-min-score is deliberately
unchanged: it is a threshold on the ratio, it is the one surface a user types, and this program has
91 long options and no aliases.
The fallback is not a formality. rugnux --mode scale reads the stored per-image value to reproduce
the ice gate the writing run applied, and an unread dataset does not fail loudly - ice_n == 0 sends
the gate to its fail-safe branch, "ice present". Demonstrated on a clean crystal whose stored file
uses the old names: with the fallback the gate reads 1.06 against a 1.50 threshold and skips ice
handling; with the fallback removed the same file has 272056 of 1032493 reflections (26%) excluded
from the scale fit, and ISa, R_meas, I/sigma, SIGANO and both error-model terms all move. That is a
silent change to merged intensities on files already written, which is why the two new CBOR test
cases were each checked to FAIL when their fallback is taken out rather than merely to pass.
Verified: same binary on an old-name and a new-name copy of one file gives byte-identical .hkl,
.mtz and unmerged .mtz and an identical report; a file written after the rename scales to the same
bytes again; the master and the data-file read paths were each exercised with both spellings; and
the live API serves the same plot under either name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
Two things, both about telling one ice quantity from another.
The ice score's spot channel had its own band half-width of 0.02 A^-1 while the spot finder marks
ice rings at 0.03 (ice_ring_width_Q_recipA). The 0.02 was justified by a 5 pp specificity gain
measured on the PYTHON PROTOTYPE, which used a fitted beam centre and a mask-derived coverage table;
the shipped port, which takes the geometry's centre and the azimuthal profile's own live pixel
count, does not reproduce it. Measured over the corpus by truth class rather than by directory
label, at 0.02 vs 0.03 on the combined score: ice loops 62.13/62.19%, _icy protein 89.03/89.79%,
_clean protein 16.51/16.31%, water 17.19/20.03%. The widths are indistinguishable except on water,
where one of the four loops is independently known to carry a full hexagonal pattern. So the width
is now a parameter and the pipeline's own value is passed in - one band width, not two. The 0.012
tolerance in the radial channel is NOT a second band width, and is renamed CENTRE_SMEAR_Q to say so:
it is how far either side the channel looks for the bin a mis-set beam centre moved the ring to.
The rest is naming. Three kinds of number were all called score, or built from things called count,
and a reader could not tell from the name whether 1 meant "none" or "certain" - which are opposite.
The convention, now stated in docs/CPU_DATA_ANALYSIS.md: *_score is bounded [0,1] and 1 is
certainty, *_ratio is unbounded and 1 is nothing, *_count is a count. The C++ identifiers for the
ice ring ratio follow it (ice_ring_score -> ice_ring_ratio, GetIceRingScore -> GetIceRingRatio,
PlotType::IceRingScore -> IceRingRatio), and the local in the scaling gate that shadowed the new
ice_score while meaning the ring ratio is renamed with them.
Nothing outside the source moved: the CBOR keys ice_ring_score and ice_ring_score_mean, the datasets
/entry/MX/iceRingScore and iceRingScoreMean, the ice_ring_score plot type and the --ice-min-score
flag are all unchanged, and were checked to be after the rename. Renaming those changes stored
files, the stream format, the REST API and a CLI flag, and is a separate decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
Both lanes added a per-image scalar to the same eleven files, so every conflict was two
additions competing for one line. All were resolved by keeping both, with three that needed
more than that:
- ScanResultGenerator: rc167 changed the per-image float vectors to resize(n, NAN) so a frame
that never arrived does not read back as a real 0. v_protein_score and v_ice_score are exactly
that case - 0 is a real answer ("nothing detected here") - so they take the NAN default too.
- HDF5MetadataSource: rc167 established that NaN in a stored per-image array means "no value" and
the optional must come back absent. The two detection scores now follow it, which they did not
before the merge; without the guard a missing score would come back as a NaN that a threshold
would silently compare against.
- CBORTest: designated initialisers must follow member declaration order, so spindle_blind_fraction
precedes the two scores in the DataMessage aggregate.
Verified after the merge that every CBOR key that is encoded is also decoded (198 encoded keys,
one intentional exception: the "type" discriminator), that both lanes' fields survive in the
writer, the reader, the plots and the API, and that a stored file still round-trips.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
proteinScore and iceScore are saturating scores in [0,1] that answer presence, not quality: a superb
crystal and a barely-diffracting one both read near 1, and neither the spot count nor the resolution
enters either of them as a term. iceRingScore already reports a magnitude - a ratio, unbounded - which
is not a number that can be thresholded; these are.
The protein score counts distinct d SHELLS above 5 A rather than spots, so a parasitic ring in the
low-resolution band cannot accumulate evidence, and weights each spot against the frame's own median
so a scattering of the weakest detections cannot fill a shell either. The ice score carries two
channels and takes the stronger: a radial one over the azimuthal profile, which runs the hexagonal
and the CUBIC phase as separate hypotheses and decides between them at the end (flash-cooled loops
show cubic or stacking-disordered ice at least as often as hexagonal, the two share only three
lines, and dropping the cubic hypothesis costs about 5 pp on iced loops), and a spot one that reads
an excess on the ice radii against the same band slid to every ice-free offset, which is what
catches ice arriving as discrete crystallites and leaving the radial profile flat.
Both read d out of the geometry, so both move with a beam-centre error; the centre is not fitted
here, and the one they were computed with is written beside them as scoreBeamCenterX/Y so a later
rescoring can tell an algorithm disagreement from a geometry one.
Ported from validated prototypes and checked against them frame by frame on stored data: mean
absolute difference 2.7e-5 (protein), 1.3e-8 (ice radial) and 3.4e-4 (ice spots). On a 41-loop
battery the protein score reaches 98.4% of confirmed-protein frames and 0.00% of water frames, and
finds no cluster on any water or ice raster. Cost is 0.01 ms/frame for the protein score and
0.08-0.32 ms/frame for the ice score.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
The score was computed and then thrown away - carried on the per-image message but transported
nowhere - so the automation it exists for could not read it. It now flows like bkg_estimate at
every layer: CBOR (per-image key, END-block run mean and per-image array), HDF5 (per-image
/entry/MX/spindleBlindFraction in the data files, the array and spindleBlindFractionMean in the
master), read-back into a re-opened dataset, the scan result, the receiver plots, the REST plot
and scan_result schemas, and the viewer and frontend plot menus.
Absence is load-bearing and every transport keeps it distinguishable from a measured zero: the
CBOR key is simply missing, the HDF5 array holds NaN, and read-back turns NaN back into an
absent optional rather than a value. A pipeline that read 0 where the truth is "no value" would
take exactly the wrong action - a measured 0 says one sweep loses nothing, absence says nobody
could look, and the second must engage the recovery protocol while the first must not. The
round-trip tests pin all three states through CBOR and through a written-and-reopened file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.
**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.
**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.
**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**
Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Better track time for each operation in the processing stack
* jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress)
* jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress)
* jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0)
Reviewed-on: #46
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files)
* jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction
* jfjoch_broker: Image pusher statistics are accessible via the REST interface
* jfjoch_writer: Supports TCP image socket and for these auto-forking option
Reviewed-on: #36
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them,
especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123.
* jfjoch_broker: Improve refection position prediction and Bragg integration code.
* jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation.
* jfjoch_writer: Fix saving mosaicity properly in HDF5 file.
* jfjoch_viewer: Introduce high-dynamic range mode for images
* jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%)
* jfjoch_viewer: Zoom-in numbers have better readability
Reviewed-on: #31
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Add thresholding to prefer shorter vectors after FFT
* jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress)
* jfjoch_viewer: Display file opening errors
* jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available
Reviewed-on: #29
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.111 instead).
* jfjoch_broker: Improve handling of rotation indexing
* jfjoch_broker: More information saved in CBOR end message (WIP)
* jfjoch_writer: Save rotation indexing lattice parameters and Niggli class
* jfjoch_viewer: Remove (for now) primitive cell information
Reviewed-on: #19
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_viewer: Minor polishing of new functionality
* jfjoch_broker: User NaN for empty azimuthal bins
Reviewed-on: #12
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5