The profile is the MEAN of each bin, so a few strong reflections landing in a bin
lift it exactly as a smooth powder ring does. That is the wrong quantity whenever
the profile is wanted as a background rather than as a measurement of what is in
the bin - the ice score being the case in point, where reading a plain profile
INVERTED the metric: over 37 rotation crystals the two highest-scoring crystals
had no ice at all.
The adaptive spot finder already computes the right thing, a sigma-clipped
per-resolution-ring background, as a byproduct of its own threshold. Where it
runs, the ice score uses that. Where it does not - --no-adaptive-spots,
--azint-only, and anything reading the profile the broker wrote - there was no way
to get it. This adds one: azim_int_settings.sigma_clip (rugnux --azim-sigma-clip),
0 = off, minimum 2 because a tighter clip rejects a large part of a clean Gaussian
bin and biases the estimate low rather than removing outliers.
Two clip passes follow the plain one, matching the finder's recipe - the first
pass's standard deviation is itself inflated by the peaks being removed, so one
pass leaves a threshold that is still too generous. A bin with fewer than eight
pixels is left alone: at the detector edge and behind the beam stop there is no
spread to clip on.
Both engines do it. On the GPU the accept range is computed by a small kernel and
stays resident, so a clip pass is one more read of the same pixels and no round
trip; the two accumulation kernels take the range as a pointer that is null on the
plain pass. Measured on a JUNGFRAU rotation dataset, non-adaptive path: azimuthal
integration 0.02 -> 0.06 ms per image, exactly the 3x the extra passes predict,
against a 0.34 ms per-image total.
Note what the result IS: the smooth background under the peaks, not the bin mean.
It should not be switched on where a ring's integrated intensity is wanted - the
powder-ring geometry fit reads ring peaks, and those are what a clip is designed
to remove. Off by default, so nothing changes unless it is asked for.
Not exposed over the REST API - that needs the generated model regenerated, which
is a separate step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis.
If things go wrong with analysis, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Improve logic on switching on/off spot finding
* jfjoch_broker: Increase maximum spot count for FFBIDX to 65536
* jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD)
* jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis
* jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection
* jfjoch_writer: Save more information about each reflection
Reviewed-on: #32
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5