The limit is EXCLUSIVE inside Jungfraujoch - the first value that is no longer a count - and NXmx
saturation_value is INCLUSIVE, the highest value that still is one. XDS OVERLOAD and the DIALS
trusted_range read it inclusively too. The write side subtracted the count and no read side added it
back, so the value fell by one on every write-read-write cycle, unbounded: four chained runs over one
dataset gave 32766, 32765, 32764, 32763. It also fed the preprocessor, so one more real count was
called saturated after each cycle.
SaturationValueFromLimit / SaturationLimitFromValue now carry the conversion, used by the writer and
by all three readers (HDF5, the lite receiver, the viewer). JFJochReaderImage's summation test moves
from > to >= in the same commit: it was silently compensating for the missing count, and correcting
one without the other would have shifted it instead.
Two more places said the wrong thing about the same pixels:
error_value was GetUnderflow(), which is -1 for an unsigned image - a value no unsigned pixel can
hold. The marker those images really carry is UINTx_MAX, and GetImageFillValue() already returned it,
so the class held two disagreeing definitions of one marker.
bit_depth_readout is now written for unsigned images only. DIALS remaps the top two codes of
2^bit_depth_readout to -1 and -2 whenever the field is present, without looking at the pixel type.
For an unsigned image those fall below underload_value and are masked, which is what we want. For a
signed one they land INSIDE the trusted range, so a saturated pixel reached DIALS as a trusted count
of -2 - on the strongest reflections. Verified with DIALS 3.27: an int32 file now masks both
sentinels. The field stays where it earns its keep, since dxtbx cannot read unsigned 32-bit without
it.
Neither the values themselves nor the wire format change. Verified against NXmx, DECTRIS SIMPLON,
Durin (Global Phasing fork), XDS and DIALS 3.27; the chained run now holds at 32766.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z
The value means three different things at once. It is the unit of writer
parallelism - whole files go round-robin to the writers,
(image_number / images_per_file) % socket.size() in ZMQStream2Pusher::SendImage
and TCPStreamPusher - it multiplies writer memory linearly, since every data-file
plugin reserves per file, and it decides whether a legacy master is readable at
all, because dxtbx follows only the first data file of one.
That last point is what makes a flat default wrong. Measured with DIALS on a
2500-image rotation sweep written as legacy: split into five files it reports
2500 images and then raises IndexError beyond image 499, so it half-works
silently; in one file all 2500 read. AutoPROC does not read VDS, so legacy has to
stay the default, which leaves the file count as the only lever.
So make it optional and resolve it from the acquisition. A rotation sweep of at
most 20000 images goes into one data file - rotation datasets are small enough,
and one writer keeps up with them. A grid scan splits on whole fast-axis rows, so
a file is a meaningful piece of the grid. Stills and serial keep 1000, where the
image count far exceeds it and the parallelism and the bounded writer memory are
what matter. An explicit value is always taken literally.
GetImagesPerFile is the single place this is resolved, and it must always return
a fixed non-zero number, because everything downstream - receiver, pusher, puller,
writer - requires one. That was already true of the old 0 = "one file" spelling;
0 is now gone from the API and omitting the field says the same thing better.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Default EIGER readout time is 20 microseconds
* jfjoch_broker: Multiple improvements regarding performance
* jfjoch_broker: Image buffer allows to track frames in preparation and sending
* jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer
* jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets
* jfjoch_writer: Fixes regarding properly closing files in long data collections
* jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code
Reviewed-on: #34
This is an UNSTABLE release.
* jfjoch_viewer: Add keyboard shortcuts and option to copy image to clipboard
* jfjoch_broker: Fix bit-width and exposure time for PSI EIGER detectors
Reviewed-on: #15
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.96 instead).
* jfjoch_broker: For DECTRIS detectors add dark data collection during initialization for bad pixel mask
* jfjoch_broker: Refactor of calibration logic for more clear code (likely to introduce problems)
* jfjoch_viewer: Add option to handle user pixel mask (experimental)
* jfjoch_viewer: More options for ROI
* jfjoch_viewer: Add window to display calibration
Reviewed-on: #2
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
* Writer can send a PUB message with information on closed file
* Writer uses getopt to format input parameters
* DiffractionExperiment: Add frame number limit
leonarski_fandClaude Opus 5