This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
* jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin.
* jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins.
Reviewed-on: #60
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress).
Reviewed-on: #45
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long
* jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata
* XDS plugin: Add generation of Jungfraujoch plugin for XDS
* CI: Add tests with XDS and DIALS (`xia2.ssx`)
Reviewed-on: #43
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files)
* jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction
* jfjoch_broker: Image pusher statistics are accessible via the REST interface
* jfjoch_writer: Supports TCP image socket and for these auto-forking option
Reviewed-on: #36
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Default EIGER readout time is 20 microseconds
* jfjoch_broker: Multiple improvements regarding performance
* jfjoch_broker: Image buffer allows to track frames in preparation and sending
* jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer
* jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets
* jfjoch_writer: Fixes regarding properly closing files in long data collections
* jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code
Reviewed-on: #34
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them,
especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123.
* jfjoch_broker: Improve refection position prediction and Bragg integration code.
* jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation.
* jfjoch_writer: Fix saving mosaicity properly in HDF5 file.
* jfjoch_viewer: Introduce high-dynamic range mode for images
* jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%)
* jfjoch_viewer: Zoom-in numbers have better readability
Reviewed-on: #31
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Use newer version of Google Ceres for (potential) CUDA 13 compatibility
* jfjoch_broker: Improve performance of generating preview images, especially for large detectors (9M-16M)
* jfjoch_viewer: Improve performance of displaying images, especially for large detectors (9M-16M)
* jfjoch_viewer: Add more color schemes for better image readability
* HDF5: Common mutex for reading and writing HDF5 if both operations were to happen in the same executable
* HDF5: suppress warning if path (upstream group) doesn't exists when checking if leaf exists
Reviewed-on: #30
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Add thresholding to prefer shorter vectors after FFT
* jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress)
* jfjoch_viewer: Display file opening errors
* jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available
Reviewed-on: #29
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Report changes in the image buffer, so viewer doesn't reload constantly
* jfjoch_viewer: Improve performance of loading images
* jfjoch_viewer: Auto-throttle image loading in HTTP-sync / movie modes
* jfjoch_viewer: Auto-foreground calculated with histogram
* jfjoch_viewer: Fix rare segmentation fault
Reviewed-on: #28
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.111 instead).
* jfjoch_broker: Improve handling of rotation indexing
* jfjoch_broker: More information saved in CBOR end message (WIP)
* jfjoch_writer: Save rotation indexing lattice parameters and Niggli class
* jfjoch_viewer: Remove (for now) primitive cell information
Reviewed-on: #19
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.11 instead).
* jfjoch_broker: Experimental rotation (3D) indexing
* jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values
Reviewed-on: #18
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release and not recommended for production use (please use rc.96 instead).
* jfjoch_broker: For DECTRIS detectors add dark data collection during initialization for bad pixel mask
* jfjoch_broker: Refactor of calibration logic for more clear code (likely to introduce problems)
* jfjoch_viewer: Add option to handle user pixel mask (experimental)
* jfjoch_viewer: More options for ROI
* jfjoch_viewer: Add window to display calibration
Reviewed-on: #2
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_f