18 Commits

Include renames

Author	SHA1	Message	Date
jungfrauandClaude Opus 5	3aa239fce8	Parallelise the incident-flux divide, drop a redundant sync ... DivideOutIncidentFlux was still the last fully serial pass in Ingest: a sweep over every observation to take each frame's mean background, and another to divide every rlp by its frame's flux. Ten gigabytes of traffic on one thread. The per-frame means go a frame at a time rather than an observation at a time, so each frame's running sum stays in one thread and in the order it had - splitting by observation would cut a frame across two threads and the partial sums would have to be recombined, which is a different sequence of roundings. The divide is per-element and splits anywhere. The adaptive spot finder synchronised after flagging strong pixels. The extractor that reads those pixels runs on the same stream, so the ordering already guaranteed the flagging had finished; the wait only idled the host, once per image. Measured on a crystal with 66 million partial observations: Ingest 8.5 s and 7.7 s -> 7.1 s and 6.6 s, whole crystal 1m24s -> 1m17s. Merged statistics unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 22:57:24 -04:00
jungfrauandClaude Opus 5	eb634400ea	Read four pixels at a time in the ring reduction ... reduce_rings_shared was 69% of all GPU kernel time - 116.8 s of a 70 s run across four cards. It is not bandwidth bound: flag_strong streams the same two arrays through the same grid-stride loop and reaches 196 GB/s, while this reached 30. The difference is the shared-memory atomics. Lanes in a warp read consecutive pixels along a detector row, a ring is a few pixels wide, so most of a warp lands in a handful of rings and the atomics to each one serialise. Two changes. The block reads four pixels per thread as one 16-byte and one 8-byte transaction, and merges the ones that fall in the same ring in registers before touching shared memory. Consecutive pixels usually DO share a ring, so this is where the win is: a run costs one set of atomics instead of one per pixel. npix is not guaranteed to be a multiple of four - it is width x height on the converted path, and detectors are not obliged to be even - so the vector loop stops short and a scalar loop finishes the remainder. Reading past the end would not fault, which is worse than if it did: it would fold uninitialised device memory into the accumulators and move the detection threshold in a way that does not reproduce. And the grid is sized from the occupancy the device reports, per pass. The two passes have different shared footprints - the first carries the corrected rings as well - so they do not fit the same number of blocks, and a grid sized for one left the other running a second wave at a quarter occupancy. The comment that justified the old grid reasoned from 1536 threads per SM, which is an Ada number; the card it ran on holds 1024. The run totals are still exactly what they were. The accumulators are unsigned 64-bit, so summing a run in a register and adding it once is the same value as adding each pixel separately - addition mod 2^64 is associative, overflow included - which is what keeps the ring statistics, and therefore the detection threshold, independent of how the work was grouped. That is the property the integer accumulators exist for. (The run accumulators are unsigned for the same reason: signed overflow would be undefined, and four squares of a large pixel value reach 2^64.) The corrected float sums, which feed the reported profile rather than any decision, change in their last bits as they already did between runs. Measured on a 16M-pixel rotation dataset: the kernel 116.8 s -> 19.1 s (6.1x), no longer the largest; the whole run 70 s -> 39.5 s. Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 12m47s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 19:21:24 -04:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	d760b12a18	v1.0.0-rc.141 (#51) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times * frontend: Fix selection of FFTW in indexing settings Reviewed-on: #51 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 13:04:54 +02:00
leonarski_fandtakaba_k	230480e390	v1.0.0-rc.138 (#48) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Cleanup DECTRIS start-up code to enable a shorter start time jfjoch_broker: Allow for asynchronous start to allow overlapping detector configuration with other beamline preparations jfjoch_broker: Goniometer axis name is converted to lowercase jfjoch_broker: Fix bug, where wrong HTTP error codes were returned jfjoch_broker: Improve sigma estimation during merging (K. Takaba) --------- Co-authored-by: takaba_k <kiyofumi.takaba@psi.ch> Reviewed-on: #48 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-27 19:56:14 +02:00
leonarski_f	c981e1b91c	v1.0.0-rc.137 (#46) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Better track time for each operation in the processing stack * jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress) * jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress) * jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0) Reviewed-on: #46	2026-04-25 19:59:21 +02:00
leonarski_f	bb9f5c715f	v1.0.0-rc.135 (#44) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4) * jfjoch_viewer: Improve image render performance Reviewed-on: #44 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-16 11:59:59 +02:00
leonarski_f	06949caf1a	v1.0.0-rc.112 (#18) ... This is an UNSTABLE release and not recommended for production use (please use rc.11 instead). * jfjoch_broker: Experimental rotation (3D) indexing * jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values Reviewed-on: #18 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-30 17:39:22 +01:00
leonarski_f	061152279c	v1.0.0-rc.91	2025-10-20 20:43:44 +02:00
leonarski_f	14f5051692	1.0.0-rc.89	2025-10-01 22:54:40 +02:00
leonarski_f	a48d8fb5b9	1.0.0.rc-88	2025-10-01 11:18:10 +02:00
leonarski_f	99b7dc07f7	1.0.0-rc.87	2025-09-30 20:43:53 +02:00
leonarski_f	5d9d2de4a4	v1.0.0-rc.81	2025-09-21 19:27:51 +02:00
leonarski_f	dba807fadd	v1.0.0-rc.71	2025-08-28 07:07:01 +02:00
leonarski_f	bb32f27635	v1.0.0-rc.70	2025-08-27 06:21:10 +02:00