13 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	41983ca589	Broker: refuse a run the writer cannot write before arming the detector ... A writer refusing to overwrite an existing file was reported badly, and the earlier fix only covered half of it. The commit message of a6a715703 states the assumption that broke: "SendStartMessage precedes every std::async in the receiver constructor". That is true of JFJochReceiverFPGA, where the start message goes out from the constructor body, and false of JFJochReceiverLite, where it goes out from MeasurementThread - launched by std::async from the constructor, which waits only on data_analysis_started. On the DECTRIS path the writer's refusal therefore cannot reach services.Start(); it arrives from services.Stop(), i.e. the second try-block of the state machine's MeasurementThread, which never touched start_exception. /wait_till_done answered 200 for a run that never wrote anything. Two changes, in front of the problem and behind it. In front: a PREFLIGHT frame (TCP protocol version 3 -> 4), sent to every writer before the receiver is built and before the detector is armed. It carries the same CBOR start message a START does and asks one question: could this run be written? The writer runs FileWriter::Preflight - the first three statements of the FileWriter constructor, stopping short of the only one that creates anything - and answers with an ACK. Shared code, so a pre-flight that passes cannot be contradicted by the start behind it. A refusal is not fatal: nothing was started, the connection stays usable, and above all no series is left armed waiting for a trigger that will not come. Both ends already reject a frame of another protocol version, so the bump costs no compatibility. The check now covers the data files, not only the master. A stale _data_000001.h5 used to sail through and fail when the file was renamed into place at the end of the run, after the whole dataset had been collected. The writer holding the master file checks for all of them - they share a directory by construction, the master linking them by relative name - so the siblings do not race each other statting files the others are creating. Behind: /wait_till_done maps an Idle state to 200 or 500 by the severity of the end-of-run message, so a collection that lost packets, or whose writer could not write, is no longer indistinguishable from a good one to a script. A cancelled collection is not affected - MeasurementThread checks the cancel before it looks at the completeness of the data, and reports it with Info severity - and a warning is still a 200. start_exception is untouched. The PREFLIGHT payload is built from FillMessage alone, which is what keeps it small: the per-pixel arrays are added by SendStartMessage and are not in it. Measured on a JUNGFRAU 9M, 1.4 kB against 543 kB for the start message. It is deliberately the whole of FillMessage rather than the fields the check reads today - a hand-picked subset would answer for a different set of file names the day the check learns to read one more field, and would do it silently. Three defects found on the way: TCPImagePuller republished a PREFLIGHT as a real start message, announcing to downstream consumers a run that was not going to happen. TCPStreamPusher::run_number was a plain uint64_t written by the control thread and read by every ACK thread; Preflight adds a second writer to it. StreamWriter::CollectImages reprocessed the last frame whenever WaitForImage returned false - run is only tested at the top of the loop - so an abort wrote the last image a second time. Pre-existing; the new branch made it visible. Verified against a running broker on the local configuration: a colliding master file and a colliding data file are both refused by /start, /wait_until_running and /wait_till_done with the writer's own message, and the log shows the receiver was never started for either; a cancelled collection still answers 200. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WmryXe8ASbNi632sUMfsa	2026-08-27 22:09:50 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	6133da1377	v1.0.0-rc.133 ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Use httplib for HTTP server instead of Pistache * jfjoch_broker: Drop OpenSSL support * jfjoch_broker: Base work for multi-lattice support in the future * Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib) Reviewed-on: #41	2026-03-26 20:50:33 +01:00
leonarski_f	64002f1e29	v1.0.0-rc.129 (#36) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files) * jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction * jfjoch_broker: Image pusher statistics are accessible via the REST interface * jfjoch_writer: Supports TCP image socket and for these auto-forking option Reviewed-on: #36 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-03-05 22:13:12 +01:00
leonarski_f	f3e0a15d26	v1.0.0-rc.127 (#34) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Default EIGER readout time is 20 microseconds * jfjoch_broker: Multiple improvements regarding performance * jfjoch_broker: Image buffer allows to track frames in preparation and sending * jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer * jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets * jfjoch_writer: Fixes regarding properly closing files in long data collections * jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code Reviewed-on: #34	2026-03-02 15:57:12 +01:00
leonarski_f	06c5b9cf7f	1.0.0-rc.65	2025-07-16 20:19:48 +02:00
leonarski_f	8e449fc581	v1.0.0-rc.47	2025-06-17 16:28:28 +02:00
leonarski_f	040cf08386	v1.0.0-rc.36	2025-05-05 19:32:22 +02:00
leonarski_f	ddf4c75645	v1.0.0-rc.31	2025-03-02 13:15:28 +01:00
leonarski_f	28d224afab	version 1.0.0-rc.25	2024-11-22 21:25:20 +01:00
leonarski_f	e812918e2e	version 1.0.0-rc.13	2024-10-05 13:14:49 +02:00