images_sent was incremented right after image_pusher.SendImage(*loc), but the
ZeroCopyReturnValue overload was void and, for the TCP pusher, asynchronous: it
silently drops the image (releases the slot and returns) when there is no live
connection or the 2 s enqueue deadline expires. So images_sent over-counted on a
broken/slow writer connection and disagreed with the ACK-based GetImagesWritten().
Make SendImage(ZeroCopyReturnValue&) return whether the image was accepted
(enqueued/handed off) and only increment images_sent on success. The slot is
still released on the drop path. The authoritative delivered count remains
GetImagesWritten() (total_data_acked_ok for TCP). File/ZMQ pushers return true on
accept, preserving their previous always-counted behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three robustness fixes for the writer-facing TCP stream, addressing the
spurious "wrong number of images" / connection failures seen under load.
1. Never MSG_ZEROCOPY a transient buffer. The synchronous SendImage path
passes a caller-owned buffer with z == nullptr and reuses it for the next
frame immediately. With MSG_ZEROCOPY the kernel still references that buffer
after send() returns, so the peer could receive corrupted frames and drop
the connection mid-stream, truncating the run. Zero-copy is now gated on a
ZeroCopyReturnValue that keeps the buffer alive until completion.
2. Make the SendAll watchdog measure lack of progress, not total wall-clock.
The previous absolute deadline tore down a healthy but back-pressured
connection (slow/starved writer) after a fixed window; the watchdog now
resets on every byte actually sent, so only a genuinely stuck socket is
closed. Dead peers are still caught by OS keepalive and POLLHUP/POLLERR.
3. Make the END-ack wait progress-based (WaitForEndAck). The writer may still
be draining a backlog of DATA frames when END is sent; each DATA ACK is
progress, so the timeout only arms once the writer falls silent rather than
firing while images are legitimately still being drained.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files)
* jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction
* jfjoch_broker: Image pusher statistics are accessible via the REST interface
* jfjoch_writer: Supports TCP image socket and for these auto-forking option
Reviewed-on: #36
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Handle properly reuse of image buffer locations
* jfjoch_broker: Fix bug in counting idle slots
* jfjoch_broker: Force obtuse angle for monoclinic cells
* jfjoch_process: Change scaling refinement tolerance
Reviewed-on: #35
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Default EIGER readout time is 20 microseconds
* jfjoch_broker: Multiple improvements regarding performance
* jfjoch_broker: Image buffer allows to track frames in preparation and sending
* jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer
* jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets
* jfjoch_writer: Fixes regarding properly closing files in long data collections
* jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code
Reviewed-on: #34
leonarski_fandClaude Opus 4.8