8 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	9da3344639	ZeroMQ puller: the message region without a ring takes the message's read-only bytes ... ZMQMessage::data() returns const uint8_t *, and GCC refuses the implicit conversion to the region's uint8_t *. In this mode the region is only read (deserialised and republished), never written, so the cast is safe. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-13 13:17:48 +02:00
leonarski_fandClaude Fable 5.1	1be6093429	Writer: received images live in a byte-bounded ring buffer until written and republished ... Every stage between the socket and the HDF5 write was bounded by a message count and never by bytes: ZeroMQ receive watermark 100, cbor_fifo 200, outside_fifo 1000, repub_fifo 200 - up to some 1500 images in flight per writer process, each a whole compressed image, so 6-12 GB for a JUNGFRAU 4M and 30-60 GB for a 9M, and the writer's memory was a quantity nobody had chosen. ReceiveRingBuffer is a fixed ring of bytes owned by the ImagePuller base class, so the ZeroMQ and TCP pullers share it. A message is received straight into it (TCP reads the payload into the ring; ZeroMQ copies the message in) and travels as a shared_ptr region in ImagePullerOutput; the deserialised image points into that region, republish sends it as is, and the bytes come back when the last holder lets go. Space is reclaimed in arrival order: a region released out of turn - republish finishes with messages in its own order - is marked and waits for the ones in front of it. Because of that, what republish may hold is capped at a quarter of the ring, and an image over the cap is not republished, exactly as one that finds the republish queue full; otherwise a stalled republish consumer would pin the ring and slow the writer to its rate. A message larger than the whole ring (a START or END of a very long run) goes on the heap, so it cannot wedge the receiver. A full ring stalls the receive thread, and what that does depends on the stream. On TCP the pusher waits behind the writer's BUSY heartbeats and gives up only after a minute in which no image at all was written. On ZeroMQ the pusher's send blocks for one second, then drops the image and sends without waiting for the rest of the run - so on ZeroMQ a ring that is smaller than the writer's stalls costs images, where the old 1500-message queues absorbed them. That is the trade: the budget is now a number, and it has to be chosen. jfjoch_writer sets it with -m/--rcv_ring_mb, default 2048 MB per writer process; the ZeroMQ receive watermark default goes from 100 to 10 messages, since it now only counts what sits beyond the ring. The ring's peak use is in the end-of-run log line, and docs/JFJOCH_WRITER.md has a Memory section with the sizing rule. The puller's own default is no ring at all, which keeps every other user as it was: the broker's DECTRIS stream2 puller (JFJochServices, JFJochReceiverService) stays zero-copy with the ZeroMQ message as the buffer, and the broker's direct HDF5 writer (HDF5FilePusher -> FileWriter) never goes through a puller. The puller threads now also catch std::exception, as the writer does since the previous commit: deserialising the END message of a million-image run allocates a vector per per-image array, and a bad_alloc leaving the thread ended the process. Tests: ReceiveRingBufferTest (reuse, wrap-around, full ring, out-of-order release, heap fallback, zero size, stop, a region outliving its ring); ZMQImageCommTest_SmallRingBuffer (64 one-megabyte images through a three-image ring, nothing lost, peak within the bound); one of the two writers in TCPImageCommTest_2Writers_WithAck runs on a three-image ring. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-13 12:51:34 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	239a441ee6	v1.0.0-rc.140 (#50) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection * jfjoch_broker: Minor bug fixes for rare conditions Reviewed-on: #50	2026-04-29 21:40:22 +02:00
leonarski_f	4878318c27	v1.0.0-rc.139 (#49) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start` * jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start' * jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time * jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received * jfjoch_broker: Use nanoseconds internally for frame/image/readout time * jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished) * jfojch_broker: Improve profiling of different data analysis steps * jfjoch_broker: Record integration reflection count * jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds) * jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked * jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets **Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time. Reviewed-on: #49	2026-04-29 09:50:50 +02:00
leonarski_f	f3e0a15d26	v1.0.0-rc.127 (#34) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Default EIGER readout time is 20 microseconds * jfjoch_broker: Multiple improvements regarding performance * jfjoch_broker: Image buffer allows to track frames in preparation and sending * jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer * jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets * jfjoch_writer: Fixes regarding properly closing files in long data collections * jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code Reviewed-on: #34	2026-03-02 15:57:12 +01:00
leonarski_f	e4ec9fb38f	v1.0.0-rc.61	2025-07-08 07:16:59 +02:00
leonarski_f	040cf08386	v1.0.0-rc.36	2025-05-05 19:32:22 +02:00