Live stills: a collection whose pre-pass does not fit the image buffer is refused at start

As the owner decided: no capping. Where min(stills_prepass_images, images)
is more than half of the image buffer's slots, JFJochReceiverService::Start
throws InputParameterInvalid before the stills receiver is built - and so
before the detector is armed (JFJochServices starts the receiver first). A
synchronous /start answers 400, an asynchronous one reports the error through
/wait_till_done, and the broker goes back to Idle. The message gives the
slots, the slot size, N, the image buffer it needs (2 x N x slot, as
image_buffer_MiB) and the stills_prepass_images that would fit.
CheckStillsPrePassFits (StillsGPUAnalysis.h) is shared with
jfjoch_stills_gpu_replay, which exits with the same message.

The capping in JFJochReceiverRugnuxStills::OnStart and its "the pre-pass
runs on the first K images" note are removed. JFJOCH_BROKER.md and the
stills_prepass_images description say the collection is refused (C++ model,
Python docs and frontend types regenerated).

Tests: JFJochReceiverRugnux_Stills_PrePassDoesNotFit (7 of a 12-slot buffer:
refused, InputParameterInvalid, nothing sent); BufferFullDuringPrePass keeps
200 frames through a 12-slot buffer with a pre-pass of 6, all written;
CancelDuringRun takes a pre-pass of 2 for its 4 slots;
GPUStills_GeometryRefinedOnModules and GPUStills_cancel_during_prepass get an
image buffer of 2 x N slots. Replay: -b 30000 at -P 1000 is refused; with
2020 slots per-frame results as before on both windows checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-19 15:36:33 +01:00
1 parent aa676e215d
commit 283adb8d98
12 files changed
+91 -43

No files matched your search

+1 -1
View File
@@ -100,7 +100,7 @@ public:
bool stillsDownsampleIsSet() const;
void unsetStills_downsample();
/// <summary>
/// Serial collections: the first images the pre-pass looks at. They are held in the image buffer, and the stream waits, while the pre-pass measures the mask, polarization and integration settings on them and, where at least 6 of them are strong indexed frames (a unit cell given), refines the beam centre, distance and cell; with fewer, the distance and cell of the dataset settings are used, at the beam centre (the stated one or the scattered background&#39;s) that indexes better. Each image waits in a slot of the image buffer of its own, and the pre-pass may take half of the slots: N images need 2N slots in the buffer. Where they do not fit, the pre-pass runs on the first &#x60;slots / 2&#x60; images (said in the log and &#x60;/rugnux/status&#x60; at the start). Rotation collections are not affected. Reset to 1000 by &#x60;POST /initialize&#x60;.
/// Serial collections: the first images the pre-pass looks at. They are held in the image buffer, and the stream waits, while the pre-pass measures the mask, polarization and integration settings on them and, where at least 6 of them are strong indexed frames (a unit cell given), refines the beam centre, distance and cell; with fewer, the distance and cell of the dataset settings are used, at the beam centre (the stated one or the scattered background&#39;s) that indexes better. Each image waits in a slot of the image buffer of its own, and the pre-pass may take half of the slots: N images need 2N slots in the buffer. Where they do not fit, &#x60;/start&#x60; refuses a serial collection (with the image buffer size it needs). Rotation collections are not affected. Reset to 1000 by &#x60;POST /initialize&#x60;.
/// </summary>
int64_t getStillsPrepassImages() const;
void setStillsPrepassImages(int64_t const value);
+2 -2
View File
@@ -2869,8 +2869,8 @@ components:
distance and cell; with fewer, the distance and cell of the dataset settings are used, at the beam centre
(the stated one or the scattered background's) that indexes better. Each image waits
in a slot of the image buffer of its own, and the pre-pass may take half of the slots: N images need 2N
slots in the buffer. Where they do not fit, the pre-pass runs on the first `slots / 2` images (said
in the log and `/rugnux/status` at the start). Rotation collections are not affected. Reset to 1000 by `POST /initialize`.
slots in the buffer. Where they do not fit, `/start` refuses a serial collection (with the image
buffer size it needs). Rotation collections are not affected. Reset to 1000 by `POST /initialize`.
rugnux_status:
type: object
required:
+5 -5
View File
@@ -377,8 +377,8 @@ and the end message carries the centre and distance used (`refined_beam_center_x
[CBOR](CBOR.md)). The broker's own distance is taken as the beamline's: unlike offline rugnux, the pre-pass does not
search the whole distance range. It refines the centre and distance for a detector square to the beam, as the frames
are then analysed. The pre-pass holds its frames in half of the image buffer's slots, one frame to a slot (below):
where the `stills_prepass_images` frames do not fit there, it runs on the first `slots / 2` frames, and says so in
the log and the live status when the collection starts.
where the `stills_prepass_images` frames do not fit there, `/start` refuses the collection (HTTP 400, or the error of `/wait_till_done` after an asynchronous start) before the
detector is armed, saying how large the image buffer must be.
The analysis starts when the pre-pass ends and catches up from the frames waiting.
The geometry is fixed when the pre-pass ends and does not change during the collection, so every frame is analysed
@@ -394,9 +394,9 @@ go to the writer from a slot each. A slot is the largest compressed image plus 4
slots = image_buffer_MiB × 1 MiB / (largest compressed image + 4 MiB)
and N = `stills_prepass_images` frames need N slots, which are half of the buffer's: 2N slots in all. Nothing is
written before the pre-pass, so it always holds `slots / 2` frames; where N is larger, it runs on the first
`slots / 2`. After the pre-pass, frames waiting for the writer use slots too, and fewer frames wait. The
and N = `stills_prepass_images` frames need N slots, which are half of the buffer's: 2N slots in all, an image
buffer of at least 2 × N × slot. Nothing is written before the pre-pass, so it always holds `slots / 2` frames; a
collection with N larger (and more than N images) is refused at the start. After the pre-pass, frames waiting for the writer use slots too, and fewer frames wait. The
receiver's image memory is the image buffer (`image_buffer_MiB`, `-B` of `jfjoch_fpga_stills_test`) and nothing
beside it. No stage on the host discards a frame: a frame that finds no slot it may take waits until the analysis
has made room, and the wait goes back to the source - from FPGA cards, `FPGAImagePuller` holds the cards' buffers and
+1 -1
View File
@@ -12,7 +12,7 @@ Name | Type | Description | Notes
**refine_solvent** | **bool** | With a model: the refinement also builds the solvent - waters added at difference-map peaks and removed where the maps no longer hold them, waters and ions relabelled (as &#x60;rugnux --refine-solvent&#x60;). Reset to false by &#x60;POST /initialize&#x60;. | [optional] [default to False]
**force_still** | **bool** | Process DECTRIS rotation collections as stills, on the serial path (pre-pass, GPU analysis, stills merge), as &#x60;rugnux --force-still&#x60; does. The written file keeps its goniometer. Reset to false by &#x60;POST /initialize&#x60;. | [optional] [default to False]
**stills_downsample** | **int** | Serial collections: analyse on average 1 of every this many images, to keep up with a fast stream. Each image is chosen on its own with probability 1 / stills_downsample from a hash of its number - not a stride, which would be in step with periodic patterns (pump-probe on/off, storage cells). The images the pre-pass looks at (the first &#x60;stills_prepass_images&#x60;) are all analysed. Images not chosen are written as they came, without their analysis. 1: every image. Reset to 1 by &#x60;POST /initialize&#x60;. | [optional] [default to 1]
**stills_prepass_images** | **int** | Serial collections: the first images the pre-pass looks at. They are held in the image buffer, and the stream waits, while the pre-pass measures the mask, polarization and integration settings on them and, where at least 6 of them are strong indexed frames (a unit cell given), refines the beam centre, distance and cell; with fewer, the distance and cell of the dataset settings are used, at the beam centre (the stated one or the scattered background&#39;s) that indexes better. Each image waits in a slot of the image buffer of its own, and the pre-pass may take half of the slots: N images need 2N slots in the buffer. Where they do not fit, the pre-pass runs on the first &#x60;slots / 2&#x60; images (said in the log and &#x60;/rugnux/status&#x60; at the start). Rotation collections are not affected. Reset to 1000 by &#x60;POST /initialize&#x60;. | [optional] [default to 1000]
**stills_prepass_images** | **int** | Serial collections: the first images the pre-pass looks at. They are held in the image buffer, and the stream waits, while the pre-pass measures the mask, polarization and integration settings on them and, where at least 6 of them are strong indexed frames (a unit cell given), refines the beam centre, distance and cell; with fewer, the distance and cell of the dataset settings are used, at the beam centre (the stated one or the scattered background&#39;s) that indexes better. Each image waits in a slot of the image buffer of its own, and the pre-pass may take half of the slots: N images need 2N slots in the buffer. Where they do not fit, &#x60;/start&#x60; refuses a serial collection (with the image buffer size it needs). Rotation collections are not affected. Reset to 1000 by &#x60;POST /initialize&#x60;. | [optional] [default to 1000]
## Example
+2 -2
View File
@@ -2064,8 +2064,8 @@ export type rugnux_settings = {
* distance and cell; with fewer, the distance and cell of the dataset settings are used, at the beam centre
* (the stated one or the scattered background's) that indexes better. Each image waits
* in a slot of the image buffer of its own, and the pre-pass may take half of the slots: N images need 2N
* slots in the buffer. Where they do not fit, the pre-pass runs on the first `slots / 2` images (said
* in the log and `/rugnux/status` at the start). Rotation collections are not affected. Reset to 1000 by `POST /initialize`.
* slots in the buffer. Where they do not fit, `/start` refuses a serial collection (with the image
* buffer size it needs). Rotation collections are not affected. Reset to 1000 by `POST /initialize`.
*
*/
stills_prepass_images?: number;
+4 -13
View File
@@ -54,11 +54,10 @@ JFJochReceiverRugnuxStills::~JFJochReceiverRugnuxStills() {
void JFJochReceiverRugnuxStills::OnStart(std::chrono::steady_clock::time_point received) {
// The frames that wait take a slot each, at most half of the slots; the other half is for those analysed, sent
// from a slot each. Nothing is sent before the pre-pass, so it holds half of the slots, whatever the frames.
// from a slot each. Nothing is sent before the pre-pass, so it holds half of the slots, whatever the frames: the
// receiver service has checked that its images fit (CheckStillsPrePassFits).
const int64_t slots = image_buffer.GetStatus().total_slots;
hold_keep_free = slots - slots / 2;
const int64_t prepass_asked = std::min(prepass_images, experiment.GetImageNum());
const int64_t prepass_held = std::min(prepass_asked, slots / 2);
const StartMessage start = MakeStartMessage(experiment, pixel_mask, *az_int_mapping);
// The frames are shown from their arrival, on the stated geometry until the pre-pass has decided its own.
// To the image buffer only: the writer and the ZeroMQ streams get the one start message, after the pre-pass.
@@ -74,16 +73,8 @@ void JFJochReceiverRugnuxStills::OnStart(std::chrono::steady_clock::time_point r
auto analysis = std::make_unique<StillsGPUAnalysis>(
analysis_experiment, pixel_mask, analysis_start,
indexing_thread_pool, stills_nthreads,
prepass_held, logger,
prepass_images, logger,
*this, live_status, true, model, deadline_s, refine_solvent);
// Noted once the analysis has started the live status
if (prepass_held < prepass_asked) {
const std::string text = fmt::format("The image buffer has {} slots, half of them for the frames waiting: "
"the pre-pass runs on the first {} images, not the {} asked "
"(stills_prepass_images)", slots, prepass_held, prepass_asked);
logger.Warning("{}", text);
live_status.Note(text);
}
std::unique_lock ul(stills_mutex);
stills = std::move(analysis);
if (cut)
@@ -101,7 +92,7 @@ void JFJochReceiverRugnuxStills::OnImage(uint32_t id, DataMessage &message, std:
const size_t size = message.image.GetCompressedSize();
// Waits for the analysis in a slot of the image buffer, where it can be seen while it waits (it is not sent from
// there). With no slot to take, the frame waits until the analysis has made room: the pre-pass window fits the
// there). With no slot to take, the frame waits until the analysis has made room: the pre-pass's images fit the
// slots (OnStart). No frame is lost here; a source faster than the analysis loses frames at the card.
auto held = std::make_shared<HeldImage>();
held->slot = image_buffer.GetImageSlot(hold_keep_free);
+5 -2
View File
@@ -210,13 +210,16 @@ void JFJochReceiverService::Start(const DiffractionExperiment &experiment,
std::max<int>(1, static_cast<int>(AvailableCpus()) - rugnux_reserved_cpus),
rugnux_model, rugnux_deadline_s, rugnux_refine_solvent, rugnux_status, spot_finding_settings,
receiver_status, plots, image_buffer, zmq_preview_socket.get(), zmq_metadata_socket.get());
else if (rugnux_enable && (experiment.GetDetectorMode() == DetectorMode::Standard) && !rotation_data)
else if (rugnux_enable && (experiment.GetDetectorMode() == DetectorMode::Standard) && !rotation_data) {
// Refused before the detector is armed
CheckStillsPrePassFits(slots, experiment.GetImageBufferLocationSize(),
std::min(rugnux_stills_prepass_images, experiment.GetImageNum()));
receiver = std::make_unique<JFJochReceiverRugnuxStills>(
experiment, pixel_mask, *image_puller, image_pusher, logger, nthreads,
rugnux_model, rugnux_deadline_s, rugnux_refine_solvent, rugnux_status, spot_finding_settings,
receiver_status, plots, image_buffer, zmq_preview_socket.get(), zmq_metadata_socket.get(),
indexer_thread_pool.get(), rugnux_stills_downsample, rugnux_stills_prepass_images);
else
} else
receiver = std::make_unique<JFJochReceiverLite>(experiment,
pixel_mask,
*image_puller,
+15
View File
@@ -3,6 +3,7 @@
#include "StillsGPUAnalysis.h"
#include <cmath>
#include <thread>
#include "../common/CUDAWrapper.h"
@@ -14,6 +15,20 @@
#include "../writer/RugnuxResultBlocks.h"
#include "RugnuxStructure.h"
void CheckStillsPrePassFits(int64_t slots, int64_t slot_size, int64_t prepass_images) {
if (prepass_images <= slots / 2)
return;
const double slot_MiB = static_cast<double>(slot_size) / (1024.0 * 1024.0);
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
fmt::format("Serial collection refused: the pre-pass needs its first {} images in the image "
"buffer, one image to a slot of {:.1f} MiB, in half of the slots, but the buffer "
"has {} slots ({} for the pre-pass). Make the image buffer (image_buffer_MiB) at "
"least {:.0f} MiB (2 x {} x {:.1f} MiB), or lower stills_prepass_images to {}.",
prepass_images, slot_MiB, slots, slots / 2,
std::ceil(2.0 * static_cast<double>(prepass_images) * slot_MiB),
prepass_images, slot_MiB, slots / 2));
}
StillsGPUAnalysis::StillsGPUAnalysis(const DiffractionExperiment &in_experiment, const PixelMask &in_mask,
const StartMessage &in_start, IndexerThreadPool *in_indexing_thread_pool,
int64_t nthreads, int64_t max_held, Logger &in_logger, StillsSink &in_sink,
+4
View File
@@ -45,6 +45,10 @@ public:
};
// Serial frames analysed on the GPU at the geometry of a pre-pass (fixed from there on), and merged beside the analysis
// The pre-pass waits for its first `prepass_images` images in the image buffer, one to a slot, in at most half of
// the `slots`: a collection they do not fit is refused (InputParameterInvalid), before anything is started
void CheckStillsPrePassFits(int64_t slots, int64_t slot_size, int64_t prepass_images);
class StillsGPUAnalysis {
public:
constexpr static std::array<double, 2> CHECKPOINTS = {0.5, 0.75};
+6 -2
View File
@@ -282,7 +282,9 @@ TEST_CASE("JFJochIntegrationTest_GPUStills_GeometryRefinedOnModules", "[JFJochRe
aq_devices.Add(std::move(test));
NonePusher pusher;
JFJochReceiverService service(aq_devices, logger, pusher);
// The pre-pass holds all 60 frames, one to a slot, in half of the image buffer
JFJochReceiverService service(aq_devices, logger, pusher,
2 * 60 * experiment.GetImageBufferLocationSize() / (1024 * 1024) + 1);
service.NumThreads(4);
service.Indexing(experiment.GetIndexingSettings());
service.RugnuxLive(true, nullptr, 4, 60);
@@ -331,7 +333,9 @@ TEST_CASE("JFJochIntegrationTest_GPUStills_cancel_during_prepass", "[JFJochRecei
aq_devices.Add(std::move(test));
NonePusher pusher;
JFJochReceiverService service(aq_devices, logger, pusher);
// The pre-pass's 30 frames fit, one to a slot, in half of the image buffer
JFJochReceiverService service(aq_devices, logger, pusher,
2 * 30 * experiment.GetImageBufferLocationSize() / (1024 * 1024) + 1);
service.NumThreads(4);
service.Indexing(experiment.GetIndexingSettings());
service.RugnuxLive(true, nullptr, 4, 60);
+39 -10
View File
@@ -352,8 +352,8 @@ TEST_CASE("JFJochReceiverRugnux_Stills_CancelDuringRun", "[JFJochReceiverRugnux]
PixelMask mask(x);
Stream stream(x, images);
// Four slots: the pre-pass starts once the two the frames wait in are full, and the analysis starts with most of
// the run to come. The frames are given one by one until it does.
// Four slots and a pre-pass of two frames, the two the frames wait in: the analysis starts with most of the run to
// come. The frames are given one by one until it does.
const size_t buffer_MiB = 4 * x.GetImageBufferLocationSize() / (1024 * 1024) + 1;
RecordingPusher pusher;
@@ -362,7 +362,7 @@ TEST_CASE("JFJochReceiverRugnux_Stills_CancelDuringRun", "[JFJochReceiverRugnux]
service.NumThreads(2);
service.Indexing(x.GetIndexingSettings());
// A model only so that there is a model step for the cancel to cut.
service.RugnuxLive(true, std::make_shared<RugnuxModel>(), 4, 60);
service.RugnuxLive(true, std::make_shared<RugnuxModel>(), 4, 60, false, false, 1, 2);
service.Start(x, mask, nullptr, stream.puller);
int64_t given = 0;
@@ -649,10 +649,9 @@ TEST_CASE("JFJochReceiverRugnux_Stills_ZMQWriterDown", "[JFJochReceiverRugnux]")
REQUIRE_NOTHROW(writer_future.get());
}
TEST_CASE("JFJochReceiverRugnux_Stills_BufferFullDuringPrePass", "[JFJochReceiverRugnux]") {
// More frames than half of a twelve-slot image buffer holds, all there before the pre-pass ends. The frames that
// wait take a slot each, at most half of the slots: the pre-pass runs on the first six, said at the start, a frame
// past them waits and the analysis drains them. Every frame is written.
TEST_CASE("JFJochReceiverRugnux_Stills_PrePassDoesNotFit", "[JFJochReceiverRugnux]") {
// The pre-pass's images wait one to a slot in half of the slots: seven of them in a twelve-slot image buffer do
// not fit, and the collection is refused at the start, with nothing sent
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
@@ -668,7 +667,38 @@ TEST_CASE("JFJochReceiverRugnux_Stills_BufferFullDuringPrePass", "[JFJochReceive
JFJochReceiverService service(group, logger, pusher, buffer_MiB);
service.NumThreads(2);
service.Indexing(x.GetIndexingSettings());
service.RugnuxLive(true, nullptr, 4, 60);
service.RugnuxLive(true, nullptr, 4, 60, false, false, 1, 7);
try {
service.Start(x, mask, nullptr, stream.puller);
FAIL("the collection was not refused");
} catch (const JFJochException &e) {
CHECK(e.Category() == JFJochExceptionCategory::InputParameterInvalid);
CHECK_THAT(e.what(), Catch::Matchers::ContainsSubstring("has 12 slots (6 for the pre-pass)"));
CHECK_THAT(e.what(), Catch::Matchers::ContainsSubstring("lower stills_prepass_images to 6"));
}
CHECK(pusher.Events().empty());
}
TEST_CASE("JFJochReceiverRugnux_Stills_BufferFullDuringPrePass", "[JFJochReceiverRugnux]") {
// Many more frames than half of a twelve-slot image buffer holds, all there before the pre-pass ends. A frame past
// the half waits for the pre-pass and the analysis to make room: every frame is written.
Logger logger(Catch::getResultCapture().getCurrentTestName());
RegisterHDF5Filter();
const int64_t images = 200;
DiffractionExperiment x = StillsExperiment(images);
PixelMask mask(x);
Stream stream(x, images);
const size_t buffer_MiB = 12 * x.GetImageBufferLocationSize() / (1024 * 1024) + 1;
RecordingPusher pusher;
AcquisitionDeviceGroup group;
JFJochReceiverService service(group, logger, pusher, buffer_MiB);
service.NumThreads(2);
service.Indexing(x.GetIndexingSettings());
service.RugnuxLive(true, nullptr, 4, 60, false, false, 1, 6);
stream.Frames(0, images);
stream.End();
@@ -689,8 +719,8 @@ TEST_CASE("JFJochReceiverRugnux_Stills_BufferFullDuringPrePass", "[JFJochReceive
const auto live = service.GetRugnuxStatus();
CHECK(live.state == RugnuxLiveState::Done);
CHECK(HasNote(live, "The image buffer has 12 slots, half of them for the frames waiting: the pre-pass runs on the first 6 images"));
REQUIRE(live.stills);
CHECK(live.stills->images_needed == 6);
CHECK(live.stills->images_analysed == out.status.images_sent);
}
@@ -725,7 +755,6 @@ TEST_CASE("JFJochReceiverRugnux_Stills_PrePassImages", "[JFJochReceiverRugnux]")
const auto live = service.GetRugnuxStatus();
CHECK(live.state == RugnuxLiveState::Done);
CHECK(HasNote(live, "Fewer than 6 strong frames in the first 5: header geometry used"));
CHECK(!HasNote(live, "The image buffer has"));
REQUIRE(live.stills);
CHECK(live.stills->images_needed == 5);
CHECK(live.stills->images_analysed == images);
+7 -5
View File
@@ -179,10 +179,12 @@ int main(int argc, char **argv) {
ImageBuffer buffer(buffer_MiB * 1024 * 1024);
buffer.StartMeasurement(x);
const int64_t slots = buffer.GetStatus().total_slots;
const int64_t prepass_held = std::min(prepass_images, slots / 2);
if (prepass_held < std::min(prepass_images, n))
logger.Warning("The image buffer has {} slots, half of them for the frames waiting: the pre-pass runs on the "
"first {} images, not the {} asked (-P)", slots, prepass_held, std::min(prepass_images, n));
try {
CheckStillsPrePassFits(slots, x.GetImageBufferLocationSize(), std::min(prepass_images, n));
} catch (const JFJochException &e) {
logger.Error("{}", e.what());
return EXIT_FAILURE;
}
std::atomic<int64_t> analysed = 0, indexed = 0, spots = 0, given = 0;
std::atomic<bool> caught_up = false;
@@ -236,7 +238,7 @@ int main(int argc, char **argv) {
++indexed;
};
sink.settings = spot_finding;
StillsGPUAnalysis gpu(x, mask, start, &pool, nthreads, prepass_held, logger, sink, status,
StillsGPUAnalysis gpu(x, mask, start, &pool, nthreads, prepass_images, logger, sink, status,
merge, model, deadline_s);
JFJochReaderRawImage raw;