diff --git a/docs/JFJOCH_BROKER.md b/docs/JFJOCH_BROKER.md index d34c50e19..7e4ae2ecf 100644 --- a/docs/JFJOCH_BROKER.md +++ b/docs/JFJOCH_BROKER.md @@ -383,9 +383,10 @@ Frames arriving during the pre-pass, or faster than the analysis, wait in at mos of its slots hold a frame each (it can be seen through the image routes while it waits), the rest of that half holds frames as their compressed bytes, several to a slot; the other half is for the analysed frames, which go to the writer from a slot each, with one slot kept per analysis thread. The receiver's image memory is the image -buffer (`image_buffer_MiB`, `-B` of `jfjoch_fpga_stills_test`) and nothing beside it. A frame that finds the waiting -half full is dropped and counted as for a full image buffer, so analysis slower than the frame rate costs frames, -not memory. Where the writer lags behind, the analysis +buffer (`image_buffer_MiB`, `-B` of `jfjoch_fpga_stills_test`) and nothing beside it. No stage on the host discards a +frame: a frame that finds the waiting half full waits for room, and the wait goes back to the source - from FPGA +cards, `FPGAImagePuller` holds the cards' buffers and the cards wait for the host, so frames are lost only at the +card. Analysis slower than the frame rate thus costs frames at the card, not memory. Where the writer lags behind, the analysis waits for slots to come back. A pre-pass that fails leaves the stated geometry and mask in use. Where the start message cannot be sent after the pre-pass (no writer listening) or the analysis cannot be set up, the collection ends with that error, reported by `/status` and `/wait_till_done`, and the diff --git a/docs/TOOLS.md b/docs/TOOLS.md index c1baf283d..92b53c069 100644 --- a/docs/TOOLS.md +++ b/docs/TOOLS.md @@ -74,7 +74,7 @@ synthesis C model on the CPU, so no FPGA device is required. Measures rugnux live stills on the FPGA card(s), with a stored dataset standing in for the detector: ``` -jfjoch_fpga_stills_test [-i ] [-s ] [-N ] [-B ] [-C a,b,c,al,be,ga] [-X ] [-r ,...] [-p ] [-k ] [-m ] [-d] [-v] [-K[]] +jfjoch_fpga_stills_test [-i ] [-s ] [-N ] [-B ] [-C a,b,c,al,be,ga] [-S ] [-X ] [-t ] [-k ] [-m ] [-v] [-K[]] ``` - The dataset must be a stored JUNGFRAU dataset with its modules assembled (1030 x 514 each, any placement @@ -88,7 +88,7 @@ jfjoch_fpga_stills_test [-i ] [-s ] [-N ] [-B ] [-C - The frames are loaded into the card's internal frame generator and the run is made with the JUNGFRAU conversion off, so the card returns them verbatim; the host assembles them as signed 16-bit photon counts and rugnux live stills analyses them through `FPGAImagePuller`, as the broker does with rugnux live - on (`-d`: not blocking, as the broker runs; `-v`: each card's progress every 0.5 s; `-k `: analyse 1 of every + on (`-v`: each card's progress every 0.5 s; `-k `: analyse 1 of every F images on average, the rugnux live setting `stills_downsample`; `-m `: N frames summed into each image, by the card up to 256 and above that the largest divisor of N up to 256 on the card and the rest on the host, 32-bit images). Beam centre, distance, energy and the cell @@ -101,30 +101,17 @@ jfjoch_fpga_stills_test [-i ] [-s ] [-N ] [-B ] [-C apply here) and prints the mapping. These are the first ones of the file, cycled for the whole run of `-i` images (default: as many as the file has). **Images repeat**, so throughput, latency, spots per image and indexing rate are meaningful, the merge and its statistics are not. -- The run blocks at the source, no image is dropped: `FPGAImagePuller` waits for a free slot instead of - dropping an image (and gives no module up for being behind the others), the stills receiver waits for - room in the image buffer, so the card keeps its frames, its host writer waits for host buffers and the - stall reaches the internal generator through the card's AXI-stream back-pressure (the generator sits - before the network stack; a real detector's UDP packets would be lost at that point instead). The - firmware cannot pace the generator (it has no frame period): `-r` paces it from the host, the puller - taking image *i* from the cards no earlier than *i / rate* after the start, so the card waits for the - host either way. One collection is run per rate of `-r` (default `0`, as fast as it goes). `-r` is refused - with `-d`: holding a frame on the host makes the card wait, which is blocking again, and dropping the frames - that come early would keep a different subset on each card, as their generators run apart. +- One collection per invocation, as fast as it goes. Nothing on the host discards an image: `FPGAImagePuller` + waits for room in its queue to the receiver and the stills receiver for room in the image buffer, so the card + keeps its frames, its host writer waits for host buffers and the stall reaches the internal generator through + the card's AXI-stream back-pressure (the generator sits before the network stack; a real detector's UDP + packets would be lost at that point instead, the only place frames are lost). The card's own count of the + cycles it waited for host buffers is printed per card: how long the chain behind it held the source. - The cards' generators run free, each as fast as its modules' bandwidth allows, so cards with different - numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) drift apart - unlike the modules of a detector, which a - common trigger keeps together - and with `-d` the images started by the cards ahead hold the puller's slots - until the frames of the card behind arrive. Where the modules do not divide evenly, every card is therefore - given as many modules as the largest, the rest dummy modules (zeros) the generator sends and the puller does - not read (`FPGAImagePuller::CardExperiment`; not part of the image, nor of its efficiency); `-U` leaves them - out. The start lines give each card's modules and dummies; the distinct images follow the padded count. With - `-U` (or cards that still drift, `-v` shows each card's frames) give `-p` at least the frames they drift apart. -- For each rate it reports the rate images were delivered (first to last image), the fraction of that time - the source was blocked by the chain behind it (the time the puller's module threads waited for a free - slot, mean over the modules), the sustained rate (images / wall time, start to the end of the analysis) - and the analysis rate after the pre-pass, and at the end a table of them: the highest rate with ~0% - blocked is the rate the live stills chain sustains. The card's own count of the cycles it waited for host - buffers is printed too; with `-r` it includes the pacing. + numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) run at different speeds - unlike the modules of a detector, + which a common trigger keeps together; the run then goes at the pace of the slowest card. +- It reports the rate images were delivered (first to last image), the sustained rate (images / wall time, + start to the end of the analysis) and the analysis rate after the pre-pass. - At the end it prints images collected/sent/dropped, receiving time and frame rate, CPU use, indexing rate and spots per image, per-image times of the analysis stages, the pre-pass (time, beam centre and distance it decided), the latency from the last image to the end of the analysis and rugnux's steps. diff --git a/tools/jfjoch_fpga_stills_test.cpp b/tools/jfjoch_fpga_stills_test.cpp index d33901c3d..f9f441b29 100644 --- a/tools/jfjoch_fpga_stills_test.cpp +++ b/tools/jfjoch_fpga_stills_test.cpp @@ -11,7 +11,6 @@ #include #include #include -#include #include #include @@ -40,16 +39,9 @@ void print_usage(Logger &logger) { logger.Info(" -S Space group of the collection (a space group stored in the file is ignored)"); logger.Info(" -X Indexing (auto|fft|fftw|ffbidx), auto is default"); logger.Info(" -t Indexing thread pool size (default: 8, as offline rugnux on stills; the broker's default is 4)"); - logger.Info(" -r Rates the source is paced at, one collection each (default: 0 = as fast as it goes); the"); - logger.Info(" host takes frame i no earlier than i/rate, the card waits meanwhile: blocking runs only"); - logger.Info(" -p Images FPGAImagePuller keeps compressed for the receiver (default 128)"); - logger.Info(" -U No dummy modules: where the cards carry different numbers of modules, they run apart"); logger.Info(" -k Analyse 1 of every images on average (rugnux live stills_downsample, default 1)"); logger.Info(" -m Frames summed into an image (default 1): the card sums the largest divisor up to 256, the"); logger.Info(" host the rest (1000 = 250 on the card x 4 on the host)"); - logger.Info(" -d Not blocking, as the broker runs: images the host has no room for are dropped. The cards'"); - logger.Info(" generators run free, each at a speed that goes with its modules: where the modules do not"); - logger.Info(" divide evenly, each card is given dummy modules up to the largest count (-U: not)"); logger.Info(" -K{} Mock of the card instead of /dev/jfjoch* (optional: its generator memory, default 64 MiB ="); logger.Info(" 128 module frames)"); } @@ -94,9 +86,8 @@ int64_t GeneratorImages(AcquisitionDeviceGroup &aq_devices, const DiffractionExp // A stored dataset through the card and rugnux live stills; returns false where the run failed bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::string &file, std::optional nimages, uint16_t nstreams, uint16_t nthreads, size_t send_buffer_size_MiB, - const IndexingSettings &indexing, const std::optional &cell, size_t puller_queue, - const std::vector &rates, bool dropping, bool pad, bool verbose, int64_t downsample, - const std::optional &space_group, int64_t summation) { + const IndexingSettings &indexing, const std::optional &cell, bool verbose, + int64_t downsample, const std::optional &space_group, int64_t summation) { RegisterHDF5Filter(); JFJochHDF5Reader reader; reader.ReadFile(file); @@ -133,18 +124,12 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s file_x.GetGemmiSpaceGroup()->hm); x.ImportIndexingSettings(indexing); - // The generator's speed goes with the modules of its card: with the modules not dividing evenly, every card is - // given as many as the largest, the rest dummies the puller does not read (FPGAImagePuller::CardExperiment) - const int64_t modules_per_card = (pad && (x.GetModulesNum() % nstreams != 0)) ? x.GetModulesNum(0) : 0; - DiffractionExperiment card_x = FPGAImagePuller::CardExperiment(x, modules_per_card); - - const int64_t generator_images = GeneratorImages(aq_devices, card_x, reader.GetNumberOfImages()); + const int64_t generator_images = GeneratorImages(aq_devices, x, reader.GetNumberOfImages()); if (generator_images < 1) { - logger.Error("The card memory cannot hold one image of {} modules", card_x.GetModulesNum(0)); + logger.Error("The card memory cannot hold one image of {} modules", x.GetModulesNum(0)); return false; } x.InternalPacketGeneratorImages(generator_images, FPGA_INTERNAL_GENERATOR_MAX_IMAGES); - card_x.InternalPacketGeneratorImages(generator_images, FPGA_INTERNAL_GENERATOR_MAX_IMAGES); PixelMask mask(x); if (dataset->pixel_mask->GetMask().size() == x.GetPixelsNum()) @@ -153,17 +138,9 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s logger.Info("Dataset {}: {} images of {} modules, {} distinct ones in the generator, {} in the run", file, reader.GetNumberOfImages(), x.GetModulesNum(), generator_images, x.GetImageNum()); for (int i = 0; i < x.GetDataStreamsNum(); i++) - logger.Info(" card {}: modules {}-{} and {} dummy, generator memory {} module frames, {} used", i, + logger.Info(" card {}: modules {}-{}, generator memory {} module frames, {} used", i, x.GetFirstModuleOfDataStream(i), x.GetFirstModuleOfDataStream(i) + x.GetModulesNum(i) - 1, - card_x.GetModulesNum(i) - x.GetModulesNum(i), aq_devices[i].GetInternalGeneratorModuleFrames(), - card_x.GetModulesNum(i) * generator_images); - if (dropping && !pad) { - for (int i = 1; i < x.GetDataStreamsNum(); i++) - if (x.GetModulesNum(i) != x.GetModulesNum(0)) { - logger.Warning("The cards carry different numbers of modules: their generators run apart (-U)"); - break; - } - } + aq_devices[i].GetInternalGeneratorModuleFrames(), x.GetModulesNum(i) * generator_images); for (int m = 0; m < x.GetModulesNum(); m++) logger.Debug(" module {}: pixel 0 at ({}, {}), fast step {}, slow step {}", m, x.GetPixel0OfModuleConv(m) % x.GetXPixelsNum(), x.GetPixel0OfModuleConv(m) / x.GetXPixelsNum(), @@ -175,146 +152,110 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s service.Indexing(x.GetIndexingSettings()); service.RugnuxLive(true, nullptr, 4, 60, false, false, downsample); - // Error, gap and beam-stop pixels of the reader become error pixels of the module frame. Card by card: each - // card's modules of the image, then its dummies (zeros) + // Error, gap and beam-stop pixels of the reader become error pixels of the module frame std::vector frame(x.GetModulesNum() * RAW_MODULE_SIZE); - std::vector card_frame(card_x.GetModulesNum() * RAW_MODULE_SIZE); + std::vector raw_frame(x.GetModulesNum() * RAW_MODULE_SIZE); for (int64_t i = 0; i < generator_images; i++) { const auto image = reader.LoadImage(i); SqueezeToRawGeometry(x, frame.data(), image->Image().data(), BEAM_STOP_PXL_VALUE, SATURATED_PXL_VALUE, INT16_MIN, INT16_MAX); - for (int d = 0; d < x.GetDataStreamsNum(); d++) - memcpy(card_frame.data() + card_x.GetFirstModuleOfDataStream(d) * RAW_MODULE_SIZE, - frame.data() + x.GetFirstModuleOfDataStream(d) * RAW_MODULE_SIZE, - x.GetModulesNum(d) * RAW_MODULE_SIZE * sizeof(int16_t)); - service.LoadInternalGeneratorImage(card_x, card_frame, i); - } - - struct Row { - double rate_hz, delivered_hz, blocked, sustained_hz, after_prepass_hz, indexing_rate; - uint64_t images, dropped; - }; - std::vector rows; - bool ok = true; - - for (const double rate: rates) { - // Blocking at source: the puller waits for a free slot and the receiver for room in the image buffer, - // the cards hold their frames and their generators stall - service.FPGAImagePullerSettings(puller_queue, !dropping, rate, false, modules_per_card); - logger.Info(""); - logger.Info("=== FPGAImagePuller (queue {}, {}), {} ===", puller_queue, dropping ? "dropping" : "blocking", - (rate > 0) ? fmt::format("source paced at {:.1f} Hz", rate) : std::string("source as fast as it goes")); - - rusage usage_start{}; - getrusage(RUSAGE_SELF, &usage_start); - const auto wall_start = std::chrono::steady_clock::now(); - - service.Start(x, mask, nullptr); - // -v: each card's progress, the frames of its fastest and slowest module written to the host - std::atomic running = true; - std::thread progress([&] { - while (running) { - std::this_thread::sleep_for(std::chrono::milliseconds(500)); - std::string line; - for (int i = 0; i < x.GetDataStreamsNum(); i++) - line += fmt::format(" card {}: {} .. {}{}", i, aq_devices[i].Counters().GetSlowestFrameNumber(), - aq_devices[i].Counters().GetFastestFrameNumber(), - aq_devices[i].Counters().IsAcquisitionFinished() ? " done" : ""); - // Where a run stands, so that one that stops shows which part waits - if (const auto st = service.GetStatus()) - line += fmt::format(" receiver: collected {} sent {} dropped {}", st->images_collected, - st->images_sent, st->images_dropped); - const auto live = service.GetRugnuxStatus(); - if (live.stills) - line += fmt::format(" stills: held {} analysed {} pre-pass {}", live.stills->images_held, - live.stills->images_analysed, static_cast(live.stills->prepass)); - if (verbose) - logger.Info("{:.1f} s{}", std::chrono::duration(std::chrono::steady_clock::now() - - wall_start).count(), line); - } - }); - const auto output = service.Stop(); - running = false; - progress.join(); - const auto live = service.GetRugnuxStatus(); - - const double wall_time = std::chrono::duration(std::chrono::steady_clock::now() - wall_start).count(); - rusage usage_end{}; - getrusage(RUSAGE_SELF, &usage_end); - auto seconds = [](const timeval &t) { return static_cast(t.tv_sec) + static_cast(t.tv_usec) / 1e6; }; - const double cpu_time = seconds(usage_end.ru_utime) - seconds(usage_start.ru_utime) - + seconds(usage_end.ru_stime) - seconds(usage_start.ru_stime); - - int64_t spots = 0, spot_images = 0; - for (const auto &e: output.scan_result.images) - if (e.spot_count) { - spots += e.spot_count.value(); - spot_images++; - } - - // From the first image to the last one into the receiver; the source was blocked for the time the module - // threads waited for a free slot, as a fraction of that span (mean over the modules) - const double delivery_s = live.last_image_s.value_or(0); - const double delivered_hz = (delivery_s > 0) ? static_cast(output.status.images_collected - 1) / delivery_s : 0; - const double blocked = (delivery_s > 0) ? service.GetFPGAImagePullerQueueWaitTime_s() / delivery_s : 0; - - logger.Info("Images expected {} collected {} sent {} dropped {} efficiency {:.4f}", - x.GetImageNum(), output.status.images_collected, output.status.images_sent, - output.status.images_dropped, output.efficiency); - logger.Info("Delivered {:.1f} Hz (first to last image {:.2f} s), source blocked {:.1f}% of it", - delivered_hz, delivery_s, 100.0 * blocked); - logger.Info("Sustained {:.1f} Hz (images / wall time {:.2f} s, start to analysis done) CPU {:.1f} s = {:.1f} cores" - " max resident memory {:.1f} GiB", - static_cast(output.status.images_collected) / wall_time, wall_time, cpu_time, - cpu_time / wall_time, static_cast(usage_end.ru_maxrss) / (1024.0 * 1024.0)); - // The card's own count of the cycles (200 MHz) it waited for a host buffer, i.e. was held by the host - for (int i = 0; i < x.GetDataStreamsNum(); i++) { - const auto status = aq_devices[i].GetDataCollectionStatus(); - logger.Info("Card {}: waited for host buffers {:.2f} s ({:.1f}% of the delivery), stalls before HBM {} cycles", - i, status.pipeline_stalls_host / 200e6, - (delivery_s > 0) ? 100.0 * status.pipeline_stalls_host / 200e6 / delivery_s : 0.0, - status.pipeline_stalls_hbm); - } - logger.Info("Indexing rate {:.3f} spots per image {:.1f}", output.status.indexing_rate.value_or(0), - static_cast(spots) / std::max(1, spot_images)); - logger.Info("Per-image time (mean; microseconds): preprocess {:.0f} spot finding {:.0f} indexing {:.0f} " - "refinement {:.0f} integration {:.0f} total {:.0f}", - output.processing_time.preprocessing * 1e6, output.processing_time.spot_finding * 1e6, - output.processing_time.indexing * 1e6, output.processing_time.refinement * 1e6, - output.processing_time.integration * 1e6, output.processing_time.processing * 1e6); - double after_prepass_hz = 0; - if (live.stills) { - const auto &st = *live.stills; - after_prepass_hz = st.analysis_rate_hz.value_or(0); - logger.Info("Pre-pass {:.2f} s: beam {:.2f} {:.2f} px (file {:.2f} {:.2f}), distance {:.2f} mm (file {:.2f})", - st.prepass_s.value_or(0), st.beam_x_pxl.value_or(NAN), st.beam_y_pxl.value_or(NAN), - st.file_beam_x_pxl, st.file_beam_y_pxl, st.distance_mm.value_or(NAN), st.file_distance_mm); - logger.Info("Analysed {} ({:.3f} of the images) indexed {} analysis rate after the pre-pass {:.1f} Hz", - st.images_analysed, static_cast(st.images_analysed) / x.GetImageNum(), - st.images_indexed, after_prepass_hz); - } - logger.Info("Rugnux: {} {} last image at {:.2f} s, done {:.2f} s after it", live.verdict, live.message, - live.last_image_s.value_or(NAN), live.last_image_to_written_s.value_or(NAN)); - for (const auto &step: live.steps) - logger.Info(" step {:.2f} s + {:.2f} s {}", step.started_s, step.duration_s.value_or(NAN), step.title); - - rows.push_back(Row{.rate_hz = rate, .delivered_hz = delivered_hz, .blocked = blocked, - .sustained_hz = static_cast(output.status.images_collected) / wall_time, - .after_prepass_hz = after_prepass_hz, - .indexing_rate = output.status.indexing_rate.value_or(0), - .images = output.status.images_collected, .dropped = output.status.images_dropped}); - ok = ok && (output.efficiency > 0) && (live.state == RugnuxLiveState::Done); + memcpy(raw_frame.data(), frame.data(), frame.size() * sizeof(int16_t)); + service.LoadInternalGeneratorImage(x, raw_frame, i); } logger.Info(""); - logger.Info("=== Rates ({} images, {} distinct: images repeat, the merge and its statistics are not a measure of the data) ===", - x.GetImageNum(), generator_images); - logger.Info(" source Hz delivered Hz blocked sustained Hz after pre-pass Hz indexing images dropped"); - for (const auto &r: rows) - logger.Info(" {:>9} {:>12.1f} {:>6.1f}% {:>12.1f} {:>17.1f} {:>8.3f} {:>6} {:>7}", - (r.rate_hz > 0) ? fmt::format("{:.1f}", r.rate_hz) : std::string("max"), r.delivered_hz, - 100.0 * r.blocked, r.sustained_hz, r.after_prepass_hz, r.indexing_rate, r.images, r.dropped); - return ok; + logger.Info("=== FPGAImagePuller, {} images ===", x.GetImageNum()); + + rusage usage_start{}; + getrusage(RUSAGE_SELF, &usage_start); + const auto wall_start = std::chrono::steady_clock::now(); + + service.Start(x, mask, nullptr); + // -v: each card's progress, the frames of its fastest and slowest module written to the host + std::atomic running = true; + std::thread progress([&] { + while (running) { + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + std::string line; + for (int i = 0; i < x.GetDataStreamsNum(); i++) + line += fmt::format(" card {}: {} .. {}{}", i, aq_devices[i].Counters().GetSlowestFrameNumber(), + aq_devices[i].Counters().GetFastestFrameNumber(), + aq_devices[i].Counters().IsAcquisitionFinished() ? " done" : ""); + // Where a run stands, so that one that stops shows which part waits + if (const auto st = service.GetStatus()) + line += fmt::format(" receiver: collected {} sent {} dropped {}", st->images_collected, + st->images_sent, st->images_dropped); + const auto live = service.GetRugnuxStatus(); + if (live.stills) + line += fmt::format(" stills: held {} analysed {} pre-pass {}", live.stills->images_held, + live.stills->images_analysed, static_cast(live.stills->prepass)); + if (verbose) + logger.Info("{:.1f} s{}", std::chrono::duration(std::chrono::steady_clock::now() + - wall_start).count(), line); + } + }); + const auto output = service.Stop(); + running = false; + progress.join(); + const auto live = service.GetRugnuxStatus(); + + const double wall_time = std::chrono::duration(std::chrono::steady_clock::now() - wall_start).count(); + rusage usage_end{}; + getrusage(RUSAGE_SELF, &usage_end); + auto seconds = [](const timeval &t) { return static_cast(t.tv_sec) + static_cast(t.tv_usec) / 1e6; }; + const double cpu_time = seconds(usage_end.ru_utime) - seconds(usage_start.ru_utime) + + seconds(usage_end.ru_stime) - seconds(usage_start.ru_stime); + + int64_t spots = 0, spot_images = 0; + for (const auto &e: output.scan_result.images) + if (e.spot_count) { + spots += e.spot_count.value(); + spot_images++; + } + + // From the first image to the last one into the receiver + const double delivery_s = live.last_image_s.value_or(0); + const double delivered_hz = (delivery_s > 0) ? static_cast(output.status.images_collected - 1) / delivery_s : 0; + + logger.Info("Images expected {} collected {} sent {} dropped {} efficiency {:.4f}", + x.GetImageNum(), output.status.images_collected, output.status.images_sent, + output.status.images_dropped, output.efficiency); + logger.Info("Delivered {:.1f} Hz (first to last image {:.2f} s)", delivered_hz, delivery_s); + logger.Info("Sustained {:.1f} Hz (images / wall time {:.2f} s, start to analysis done) CPU {:.1f} s = {:.1f} cores" + " max resident memory {:.1f} GiB", + static_cast(output.status.images_collected) / wall_time, wall_time, cpu_time, + cpu_time / wall_time, static_cast(usage_end.ru_maxrss) / (1024.0 * 1024.0)); + // The card's own count of the cycles (200 MHz) it waited for a host buffer, i.e. was held by the host: the + // source blocked + for (int i = 0; i < x.GetDataStreamsNum(); i++) { + const auto status = aq_devices[i].GetDataCollectionStatus(); + logger.Info("Card {}: waited for host buffers {:.2f} s ({:.1f}% of the delivery), stalls before HBM {} cycles", + i, status.pipeline_stalls_host / 200e6, + (delivery_s > 0) ? 100.0 * status.pipeline_stalls_host / 200e6 / delivery_s : 0.0, + status.pipeline_stalls_hbm); + } + logger.Info("Indexing rate {:.3f} spots per image {:.1f}", output.status.indexing_rate.value_or(0), + static_cast(spots) / std::max(1, spot_images)); + logger.Info("Per-image time (mean; microseconds): preprocess {:.0f} spot finding {:.0f} indexing {:.0f} " + "refinement {:.0f} integration {:.0f} total {:.0f}", + output.processing_time.preprocessing * 1e6, output.processing_time.spot_finding * 1e6, + output.processing_time.indexing * 1e6, output.processing_time.refinement * 1e6, + output.processing_time.integration * 1e6, output.processing_time.processing * 1e6); + if (live.stills) { + const auto &st = *live.stills; + logger.Info("Pre-pass {:.2f} s: beam {:.2f} {:.2f} px (file {:.2f} {:.2f}), distance {:.2f} mm (file {:.2f})", + st.prepass_s.value_or(0), st.beam_x_pxl.value_or(NAN), st.beam_y_pxl.value_or(NAN), + st.file_beam_x_pxl, st.file_beam_y_pxl, st.distance_mm.value_or(NAN), st.file_distance_mm); + logger.Info("Analysed {} ({:.3f} of the images) indexed {} analysis rate after the pre-pass {:.1f} Hz", + st.images_analysed, static_cast(st.images_analysed) / x.GetImageNum(), + st.images_indexed, st.analysis_rate_hz.value_or(0)); + } + logger.Info("Rugnux: {} {} last image at {:.2f} s, done {:.2f} s after it", live.verdict, live.message, + live.last_image_s.value_or(NAN), live.last_image_to_written_s.value_or(NAN)); + for (const auto &step: live.steps) + logger.Info(" step {:.2f} s + {:.2f} s {}", step.started_s, step.duration_s.value_or(NAN), step.title); + + return (output.efficiency > 0) && (live.state == RugnuxLiveState::Done); } int main(int argc, char **argv) { @@ -330,18 +271,14 @@ int main(int argc, char **argv) { std::optional cell; IndexingAlgorithmEnum indexing = IndexingAlgorithmEnum::Auto; uint16_t indexing_threads = 8; // as offline rugnux's stills (Rugnux::StillsIndexerThreads) - size_t puller_queue = FPGAImagePuller::DefaultQueue; std::optional mock_card_MiB; bool verbose = false; - std::vector rates = {0}; - bool dropping = false; - bool pad = true; int64_t downsample = 1; int64_t summation = 1; std::optional space_group; int opt; - while ((opt = getopt(argc, argv, "vi:s:N:B:C:S:X:t:r:p:dUk:m:K::")) != -1) { + while ((opt = getopt(argc, argv, "vi:s:N:B:C:S:X:t:k:m:K::")) != -1) { switch (opt) { case 'v': verbose = true; @@ -380,23 +317,6 @@ int main(int argc, char **argv) { case 't': indexing_threads = atol(optarg); break; - case 'r': { - rates.clear(); - std::stringstream list(optarg); - std::string rate; - while (std::getline(list, rate, ',')) - rates.push_back(atof(rate.c_str())); - break; - } - case 'p': - puller_queue = atol(optarg); - break; - case 'd': - dropping = true; - break; - case 'U': - pad = false; - break; case 'k': downsample = atol(optarg); break; @@ -425,13 +345,6 @@ int main(int argc, char **argv) { print_usage(logger); exit(EXIT_FAILURE); } - // The generator has no frame period: -r holds frames on the host, which makes the card wait - blocking again. - // Dropping the frames that come early instead would keep a different subset on each card, as they run apart. - if (dropping && std::any_of(rates.begin(), rates.end(), [](double r) { return r > 0; })) { - logger.Error("-r paces the source through the card waiting for the host, so it cannot be used with -d"); - exit(EXIT_FAILURE); - } - // -v gives the progress lines only: the debug log has a line for every module frame logger.Verbose(false); @@ -454,7 +367,7 @@ int main(int argc, char **argv) { bool ret = false; try { ret = RunDataset(logger, aq_devices, argv[optind], nimages, nstreams, nthreads, send_buffer_size_MiB, - i_settings, cell, puller_queue, rates, dropping, pad, verbose, downsample, space_group, + i_settings, cell, verbose, downsample, space_group, summation); } catch (std::exception &e) { logger.Error(e.what());