stills geometry sample and trial spot finding run as many workers as the image pass
The geometry refinement's sample (up to 10000 frames: spot finding + indexing on the card, the same per-frame work as the image pass) and the stills geometry trial's 96-frame spot finding ran -N workers (32 here), each with its own GPU engines. The image pass caps the workers that drive a card at 16 per card, fewer where their engines would not fit half the card (GPUWorkersPerCard), because past that a card stops taking queue work and more workers make the loop slower. The cap is now one function, ImageWorkers(), used by all three loops. On a 16.9 Mpx JUNGFRAU 16M the 32 sample workers held ~13 GB of a 16 GB card's engines (24 B/px each) where the cap allows 19. Which frames the sample examines does not depend on the worker count (fixed stripes), and each trial frame is found on its own: outputs identical (JF16M int16 15 runs: same 5174 frames sampled, same module placement, geometry and merge). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
This commit is contained in:
1 parent
96322a8ec9
commit
b87b015253
3 files changed
+26
-18
No files matched your search
@@ -1439,6 +1439,9 @@ class Rugnux {
|
||||
// How many workers with GPU engines of their own - at most `most` - one card takes: decided from
|
||||
// the card's total memory and the detector, never from what is free (RugnuxImagePass.cpp).
|
||||
[[nodiscard]] int GPUWorkersPerCard(int most) const;
|
||||
// How many workers a loop analysing whole frames on the cards runs - the image pass, and the stills
|
||||
// samples ahead of it (RugnuxImagePass.cpp).
|
||||
[[nodiscard]] int ImageWorkers() const;
|
||||
// The FFT indexers of a first pass's pool: two, for its two schemes at once; one for a probe pass on a
|
||||
// card too small for it beside the main line, which then indexes the schemes one after the other.
|
||||
// An FFT indexer holds 0.3-0.6 GB of the card, and which one indexes a frame does not change what it
|
||||
|
||||
+19
-15
@@ -95,6 +95,21 @@ int Rugnux::GPUWorkersPerCard(int most) const {
|
||||
return static_cast<int>(std::clamp<int64_t>(fit, 1, most));
|
||||
}
|
||||
|
||||
// Every worker submits its own kernels to a card, and a card runs out of room to accept them long before it
|
||||
// runs out of work: measured on two GPUs, the image loop's own time falls to sixteen workers and then rises
|
||||
// again, so forty-eight is slower than sixteen on a 16 Mpx set. Sixteen is where that measurement put the
|
||||
// minimum, and a later one on a single card agrees - eight leaves the loop waiting on the queue rather than
|
||||
// the card. The cap is per card because that is what the queue depth belongs to, and it only applies when -N
|
||||
// was left at its default - an explicit -N is a deliberate instruction and is obeyed, which is what a
|
||||
// previous attempt at this got wrong. A worker also holds GPU memory, so a small card gets fewer of them
|
||||
// (GPUWorkersPerCard).
|
||||
int Rugnux::ImageWorkers() const {
|
||||
const int gpus = get_gpu_count();
|
||||
if (!config_.nthreads_auto || gpus == 0)
|
||||
return config_.nthreads;
|
||||
return std::min(config_.nthreads, GPUWorkersPerCard(16) * gpus);
|
||||
}
|
||||
|
||||
void Rugnux::ImagePass(PipelineLocals &p) {
|
||||
auto &logger = p.logger;
|
||||
auto &result = p.result;
|
||||
@@ -374,22 +389,11 @@ void Rugnux::ImagePass(PipelineLocals &p) {
|
||||
|
||||
std::function<void()> worker = per_image_analysis ? std::function<void()>(full_worker)
|
||||
: std::function<void()>(azint_worker);
|
||||
// How many workers the loop actually wants. Every one of them submits its own kernels to a card,
|
||||
// and a card runs out of room to accept them long before it runs out of work: measured on two
|
||||
// GPUs, the loop's own time falls to sixteen workers and then rises again, so forty-eight is
|
||||
// slower than sixteen on a 16 Mpx set. Sixteen is where that measurement put the minimum, and a
|
||||
// later one on a single card agrees - eight leaves the loop waiting on the queue rather than the
|
||||
// card. The cap is per card because that is what the queue depth belongs to, and it only applies
|
||||
// when -N was left at its default - an explicit -N is a deliberate instruction and is obeyed,
|
||||
// which is what a previous attempt at this got wrong.
|
||||
// Only the full-analysis worker drives a card; the azimuthal one preprocesses and integrates on
|
||||
// the CPU and wants every thread it can have, so the cap must not reach it.
|
||||
// A worker also holds GPU memory, so a small card gets fewer of them (GPUWorkersPerCard).
|
||||
// How many workers the loop actually wants (ImageWorkers). Only the full-analysis worker drives a
|
||||
// card; the azimuthal one preprocesses and integrates on the CPU and wants every thread it can have,
|
||||
// so the cap must not reach it.
|
||||
const int gpus = get_gpu_count();
|
||||
const int workers_per_card = gpus > 0 ? GPUWorkersPerCard(16) : 16;
|
||||
const int image_workers = (per_image_analysis && config_.nthreads_auto && gpus > 0)
|
||||
? std::min(config_.nthreads, workers_per_card * gpus)
|
||||
: config_.nthreads;
|
||||
const int image_workers = per_image_analysis ? ImageWorkers() : config_.nthreads;
|
||||
if (image_workers < config_.nthreads)
|
||||
logger.Info("Per-image loop: {} of {} threads ({} GPUs) - past what a card can take queue "
|
||||
"work from, more workers make it slower, not faster; pass -N to override",
|
||||
|
||||
@@ -234,10 +234,11 @@ void Rugnux::RefineStillsGeometry(int start_image, int end_image, int images_to_
|
||||
}
|
||||
};
|
||||
|
||||
// As many workers as the image pass: each drives the card the same way, and how many there are does not
|
||||
// change which frames are examined (the stripes above).
|
||||
TimingMark sampling_mark(logger, "geometry refinement sample");
|
||||
std::vector<std::future<void>> futures;
|
||||
futures.reserve(config_.nthreads);
|
||||
for (int i = 0; i < config_.nthreads; ++i)
|
||||
for (int i = 0; i < ImageWorkers(); ++i)
|
||||
futures.push_back(std::async(std::launch::async, worker));
|
||||
for (auto &fut : futures)
|
||||
fut.get();
|
||||
@@ -459,7 +460,7 @@ std::vector<DataMessage> Rugnux::FindStillsSampleSpots(const std::vector<int> &o
|
||||
}
|
||||
};
|
||||
std::vector<std::future<void>> futures;
|
||||
for (int i = 0; i < config_.nthreads; ++i)
|
||||
for (int i = 0; i < ImageWorkers(); ++i)
|
||||
futures.push_back(std::async(std::launch::async, worker));
|
||||
for (auto &f : futures)
|
||||
f.get();
|
||||
|
||||
Reference in new issue
Block a user