leonarski_fandClaude Opus 5.5
2c1123c267
Calibration (--mode calibration): RebinAndRefit gave each worker one
AzimuthalIntegrationProfile that AzIntEngineCPU::Run clears on every
image, so the re-binned profile held only the last image each worker
read, added in completion order - on a 1800-frame LaB6 run the re-binned
fit changed with -N (197 / 181 / 205 ring points at rms ~7 px). Each
fixed block of images is now summed in image order into its own slot
(ParallelBlocks) and the slots added in block order, so the profile is
the sum over every image and the same at any thread count (rms 0.93 px,
identical at -N 1, 7 and 32). On the LaB6 runs with a sane header the
first pass still wins and the .poni is unchanged; with a header 100 px
off the re-binned fit is now adopted (218 vs 140 ring points). New test
Rugnux_CalibrationRebinsEveryImage spreads the rings over six frames
and fails on the old code (241 vs 141 points at -N 1 vs -N 4).
Host memory: out-of-memory now ends with "Processing failed: out of
host memory (...) - this data set needs more RAM than is available" and
exit 1. Tested with ulimit -v and an LD_PRELOAD allocator that refuses
allocations from a chosen phase (CPU and GPU builds, image loop through
merging and writing, calibration mode). Paths that crashed instead:
- PostIndexingRefinement ran candidate blocks on bare std::threads, so
a bad_alloc there called std::terminate (seen: "terminate called
recursively", SIGABRT); now std::async futures.
- FFTW aborts on its own failed allocation (CK(p) in kernel/alloc.c,
also inside buffered transforms: BeamCenterFFTCPU, FFTIndexerCPU);
rugnux now defines fftwf_assertion_failed to report it and exit 1.
- libjpeg's default error_exit calls exit() from the diagnostic-JPEG
thread ("Insufficient memory (case 12)"); WriteJPEGToMem now
longjmps back and throws (bad_alloc for JERR_OUT_OF_MEMORY).
- WorkerPool construction that fails to start a thread destroyed
joinable threads (terminate); it now joins them and rethrows.
Also: length_error counts as a fatal resource error, pinned host
allocation failure is MemAllocFailed, and the GPU scaling fail-fast
message says the CPU path needs a lot of host memory.
Docs: very large cells and what running out of host memory looks like
(including the OOM killer) in RUGNUX_INSTALL.md, pointer in RUGNUX.md.
Validation: myob, cytc, lyso, sparse md5-identical to 82e6583e0 on
both GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
89 lines
2.8 KiB
C++
89 lines
2.8 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include <vector>
|
|
#include <csetjmp>
|
|
#include <cstdio>
|
|
#include <cstdint>
|
|
#include <cmath>
|
|
#include <new>
|
|
#include <jpeglib.h>
|
|
#include <jerror.h>
|
|
|
|
#include "../common/JFJochException.h"
|
|
#include "../common/CompressedImage.h"
|
|
#include "JFJochJPEG.h"
|
|
|
|
namespace {
|
|
// libjpeg's own error handler prints its message and calls exit(), from whichever thread was
|
|
// encoding - out of memory included. This one jumps back to WriteJPEGToMem, which throws instead.
|
|
struct JPEGErrorManager {
|
|
jpeg_error_mgr mgr;
|
|
std::jmp_buf jump;
|
|
};
|
|
|
|
void JPEGErrorExit(j_common_ptr cinfo) {
|
|
std::longjmp(reinterpret_cast<JPEGErrorManager *>(cinfo->err)->jump, 1);
|
|
}
|
|
}
|
|
|
|
|
|
std::string WriteJPEGToMem(const CompressedImage& image, int quality) {
|
|
unsigned char *buf = nullptr;
|
|
unsigned long buf_size = 0;
|
|
|
|
if (image.GetMode() != CompressedImageMode::RGB)
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
|
|
"Only RGB images allowed for JPEG saving");
|
|
|
|
std::vector<uint8_t> buffer;
|
|
image.GetUncompressed(buffer);
|
|
|
|
struct jpeg_compress_struct cinfo;
|
|
JPEGErrorManager jerr;
|
|
|
|
cinfo.err = jpeg_std_error(&jerr.mgr);
|
|
jerr.mgr.error_exit = JPEGErrorExit;
|
|
if (setjmp(jerr.jump)) {
|
|
const bool out_of_memory = (jerr.mgr.msg_code == JERR_OUT_OF_MEMORY);
|
|
char message[JMSG_LENGTH_MAX];
|
|
jerr.mgr.format_message(reinterpret_cast<j_common_ptr>(&cinfo), message);
|
|
jpeg_destroy_compress(&cinfo);
|
|
free(buf);
|
|
if (out_of_memory)
|
|
throw std::bad_alloc();
|
|
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
|
|
std::string("JPEG encoding failed: ") + message);
|
|
}
|
|
jpeg_create_compress(&cinfo);
|
|
jpeg_mem_dest(&cinfo, &buf, &buf_size);
|
|
|
|
cinfo.image_width = image.GetWidth();
|
|
cinfo.image_height = image.GetHeight();
|
|
cinfo.input_components = 3;
|
|
cinfo.in_color_space = JCS_RGB;
|
|
|
|
jpeg_set_defaults(&cinfo);
|
|
/*set the quality [0..100] */
|
|
jpeg_set_quality (&cinfo, quality, true);
|
|
jpeg_simple_progression(&cinfo);
|
|
jpeg_start_compress(&cinfo, true);
|
|
|
|
JSAMPROW row_pointer; /* pointer to a single row */
|
|
|
|
while (cinfo.next_scanline < cinfo.image_height) {
|
|
row_pointer = (JSAMPROW) (buffer.data() + cinfo.next_scanline*(sizeof(char)*3)*image.GetWidth());
|
|
jpeg_write_scanlines(&cinfo, &row_pointer, 1);
|
|
}
|
|
|
|
jpeg_finish_compress(&cinfo);
|
|
jpeg_destroy_compress(&cinfo);
|
|
|
|
std::string ret;
|
|
if (buf_size > 0)
|
|
ret = std::string((char *)buf, buf_size);
|
|
free(buf);
|
|
|
|
return ret;
|
|
}
|