leonarski_fandClaude Opus 5
1d16dcd2b9
Build Packages / build:windows:nocuda (push) Successful in 16m8s
Build Packages / build:windows:cuda (push) Successful in 18m58s
Build Packages / build:viewer-tgz:cpu (push) Successful in 20m35s
Build Packages / build:viewer-tgz:cuda (push) Successful in 22m31s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 25m9s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 25m6s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 28m57s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 28m58s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 28m58s
Build Packages / XDS test (durin plugin) (push) Successful in 12m3s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 22m24s
Build Packages / build:rpm (rocky9) (push) Successful in 21m45s
Build Packages / Generate python client (push) Successful in 53s
Build Packages / build:rpm (rocky8) (push) Successful in 26m9s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 1m37s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m34s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 22m0s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m53s
Build Packages / XDS test (neggia plugin) (push) Successful in 9m29s
Build Packages / DIALS test (push) Successful in 23m40s
Build Packages / Unit tests (push) Successful in 1h20m1s
Three costs before and around the image loop. Every image allocated a fresh buffer for its compressed chunk and resized it, which value-initialises, and the read then overwrote every byte. At a few megabytes a chunk the allocation is large enough to be mapped rather than reused, so the zeroing was page-fault bound and cost more than the read it preceded - twenty gigabytes of it over a long sweep. The buffer now uses an allocator that does not construct, and the two HDF5 read paths are templated on the allocator so every existing caller compiles unchanged. The rebind is deliberate: without it the vector base rebinds to the default allocator and the zeroing quietly returns. The bitshuffle decoder allocated a whole uncompressed frame in its constructor - seventy megabytes a worker, five hundred and fifty across the loop - for the route that decodes the shuffled image separately. That route is taken only when a bitshuffle block is too large for the fused kernel, which neither writer this pipeline reads produces, so on a real frame the buffer is allocated, never touched, and freed. It is now allocated where it is used. The comment two lines below already warned against sizing a buffer from the uncompressed size; the line above it had not been given the same treatment. The first call into cuFFT pays the library's one-time initialisation, and it landed in the middle of the first pass with nothing to overlap it. It is now forced on a background thread at startup, alongside the file open and the mapping build, in the manner the shadow finder already uses. Finally, the detector mask was copied into the start message whether or not a file would carry it, which a merging run does not. It is filled where a writer is constructed - both places one is constructed, the second being the fallback that writes a process file when nothing indexed. Faster on eleven of thirty-eight crystals and slower on none; the whole rotation test set falls from four minutes thirty to four minutes seventeen, with each binary repeating itself to within half a per cent. Space groups thirty-five of thirty-eight and no failures throughout, and every column of the comparison table is identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EGpGdgmJ8MyY9pCGWjktyi
88 lines
3.1 KiB
C++
88 lines
3.1 KiB
C++
// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <vector>
|
|
#include <unordered_set>
|
|
#include <map>
|
|
#include <mutex>
|
|
#include <optional>
|
|
|
|
#include "../common/TopPixels.h"
|
|
#include "JFJochReaderDataset.h"
|
|
#include "../common/CrystalLattice.h"
|
|
#include "../common/Histogram.h"
|
|
|
|
// Markers stored in place of an intensity. They occupy the bottom of the int32 range and
|
|
// INT32_MAX at the top, so anything in between is a real count. Add a marker at the BOTTOM and move
|
|
// MIN_REAL_PXL_VALUE with it - several places classify a pixel by range rather than by equality,
|
|
// and they all test against MIN_REAL_PXL_VALUE.
|
|
constexpr static int32_t ERROR_PXL_VALUE = INT32_MIN;
|
|
constexpr static int32_t GAP_PXL_VALUE = INT32_MIN + 1;
|
|
constexpr static int32_t BEAM_STOP_PXL_VALUE = INT32_MIN + 2;
|
|
constexpr static int32_t MIN_REAL_PXL_VALUE = INT32_MIN + 3;
|
|
constexpr static int32_t SATURATED_PXL_VALUE = INT32_MAX;
|
|
|
|
struct JFJochReaderRawImage {
|
|
RawByteBuffer image_buffer;
|
|
CompressedImage image;
|
|
};
|
|
|
|
class JFJochReaderImage {
|
|
std::shared_ptr<const JFJochReaderDataset> dataset;
|
|
|
|
std::vector<int32_t> image; // Image in the reader must be 32-bit signed, uncompressed
|
|
DataMessage message;
|
|
|
|
std::unordered_set<int64_t> saturated_pixel;
|
|
std::unordered_set<int64_t> error_pixel;
|
|
std::vector<std::pair<int32_t, int32_t>> valid_pixel;
|
|
|
|
// Fast stats without storing/sorting all valid pixels
|
|
int32_t valid_min = 0;
|
|
int32_t valid_max = 0;
|
|
size_t valid_count = 0;
|
|
bool has_valid = false;
|
|
|
|
// For overlay: track top pixels with a tiny O(K) structure; export to vector for UI
|
|
TopPixels top_pixels_acc{20};
|
|
std::vector<std::pair<int32_t, int32_t>> top_pixels;
|
|
|
|
// This histogram operates in square root of count from 0 to 10^20
|
|
Histogram count_histogram{100000};
|
|
|
|
constexpr static float auto_foreground_range = 99.0f;
|
|
int32_t auto_foreground;
|
|
void CalcAutoContrast();
|
|
|
|
template <class T>
|
|
void ProcessInputImage(const void* image, size_t npixel, int64_t sat_value, int64_t special_value);
|
|
void ProcessInputImage(const CompressedImage& image);
|
|
public:
|
|
JFJochReaderImage(const DataMessage &msg, const std::shared_ptr<const JFJochReaderDataset> &dataset);
|
|
JFJochReaderImage(const JFJochReaderImage &other);
|
|
|
|
const DataMessage &ImageData() const;
|
|
DataMessage &ImageData();
|
|
|
|
const std::vector<int32_t> &Image() const;
|
|
const std::unordered_set<int64_t> &SaturatedPixels() const;
|
|
const std::unordered_set<int64_t> &ErrorPixels() const;
|
|
const JFJochReaderDataset &Dataset() const;
|
|
|
|
std::optional<std::pair<int32_t, int32_t>> ValidMinMax() const;
|
|
const std::vector<std::pair<int32_t, int32_t>> &GetTopPixels() const;
|
|
|
|
void AddImage(const JFJochReaderImage& other);
|
|
std::vector<float> GetAzInt1D() const;
|
|
std::vector<float> GetAzInt1D_BinToQ() const;
|
|
std::shared_ptr<JFJochReaderDataset> CreateMutableDataset();
|
|
|
|
int32_t GetAutoContrastValue() const;
|
|
|
|
std::vector<float> GetHistogram() const;
|
|
};
|
|
|
|
|