leonarski_fandClaude Opus 5.5
9656a3b235
The geometry walk re-integrates the whole sweep in every round, and on an
uncompressed SMV sweep that image loop is bound by memory traffic:
smv::ReadInto was the top function (33% of all user cycles on 2xfw), and
16 threads read 360 frames no faster than 1 thread. On Linux/macOS the
frame is now mapped and widened from the mapping, which drops the
read() copy of the whole file into a scratch buffer. Windows keeps the
ifstream read. Pixel values are unchanged; every pass benefits, not only
the walk.
Measured (RTX 5080 workstation, load 9-23 from other agents, 2 repeats,
base = rc175 0457cecaf, same flags):
2xfw WALL 48.15/48.33 -> 40.49/41.98 s, Geometry check 17.2/17.4 -> 13.6/13.8 s
6zqy WALL 26.10/.. -> 23.60/23.16 s, Geometry check 8.6 -> 7.1/7.2 s
2ygz WALL 32.05 -> 26.27 s
-N 8: 2xfw 50.09 -> 42.82 s, 6zqy 26.78 -> 22.97 s
peak RSS -0.5 GB (no scratch copy)
Microbenchmark, 360 2xfw frames, 16 threads: read 1.8-1.9 s, mmap 1.2-1.4 s.
p.hkl md5 identical to the base on every set run (2xfw, 6zqy, 2ygz, -N 8,
and the small SMV sets of the open arm).
Tried first and dropped: keeping the decoded frames in memory across the
walk's rounds. Exact, but filling 13.6 GB of fresh memory cost more than
the hits saved (2xfw round 1 +2.4 s, release +0.9 s, later rounds -0.8 s
each; CBF sets 5m17/7ph1 got slower), at +12.5 GB peak RSS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
93 lines
4.4 KiB
C++
93 lines
4.4 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <array>
|
|
#include <cstdint>
|
|
#include <map>
|
|
#include <optional>
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
// SMV: the format ADSC Quantum detectors wrote, and which Rayonix and several others still write.
|
|
// A plain ASCII header of "KEY=value;" lines between braces, then the pixels - no container, no
|
|
// compression, no binary header to decode by offset. The header states its own length, so even
|
|
// that is not assumed.
|
|
//
|
|
// {
|
|
// HEADER_BYTES= 512; DIM=2; BYTE_ORDER=little_endian; TYPE=unsigned_short;
|
|
// SIZE1=3072; SIZE2=3072; PIXEL_SIZE=0.102588; DISTANCE=250.000000;
|
|
// OSC_START=325.000000; OSC_RANGE=0.200000; WAVELENGTH=0.999839;
|
|
// BEAM_CENTER_X=157.500000; BEAM_CENTER_Y=157.500000;
|
|
// }
|
|
//
|
|
// Note the beam centre is in MILLIMETRES, not pixels, and its convention is the one thing writers
|
|
// disagree about - see the comment on beam_x_mm in the struct.
|
|
//
|
|
// Rigaku's d*TREK writes the same container with a different vocabulary: the detector and the
|
|
// goniometer as named circles with their vectors (CCD_GONIO_VALUES, ROTATION_VECTOR, ...), the beam
|
|
// position in pixels, and its own overflow encoding for 16-bit pixels. A header naming
|
|
// DETECTOR_NAMES is read that way.
|
|
namespace smv {
|
|
|
|
struct Header {
|
|
int64_t nx = 0; // SIZE1, the fast dimension
|
|
int64_t ny = 0; // SIZE2, the slow dimension
|
|
size_t data_offset = 0; // HEADER_BYTES
|
|
bool little_endian = true;
|
|
int64_t bytes_per_pixel = 2; // TYPE: unsigned_short
|
|
double pixel_x_m = 0;
|
|
double pixel_y_m = 0;
|
|
double distance_m = 0;
|
|
// BEAM_CENTER_X/Y, converted to pixels here. Writers disagree about this field more than about
|
|
// any other: it is in millimetres, and which of the two is the fast direction is a convention
|
|
// rather than a rule. Where a file also carries the ADSC-era synonyms (BEAM_CENTRE_X, or the
|
|
// MOSFLM-ordered pair) they are read too. A run whose centre is wrong is recoverable - the
|
|
// beam-centre estimator measures it from the data - but a wrong one is not detectable here.
|
|
double beam_x_px = 0;
|
|
double beam_y_px = 0;
|
|
double wavelength_A = 0;
|
|
double start_angle_deg = 0; // OSC_START, falling back to PHI
|
|
double angle_increment_deg = 0; // OSC_RANGE
|
|
double two_theta_deg = 0;
|
|
double exposure_s = 0;
|
|
int64_t saturation = 0; // CCD_IMAGE_SATURATION: the value a saturated pixel carries
|
|
std::string detector; // DETECTOR_SN, where stated
|
|
std::string axis_name = "phi";
|
|
// d*TREK only (Rigaku CrystalClear): the geometry is stated as vectors rather than the ADSC
|
|
// scalars, in the imgCIF laboratory frame - z from the sample to the source, y up. Absent on an
|
|
// ADSC header, where two_theta_deg above carries the one detector circle there is.
|
|
std::optional<std::array<double, 3>> fast_direction;
|
|
std::optional<std::array<double, 3>> slow_direction;
|
|
std::optional<std::array<double, 3>> spindle_axis;
|
|
struct Circle {
|
|
std::array<double, 3> axis;
|
|
double angle_deg;
|
|
};
|
|
std::vector<Circle> detector_circles; // header order: the outermost circle first
|
|
// RAXIS_COMPRESSION_RATIO: a stored value above 32767 is (value - 32768) * ratio. 0 = not used.
|
|
int64_t overflow_ratio = 0;
|
|
std::map<std::string, std::string> raw; // every key, for anything not modelled above
|
|
};
|
|
|
|
// True if the path names something this reader can open: an SMV file, or a directory holding at
|
|
// least one. Reads a few hundred bytes at most.
|
|
bool CanRead(const std::string &path);
|
|
|
|
// The files of one sweep, in collection order. path is a directory, or one frame inside it.
|
|
std::vector<std::string> CollectSweep(const std::string &path);
|
|
|
|
// Header only - reads just the leading brace block. Cheap enough to call per frame.
|
|
Header ReadHeader(const std::string &path);
|
|
|
|
// Header + pixels. out is resized to nx * ny.
|
|
Header Read(const std::string &path, std::vector<int32_t> &out);
|
|
|
|
// The same, widening the stored pixels into memory the caller already has. scratch carries the
|
|
// raw bytes between frames so a worker walking a sweep allocates once - on Windows only; elsewhere the
|
|
// file is mapped and scratch is not used.
|
|
Header ReadInto(const std::string &path, int32_t *out, size_t capacity, std::vector<uint8_t> &scratch);
|
|
|
|
} // namespace smv
|