leonarski_fandClaude Opus 5.5
eb47013d4f
The main loop's read (ReadRawImage) now hands a camera frame up as the
file stores it - 16-bit pixels, UINT16_WIDENED, as the marCCD reader
does - wherever there is no pixel offset to take off and no gain to
divide by; the GPU preprocessor widens it on the device. The host no
longer widens 3.7 Mpx to int32 per frame (the loop that was ~79% of
the main pass's user cycles). xtc::ReadFrame becomes xtc::ReadPixels,
the stored pixels only; the int32 decode (LoadImage, or an offset or
gain) widens them in one plain loop.
The streams are indexed in parallel (one worker per stream, up to the
hardware threads), merged in the order given: each index is a walk of
small reads, waiting on the disk one at a time.
Measured on a Rayonix MX340 run, 1920x1920 (RTX 5080, 5950X, page
cache warm, other agents' builds running; output identical but for the
known run-to-run flip of one observation on one set, which the
baseline shows between its own runs):
main pass, 2000 frames: 479 -> 789 frames/s (lcls222),
413 -> 477, 388 -> 452 (the two forms of
lcls242, there indexing-bound)
Step 1, 48 streams/82k frames: 34-51 s -> 2.4 s with the headers
warm, 58.2 s -> 5.7 s evicted from cache
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
65 lines
3.2 KiB
C++
65 lines
3.2 KiB
C++
// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <cstdint>
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
// The LCLS-I XTC container, as far as reading camera frames out of it needs: what the LCLS data
|
|
// system (pdsdata) wrote at LCLS until 2023, and what serial-crystallography depositions from that
|
|
// era (Rayonix at MFX and XPP) are archived as.
|
|
//
|
|
// A file is a sequence of datagrams. Each is a 40-byte header - the event's clock time and timestamp
|
|
// (16 bytes, whose top bits name the transition: Configure, BeginRun, ..., L1Accept for an event),
|
|
// an environment word, and the outermost Xtc header - followed by the Xtc's payload. An Xtc header is
|
|
// five 32-bit words: damage, the source (two words), the type and version (bit 31: compressed), and
|
|
// the extent, which counts the header itself. A container (type Id_Xtc) holds further Xtcs one after
|
|
// another; anything else is a leaf holding one data object:
|
|
//
|
|
// - a camera image, Id_Frame (Camera::FrameV1): width, height, bit depth and an offset added to every
|
|
// pixel, 32 bits each, then the pixels row by row;
|
|
// - the electron beam, Id_EBeam version 7 (Bld::BldDataEBeamV7): a damage mask, then 20 doubles, the
|
|
// 18th the photon energy in eV;
|
|
// - a Rayonix configuration, Id_RayonixConfig version 2: the binning along fast and slow (a byte each)
|
|
// and, further on, the device name ("MX340-HS:...").
|
|
//
|
|
// Everything is little-endian and uncompressed. A run is written as several streams (-s00, -s01, ...)
|
|
// by different recorders, each holding a share of the events. Their order across streams is the event
|
|
// time, which this reader does not reconstruct: stills are independent, so the streams are read one
|
|
// after another.
|
|
namespace xtc {
|
|
|
|
// One camera image: where its pixels start in which file.
|
|
struct Frame {
|
|
std::string file;
|
|
uint64_t offset = 0; // of the first pixel
|
|
uint32_t width = 0;
|
|
uint32_t height = 0;
|
|
uint32_t pixel_offset = 0; // added to every pixel by the camera; taken off when read
|
|
uint64_t source = 0; // the camera, as the Xtc names it
|
|
};
|
|
|
|
// What one stream holds.
|
|
struct Index {
|
|
std::vector<Frame> frames; // every undamaged 16-bit image of every event
|
|
std::vector<double> photon_energy_eV; // of every event with an undamaged electron-beam record
|
|
std::string rayonix_model; // the Rayonix configuration's device name, if any
|
|
uint32_t rayonix_binning = 0;
|
|
size_t damaged = 0; // images left out: damaged, compressed or of another version
|
|
bool truncated = false; // the file ends inside its last datagram, which is left out
|
|
};
|
|
|
|
// True if the path is an XTC file - by its first datagram, not only its name. The "smalldata" copies
|
|
// (.smd.xtc) carry the events without their images and are not read.
|
|
bool CanRead(const std::string &path);
|
|
|
|
Index IndexFile(const std::string &file);
|
|
|
|
// Reads one image's pixels as stored - 16 bits each, the camera's offset still on - into out (width*height
|
|
// values).
|
|
void ReadPixels(const Frame &frame, uint16_t *out);
|
|
|
|
} // namespace xtc
|