5 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	420750e0aa	Reader: remove the background read-ahead ... rugnux will mostly run on GPFS/Lustre-type storage. GPFS caches in its own fixed-size pagepool, often smaller than a dataset, so streaming the files ahead of the consumer can evict data before it is used; and choosing the behaviour by the underlying filesystem is not wanted. This removes the ReadAhead streamer of aa3fa6b9c: reader/ReadAhead.{h,cpp}, JFJochReader::StartReadAhead/NoteImageRead, the DataFiles() lists that only it used (HDF5 reader, HDF5ImageSource, HDF5ImageLocator, CBF/marCCD/SMV), and its start in rugnux_cli.cpp. Kept from the same commit: the in-order CBF/marCCD/SMV header scan (sweep::ForEachInOrder) and the 16 kB -> 256 kB CBF header probe, with its test. They only change the order and size of reads the program makes anyway, hold no memory and read less (cold 2400-frame CBF header scan 18 s -> 6 s on the measured HDD). Output unchanged: myob p.hkl, p.mtz, p_P1.mtz and p_unmerged.mtz byte-identical to the rc173 reference (17.1 s wall warm, 3.8 GB peak RSS). Reader tests: [HDF5] 108 cases, CBF/marCCD/SMV/sweep/VDS/GetRawImage cases all pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-27 18:48:10 +02:00
leonarski_fandClaude Opus 5.5	aa3fa6b9c7	Reader: stream the frame data into the page cache ahead of the image loops ... A cold run on a spinning disk waited on the disk twice over. The CBF header scan read 256 kB from every frame on eight threads that each strode through their own share of the sweep, so they drifted apart and the scan became a seek storm (34 s for 2400 frames here); and after it, the pre-scan and the first-pass indexing touch a few hundred frames and leave the disk idle until the first image loop reads everything at seek-bound rates. - ReadAhead (reader/): once the dataset is open, rugnux starts eight threads that read the data files - HDF5 data files (legacy, VDS or the integrated master) or the per-frame CBF/marCCD/SMV files - in 4 MB pieces taken strictly in order, into a throwaway buffer. One stream reads this disk at 125 MB/s, eight in-order streams at 190 MB/s, 32 at 157 MB/s. It never gets more than a quarter of MemAvailable (GlobalMemoryStatusEx on Windows, 4 GiB where there is no figure) ahead of what ReadRawImage has handed out, so a dataset bigger than the cache does not evict its own start, and it stops with the reader. Plain ifstream reads: portable, no POSIX calls. - Header scans (CBF, marCCD, SMV) hand the files out in order from an atomic counter (sweep::ForEachInOrder) instead of striding: 18 s -> 12 s for 2400 cold CBF headers. The CBF header is first read with a 16 kB probe and again with the old 256 kB one only when the separator is not in it, so the parsed header is exactly what it was: 12 s -> 6 s. Output unchanged: p.hkl, p.mtz and p_unmerged.mtz md5-identical to the rc173 baseline on 6toc (CBF, 2400 frames, 6.0 GB) and 9q41 (HDF5 VDS, 900 frames, 5.1 GB), and on 6z9g (HDF5, 12.8 GB) to the unmodified branch; myob (p.hkl p.mtz p_P1.mtz p_unmerged.mtz) md5-identical to the reference. Measured cold (files evicted with POSIX_FADV_DONTNEED before every run), same code without this commit vs with it, on a shared box (load 20-70, other agents reading the same disk, so single runs scatter by +-20 s): 6toc wall 61.7/62.7 -> 49.3/49.8 s (clean pairs); all data resident after 62/51/50 -> 45/41/42 s 9q41 wall 67.3 -> 57.8 s (clean pair); resident after 58/46/43 -> 48/37/35 s 6z9g resident after 81 -> 69 s The first image loop can look slower with this in CBF runs: the old 256 kB header probes pulled ~70% of the data in as kernel readahead, so the old loop started warm - after a 34 s header scan instead of 10 s. Warm (myob, NVMe, cached): 19.35/20.00 s without, 19.76-20.16 s with; the read-ahead then only copies 9.3 GB out of the page cache, 0.44 s wall and 3.4 CPU-s measured standalone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-27 10:32:58 +02:00
leonarski_fandClaude Opus 5	5f4343621c	Reader: SMV states a saturation value, and a pixel at it is saturated ... CCD_IMAGE_SATURATION is in the header of every SMV file the corpus holds, and it is the value a saturated pixel carries, as the marCCD field is. The reader never read it, and said in a comment that saturation was "judged on the 16-bit container alone" - which the line below it made untrue: the pixels are handed out as 32-bit, so the fallback limit was INT32_MAX and no SMV pixel could ever be called saturated, whatever the detector did. The limit is now the header's value, or the container's maximum where a writer states none - said explicitly rather than left to the fallback, for the reason above. One of the new open-arm sets carries pixels at exactly 65535. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-23 09:19:07 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00