10 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	8e6ecedd5c	CPU pixel loops: keep the ring sums and the vertical slide out of memory for real ... Two fixes found in the post-merge profile, where the previous commit's loops did not do what they were written to do: - AccumulateRingsBlock: the ring_overflow.emplace_back call inside the loop left no callee-saved registers for the locals, so GCC kept sum/sum2/count on the stack and the store-reload chain was still there (29% of the function's samples on one stack add). The azint sums get a loop of their own over the block, and the out-of-histogram values are listed by a second loop run only when the block has any, in the same pixel order; now the sums live in registers. Same additions in the same order. - DetectPass: inlined into DetectPass's main loop the vertical slide was not vectorised (the vectorised copies GCC made were for the other call sites). It is now a free function, SlideVertical, which GCC vectorises on its own. Measured (perf, 75 s of myob, CPU-only, relative to the unchanged FlagRow/AnalyzeBlock): AccumulateRingsBlock -4..-11%, DetectPass + SlideVertical -7..-14%. Byte-identical p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz on myob, cytc, lyso, sparse (CPU) and myob (GPU); ImageSpotFinderCPU*, AdaptiveSpotFinder, SpotFinding, AzimuthalIntegration and portable tests pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 03:57:10 +02:00
leonarski_fandClaude Opus 5.5	a277b8c458	Merge rc173 (fused CPU pixel pipeline) into cpu-loops-tail ... Conflicts in the three CPU pixel loops the fused pipeline split per block, resolved by carrying this branch's loop bodies into the new functions with the per-pixel arithmetic unchanged: - AdaptiveSpotFinderCPU: the register-held ring sums now live in AccumulateRingsBlock (stored at the end of each block, so the additions stay in pixel order). - ImagePreprocessorCPU::AnalyzeBlock reads the PixelMask-derived 32-pixel mask words at first + i. - ImageSpotFinderCPU::DetectPass: rc173's new_row() marking/fill_row calls kept in place, the vertical update replaced by the vectorised slide with the prev_strong fix-up. Byte-identical p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz vs rc173 references on myob, cytc, lyso, sparse (CPU-only build) and myob (GPU build); targeted tests (ImageSpotFinderCPU*, AdaptiveSpotFinder, SpotFinding, PixelMask, Bragg*, RotationScale, AzimuthalIntegration, portable) pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 02:43:16 +02:00
leonarski_fandClaude Opus 5.5	c8692e320c	CPU pixel loops: ring sums held in registers, vectorised vertical window, shared packed mask ... AdaptiveSpotFinderCPU::AccumulateRings: consecutive pixels mostly share a ring, so the ring's sum/sum2/count and the fused azint sums are held in locals while they do and stored when the ring changes - the same additions in the same order (az_sum2 is still contracted to the same FMA), without a store-and-reload chain through memory on every pixel. ImageSpotFinderCPU::DetectPass: the vertical-sum update (add the entering row, take out the leaving one) is one branch-free loop over the raw image that GCC vectorises (int64 lanes). A pixel strong in the previous pass used to be substituted per pixel through a bit test, which kept the loop scalar; it is now added with its row and taken out again from the few set bits of prev_strong. Integer sums, so the same totals. (A first, fully branch-free version that kept the per-pixel bit test did not vectorise on the prev_strong path and was measured slower; this is its replacement.) ImagePreprocessorCPU: the per-engine std::vector<bool> built bit by bit from the 32-bit mask (~10 core-s per cytc run, one per worker per pass) is replaced by 32-pixel mask words that PixelMask derives once beside its binary mask; each engine copies 2 MB. A branch-free rewrite of the Analyze loop was measured and dropped: the loop is bound by reading the decompressed image (330 vs 328 core-s on cytc), so only the mask test changed. Measured (perf, 499 Hz, CPU-only build, cytc, first version of this change): AccumulateRings 591 -> 539 core-s. Byte-identical p.hkl, p.mtz, p_P1.mtz, p_unmerged.mtz on myob, cytc, lyso, sparse (CPU) and myob, lyso (GPU). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 02:19:31 +02:00
leonarski_fandClaude Opus 5.5	9c5141c7c0	CPU pixel pipeline: decode, preprocess and ring pass per bitshuffle block; flag rings in the local sweep ... The CPU-only image loop is DRAM-bound on 16M frames: decode, preprocess, the adaptive finder's plain ring pass and FlagRings each streamed the whole frame through memory. - JFJochDecompressHperfBlocks hands each decoded bitshuffle block to a callback; with no output buffer the block is unshuffled into a reused block-sized scratch (JFJochDecompressBlocks). - MXAnalysisWithoutFPGA::PreprocessCPU preprocesses each block into the int32 buffer (ImagePreprocessorCPU::AnalyzeBlock) and, when the fused CPU finder runs, puts it through the plain ring pass + fused azint (AdaptiveSpotFinderCPU::AccumulateRingsBlock) while it is in cache. Detect() then starts from those sums. The per-worker decompression buffer is no longer allocated for bitshuffled data. - FlagRings becomes FlagRow, called by DetectAt's first pass for row y+NBX just before that row enters the vertical sums; first_pass_needed is marked from each row's candidates at the same point. Exact: blocks arrive in pixel order, so the float azint sums see the same pixels in the same order; the per-pixel expressions are unchanged; everything else is integer. p.hkl, p.mtz, p_P1.mtz and p_unmerged.mtz byte-identical to rc173 on myob, cytc, lyso, sparse (CPU-only build), GPU myob identical (the GPU path does not take this route). CPU-only, 32 workers, under the gpulock on a shared (loaded) machine, base -> fused, two rounds (second in reversed order): myob 155.9 -> 97.7 s, 139.9 -> 88.1 s (loop 46.1 -> 26.8 s/pass; user 3690 -> 2221 s) cytc 220.8 -> 156.6 s, 216.6 -> 155.1 s (user 5160 -> 4128 s) lyso 134.5 -> 123.3 s, 69.5 -> 64.8 s peak RSS myob 15.2 -> 10.8 GB, cytc 12.6 -> 10.4 GB, lyso 7.0 -> 6.7 GB Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 02:00:50 +02:00
leonarski_fandClaude Opus 5.5	d3afe0cb07	ImageSpotFinderCPU: the first pass slides its window only where its result is read ... With candidates, the first pass already tested only the 32-column blocks within reach of a candidate, but still slid the horizontal window across every column of every row. It now slides it only over the runs of needed blocks, starting each run from the vertical sums it covers - integers, so the same window sums - and sets the bits there directly; the rest stay 0 as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 23:01:21 +02:00
leonarski_fandClaude Opus 5.5	87cfc879b6	CPU adaptive spot finder: sigma clips from a histogram, first pass only where read ... Two whole-image passes per frame out of the CPU spot finder (the pre-scan's finder on every build, and every image on the CPU-only build): - AccumulateRings ran three passes over the frame - the plain ring statistics and two sigma clips. The plain pass now also counts each ring's valid values in a histogram (0..1023, the rest in a short list), and the clip passes sum over the distinct values: each meets the same float test its pixels would, and the sums are integers, so the totals are the same. - The local test's first pass is read by DetectAt only inside a candidate's window. It now marks the row/32-column blocks those windows reach and keeps its sliding sums everywhere but skips the per-pixel test elsewhere; the bits it leaves unset are never read. md5-identical output on four sets (GPU) and three (CPU-only). CPU-only 16M: 203 -> 177 s and 303 -> 263 s; GPU 16M 23.8 -> 22.8 s (the pre-scan's finder). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 20:40:46 +02:00
leonarski_fandClaude Opus 5.5	50098d20ba	CPU adaptive spot finder: second local pass only at the ring pixels ... The adaptive finder keeps (local test & ring threshold), so the local test's second pass is only wanted at the ring pixels. Its window there is the first pass's window with the first pass's strong pixels taken out, and those are few: the first pass now records its window sums at the ring pixels, and the second pass subtracts the strong pixels in each window instead of sweeping the whole image again. Integer sums and the same acceptance test (factored into StrongInWindow), so the bits are exactly the dense pass's. CPU-only rotation run of a 16M set: md5-identical output, 262 -> 242 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 17:30:44 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	c981e1b91c	v1.0.0-rc.137 (#46) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Better track time for each operation in the processing stack * jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress) * jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress) * jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0) Reviewed-on: #46	2026-04-25 19:59:21 +02:00
leonarski_f	061152279c	v1.0.0-rc.91	2025-10-20 20:43:44 +02:00