6 Commits

Include renames

Author	SHA1	Message	Date
jungfrauandClaude Opus 5	debd73c0bd	Find the first pass's spots on every worker ... The first pass of rotation indexing feeds two sampling schemes and a validation set, and it found their spots one frame at a time on one CUDA stream. It reads 271 to 402 frames depending on the crystal, once per pass, and there are two passes. On a 16M-pixel dataset that is 6.7 s of a 30 s run at a mean occupancy of 1.9 threads out of 48. The comment said this was to keep the spot cache from depending on scheduling. The guarantee is stronger than that and survives: a frame's spots are a pure function of that frame, so whatever order the workers finish in, the cache ends up holding exactly what the serial loop put there. The feed order and the point at which the consecutive scheme stops accumulating are untouched, so the schemes see the same frames as before. The phase was never CPU-bound, which is why this works: its one thread was spin-waiting in the driver on the same six full-image passes, about 5.5 ms a frame. One engine per worker for the whole pass, not per batch. An engine on a 16M-pixel detector costs ~0.13 s to build - a first attempt rebuilt them per batch and came out SLOWER than the serial loop, four batches of eight engines against ~1.5 s of frame work. Two smaller things, in the same per-image path: The strong-pixel flagging kernel reads four pixels at a time and issues one atomicOr per four instead of per pixel. Same values, confirmed end to end. The beam-stop pre-scan sized its per-worker accumulators for the worker count rather than for the workers that use them. Two int64 arrays per pixel per shard, eight shards: 2.3 GB allocated and cleared on a 16M-pixel detector for a path the GPU never reads. They are allocated on use now, which is most of what made that phase look like eight workers with two of them running. Measured, three A/B pairs with the order alternated, whole run: 16M-pixel 3600-frame 42.8 s -> 38.8 s, 16M-pixel 1800-frame 38.7 s -> 36.1 s, both at two GPUs; a 4M-pixel control 23.3 s -> 23.1 s. First-pass indexing itself -44 % and -36 %. The two smaller datasets merge byte-identically over six runs each; the largest is not bit-reproducible in the unmodified binary either, and stayed inside that spread. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 18:06:03 -04:00
jungfrauandClaude Opus 5	6368c00173	Decode and accumulate the beam-stop projection on the GPU ... The pre-scan decompressed its frames on the host and folded them into a per-pixel projection there. On a 16M-pixel detector that is 60 frames of 72 MB to decompress and 20 bytes per pixel to read and write back per frame - about 40 GB of memory traffic - and it was the whole cost of the phase once the mask was no longer the bottleneck. Only the compressed chunk crosses PCIe now. BSLZ4DecoderGPU already exposes the raw decoded bytes (Decode(), the path its own tests use), which is what this needs: the projection is defined on the RAW STORED COUNTS with the pixel type's sentinel skipped, not on the preprocessed image, so nothing here goes through the preprocessor. Sums, maxima and counts are integers, so the device result is identical to the host's rather than merely close. Frames are folded in batches of four. The fold reads and writes the whole accumulator whatever the batch holds, so per frame it was spending most of the bandwidth on the accumulator rather than on the data; four is where that stops mattering, and every frame beyond it is another full frame of device memory, which costs more in cudaMalloc - device-synchronizing - than it saves. The accumulator is built on a thread of its own. It allocates and clears several hundred megabytes, and doing that in the constructor stalled the caller before it had read its first frame. Frames the device cannot take - anything but bitshuffle+LZ4 - still go to a host shard, so a run mixing compressions needs no second code path, and a build without CUDA is unchanged. RotationScaleMergeGPU set the CUDA device in its constructor and never put it back. CUDA's current device is per-thread, so that silently re-pinned the calling thread for the rest of its life, and the destructor freed several gigabytes against whatever device happened to be current by then - CudaDevicePtr records no device of its own. Every entry point now sets the device on entry and restores it on exit. ParallelFor/ParallelChunks moved to common/ParallelFor.h; two files had copies and a third wants them. Measured on a 16M-pixel rotation dataset: pre-scan 4.78 s -> 2.37 s -> ~2.0 s, shadow unchanged at 139126 pixels (22143 on a 2M-pixel dataset). Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 14m49s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 18:38:26 -04:00
jungfrauandClaude Opus 5	50941f8df0	Make the beam-stop mask O(pixels) and parallel ... GetMask() was 3.28 s of the 4.78 s pre-scan on a 16M-pixel detector, all on one thread. Four changes, none of which alters the mask: dilate() was a multi-source BFS. On a full rectangle with no obstacles the 8-connected graph distance IS the Chebyshev distance - a path stepping towards the target never has to leave the frame - so the result is a dilation by the (2r+1) square clipped to the frame, which separates into a pass along x and a pass along y. That is O(1) per pixel whatever r is, with no queue and no 4-bytes-per-pixel distance array (72 MB, allocated and filled five times per call). The erode() case is the one that hurt: it dilates the COMPLEMENT, so on a detector whose shadow is under 1% of the pixels it seeded the BFS from essentially every pixel. fill_holes() floods the background from the border. It now floods the bounding box of the region grown by one: everything outside that box is background and the box's own ring is background, so the whole outside is one border-connected component and a background pixel inside the box is border-connected exactly when it reaches the ring. The three baseline iterations re-binned every pixel by radius and re-took a median each time. The iteration only ever excludes pixels whose background is below a cut, and dividing by a positive baseline is monotone, so a ring's excluded pixels are exactly its lowest ones and the next median is an order statistic of the same, unchanging ring. The rings are binned and sorted once; each iteration then picks a rank and counts a prefix. Nine full-image passes become one. box_sum's vertical pass walked one column at a time, striding a whole row per step and missing on every access; it now carries a strip of columns together. Each row's and each column's running sum keeps its terms in its order, so the floating-point rounding is unchanged - only the traversal differs. The pooled COUNT is a count of at most 25 pixels, so it is an exact integer box sum now rather than a floating-point one; the background itself stays in double, because its running sum adds and subtracts across a whole row and in float the two roundings would not cancel. The per-pixel passes then run on all threads, and GetMask takes a thread count. Measured on a 16M-pixel rotation dataset: GetMask 3.28 s -> 0.99 s, whole pre-scan 4.78 s -> 2.37 s, whole run 1m10s -> 1m03s. The mask is unchanged on both a 16M and a 2M-pixel dataset (139126 and 22143 shadow pixels), as are the space group, the merged reflection count and the merging statistics. Also corrected the comment on erode(): the dilation cannot seed outside the frame, so outside behaves as foreground, not as complement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 17:50:06 -04:00
jungfrauandClaude Opus 5	f12e3b6252	Parallelize the beam-stop pre-scan ... The pre-scan read its sample of frames in a plain serial loop: one thread did the HDF5 read, the decompression and the full-detector accumulation for every frame. The cost is fixed per frame rather than per dataset, so it grew straight with detector area - measured at 0.9 s on a 2M-pixel detector and 7.9 s on a 16M-pixel one, where it was 11% of the whole run with 47 of 48 cores idle. Frames are now read on several workers. ShadowFinder keeps one projection per worker so nothing is locked while an image is added, and the projections are summed when the mask is read; the sums and counts are integers, so the result does not depend on how the frames were spread over the workers. A shard that never counted a pixel is skipped when the maxima are merged - it holds 0, which would otherwise beat a genuinely negative maximum. Worker count is capped (PRESCAN_MAX_WORKERS): a shard costs 20 bytes per pixel, and the accumulation is memory-bound, so a handful of workers already saturates it. The beam-centre spot pool is stitched together in sample order after the workers join, so frame numbering and the spot list are what the serial read produced regardless of how the workers interleaved. A frame still joins the pool only if it could be read. ShadowFinder::AddImage took its decompression scratch buffer BY VALUE, so the caller's buffer stayed empty and every frame allocated and zero-filled a fresh full-size uncompressed image (72 MB on a 16M-pixel detector) and freed it again. It takes a reference now, and each worker reuses one buffer. Measured on a 16M-pixel rotation dataset: pre-scan 7.9 s -> 4.8 s, whole run 69.2 s -> 65.1 s. Results are unchanged - same shadow pixel count, same space group, same merged reflection count and merging statistics on both a 16M and a 2M-pixel dataset, and the beam-centre path still commits the same centre. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 17:13:53 -04:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00