diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 014a35e6..2092065c 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -26,11 +26,11 @@ This is an UNSTABLE release. It includes many experimental features, as well as * The writer refuses a stream whose start message declares a different pixel format than its images carry. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * rugnux: observations outside the scaling resolution range are dropped as they are ingested, rather than scaled, combined and error-modelled first; merged results are unchanged. -* rugnux: two runs of the same command on the same images now produce the same intensities - the GPU prediction, profile-learning and profile-fit steps no longer depend on the order their blocks happen to run in. +* rugnux: two runs of the same command on the same images now produce the same merged intensities - the GPU prediction, profile-learning and profile-fit steps no longer depend on the order their blocks happen to run in. The azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. -* rugnux: a rotation run is substantially faster - a 24-crystal set that took 7m55s takes 6m28s, with the space group unchanged on every one of them. Most of it is work that was being done and thrown away: the geometry pre-pass no longer merges, corrects, and writes a result the second pass replaces, observations outside the scaling resolution range are never built, and the detector lookup tables are built once for the run rather than once per worker. -* rugnux: the geometry pre-pass no longer writes `_01.mtz`, `_01.cif`, `_01.hkl` or its scaling table. The second pass rewrites all of it at the refined geometry seconds later, and that is the result to use. -* rugnux: with `-N` left at its default the per-image loop uses at most eight workers per GPU. Past that, workers spend their time queueing kernels rather than running them - measured, a 16 Mpx set is slower at 48 workers than at 8. An explicit `-N` is obeyed as given. +* rugnux: a rotation run is substantially faster. +* rugnux: a rotation run no longer writes the geometry pre-pass files `_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes the same files under ``, and that is the result to use. +* rugnux: with `-N` left at its default the per-image loop now uses at most eight workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector rather than on the reflections that happen to be present, so whether it is applied no longer depends on how far integration reached. * rugnux: the first-pass rotation indexing finds its spots on every worker rather than one, which is worth most on large detectors. The lattice it picks is unchanged. * rugnux: beam-stop detection is substantially faster - the pre-scan reads and accumulates its frames on several threads, the mask is built in one pass over the image rather than by repeated searches, and a worker's accumulator is allocated only if it is used. The shadow it finds is unchanged. diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index a5c605fb..de41f64e 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -48,25 +48,32 @@ the GPU when one is present. Instead of decompressing on the host and uploading compressed chunk is uploaded — a few MB rather than tens of MB — and decoded on the device. The approach follows Jon Wright (ESRF); the kernels are Jungfraujoch's own. -Two kernels do the work: +One kernel does the work: one CUDA block owns one bitshuffle block, from the compressed payload +through to finished pixels. -1. **LZ4, one warp per bitshuffle block.** Blocks are independent, so the parallelism is across - them; within a warp every lane runs the same sequence parser over the same bytes, and the - literal and match copies are split across the 32 lanes so the stores coalesce. An overlapping - match is treated as a pattern of period `offset` sourced from bytes that already precede the - write position, which keeps it parallel rather than a serial byte loop; `offset == 1` (a run of - one repeated byte, the common case in sparse detector data) and power-of-two offsets avoid the - modulo altogether. Because the lanes cooperate on the copies, each one is followed by - `__syncwarp()` — a later match can read bytes another lane wrote, and since Volta that ordering - is not implicit. -2. **The bitshuffle inverse fused with preprocessing.** One thread owns one group of 8 elements - across every byte-plane, so once it has transposed its 8 bytes out of each plane it holds 8 - complete elements — and it applies the pixel mask, the error marker and the saturation cap and - emits 8 finished `int32` pixels directly. The decompressed image is therefore never materialised - in device memory at all, which removes a frame-sized buffer per worker and a full-frame write - plus read from the pipeline. Staging nothing in shared memory also means the kernel has no - dynamic-shared-memory request, so it is indifferent to the bitshuffle block size the file - declares. For 8-bit images there is a single plane and the assembly degenerates to a copy. +1. **LZ4 into shared memory, one warp per bitshuffle block.** Blocks are independent, so the + parallelism is across them; within the warp every lane runs the same sequence parser over the + same bytes, and the literal and match copies are split across the 32 lanes so the stores + coalesce. An overlapping match is treated as a pattern of period `offset` sourced from bytes + that already precede the write position, which keeps it parallel rather than a serial byte loop; + `offset == 1` (a run of one repeated byte, the common case in sparse detector data) and + power-of-two offsets avoid the modulo altogether. Because the lanes cooperate on the copies, + each one is followed by `__syncwarp()` — a later match can read bytes another lane wrote, and + since Volta that ordering is not implicit. +2. **The bitshuffle inverse fused with preprocessing.** The whole CUDA block then reads that shared + buffer back: one thread owns one group of 8 elements across every byte-plane, so once it has + transposed its 8 bytes out of each plane it holds 8 complete elements — and it applies the pixel + mask, the error marker and the saturation cap and emits 8 finished `int32` pixels directly. + Nothing of the block reaches device memory but the pixels — neither the bitshuffled bytes nor the + decompressed image is ever materialised. For 8-bit images there is a single plane and the + assembly degenerates to a copy. + +Decoding into shared memory is worth more than the bandwidth it saves: an LZ4 match reads back bytes +written a few sequences earlier, so every copy step is a dependent round trip — tens of cycles in +shared memory against hundreds in device memory. It is paid for in residency, because the buffer +holds a whole bitshuffle block, and a block larger than 16 kB (larger than either writer this +pipeline reads produces) falls back to a pair of kernels instead, the first writing the shuffled +image to device memory and the second un-transposing and preprocessing out of it. The block offsets inside the container can only be discovered by reading the block lengths in order, so that scan stays on the host. @@ -509,7 +516,7 @@ The refinement above (§7.2) runs per image against that image's spots. For rota Each step is **cross-validated** on a deterministic split of the *reflections* (an avalanche-mixed $hkl$ hash, not a frame split and not an $h+k+l$ parity, which would collide with a centering condition and leave the held-out half empty): fitted on one half, committed only if it lowers the held-out residual, otherwise left at nominal. The solver bounds the move — distance within ±5 %, beam centre within ±15 px — and detector tilt is held fixed, being gauge-coupled to the crystal orientation on a single crystal. 3. **Pass 2** re-indexes de novo and re-integrates at the committed geometry, reusing pass-1's space group for the merge only. Only the **detector distance and beam centre** carry over: the refined cell and axis are used to make step B well-posed, but pass 2 re-indexes from scratch, so they are not propagated. -The refined pass is written as the canonical `_*` output; the pass-1 (header-geometry) result is kept alongside as `_01_*` for comparison. +Only pass 2 is written, as the canonical `_*` output. Pass 1's merge exists to choose the space group and to give the guard something to judge pass 2 against, so it stops short of the parts of the merge that only fill in a file — the correction surfaces, the twinning and radiation-damage analyses, the R-free flags and the amplitudes — and writes no merged files of its own. **Goniometer rotation scale (report only).** A stage that turns further than it was commanded to leaves no trace in the file, because the stored $\omega$ values *are* the commanded ones; the excess then presents as the crystal drifting, in this program and in others. Step A already measures it without a new degree of freedom: its residual rotates by $-\phi\,\mathbf{u}$ with $\mathbf{u}$ an **unnormalised** 3-vector, so $|\mathbf{u}|$ is the factor by which the stage actually turned, and normalising the axis throws it away. It is reported, and warned about beyond 0.5 %, under the same cross-validation that gates the cell move — a fold that merely soaked up noise cannot raise the flag. It is a detector, not a calibration: nothing corrects the data, and it **under-reads** the true magnitude, because the fit only sees reflections that indexed at the nominal angle and per-frame orientation refinement has already absorbed part of the error. diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index 237407f7..de5063db 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -29,8 +29,10 @@ parameters, or to produce merged intensities for downstream structure solution. As with the rest of Jungfraujoch, **serious performance requires an NVIDIA GPU**. The CUDA build provides the GPU fast-feedback indexer (`ffbidx`) and the GPU FFT indexer (`fft`); without CUDA -only the CPU `fftw` indexer is available. Spot finding, integration and scaling run on the CPU and -scale with the thread count (`-N`). +only the CPU `fftw` indexer is available. With a GPU present most of the per-image pipeline runs on +the device — bitshuffle+LZ4 decompression, image preprocessing, azimuthal integration, spot finding, +prediction and Bragg integration — as does rotation scaling and merging, with CPU implementations as +the fallback where there is no GPU. The thread count (`-N`) governs the CPU side of all of it. The released CUDA builds need only an NVIDIA **driver** on the host, no CUDA toolkit: 525.60.13 or newer for the CUDA 12 artefacts (RHEL 8 packages, portable Linux `.tgz`) and 580.65.06 or newer for @@ -68,8 +70,7 @@ reusing them would hide the spot-finding settings from the lattice search. ends with the `0 0 0` terminator record. All three carry the **refined unit cell** (from rotation indexing) and the **space group determined - from systematic absences** (constrained to the indexed lattice symmetry). No-reference scaling - additionally emits per-iteration `_iterN_scale.dat`. + from systematic absences** (constrained to the indexed lattice symmetry). - `_report.txt` — the **results report**: what the run determined, in a form both a person and a beamline script can read. Always written, next to the files above. See [The results report](#the-results-report) below. @@ -151,11 +152,11 @@ Sections, in order: `1. DATA SET`, `2. INDEXING`, `3. GEOMETRY POST-REFINEMENT` `4. SPACE GROUP DETERMINATION`, `5. SCALING AND MERGING`, `6. TWINNING`, `7. RADIATION DAMAGE`, `8. SWEEP QUALITY`, `9. WARNINGS`. -**Which pass.** A rotation run integrates twice — once at the geometry in the input file -(`_01.*`), then again at the post-refined geometry (`.*`) — and can integrate a -third time if a guard rejects the second pass. There is **one** report, for the pass that became the -canonical output, and `PASS=` / `PASS_DECISION=` in section 1 say which pass that is and on what -evidence, so no number in the file is ambiguous about which geometry produced it. +**Which pass.** A rotation run integrates twice — once at the geometry in the input file, then again +at the post-refined geometry — and can integrate a third time if a guard rejects the second pass. +There is **one** report, for the pass that became the canonical output, and `PASS=` / +`PASS_DECISION=` in section 1 say which pass that is and on what evidence, so no number in the file +is ambiguous about which geometry produced it. **Not in the report:** timing, frame rates, thread counts, per-image progress and library banners. Those are process, not result, and stay on stdout. @@ -331,9 +332,10 @@ By default a rotation run also **post-refines the geometry** in a second pass: t integrates and merges at the header geometry, then the detector distance + beam centre and the crystal cell / rotation-axis are refined against the merged fulls (cross-validated, and committed only for a small < 1 % move, with the gauge-weak beam centre restrained toward the header), and the second pass -re-indexes de novo and re-integrates at the refined geometry. The refined pass is the canonical -`_*` output; the header-geometry pass is kept alongside as `_01_*` for comparison. -Disable it with `--rotation-no-postrefine`. +re-indexes de novo and re-integrates at the refined geometry. Only the refined pass is written, as +the canonical `_*` output; the header-geometry pass merges only to choose the space group and +to judge the refined pass against, and writes no merged files of its own. Disable it with +`--rotation-no-postrefine`. After the per-frame scale-fulls step, rotation scaling applies three **correction surfaces**, **on by default** (`--no-scaling-corrections` disables all): @@ -401,7 +403,7 @@ General: | Option | Description | | --- | --- | | `-o, --output-prefix ` | Output file prefix (default: `output`) | -| `-N, --threads ` | Number of worker threads (default: all hardware threads) | +| `-N, --threads ` | Number of worker threads (default: all hardware threads, with the per-image loop capped at 8 workers per GPU; an explicit value is used as given) | | `-s, --start-image ` | First image to process (default: 0) | | `-e, --end-image ` | Last image to process (default: all) | | `-t, --stride ` | Process every *n*-th image (default: 1) | diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index a33d1736..8b6fd1d4 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -85,8 +85,7 @@ std::string RenderResultReport(const std::string &output_prefix, os << "\n" << " A rotation run integrates twice: once at the geometry in the input file, then again at\n" << " the post-refined geometry. Every number in this report describes the pass named above,\n" - << " whose files are " << output_prefix << ".*; the header-geometry pass was written to\n" - << " " << output_prefix << "_01.* and is kept only for comparison.\n"; + << " whose files are " << output_prefix << ".*; the header-geometry pass is not written.\n"; } // ---------------------------------------------------------------- 2. INDEXING diff --git a/rugnux_anomalous.py b/rugnux_anomalous.py index 056a142f..5cefda17 100644 --- a/rugnux_anomalous.py +++ b/rugnux_anomalous.py @@ -364,7 +364,7 @@ def resolve_hkl(path, name): p = pathlib.Path(path.format(name=name)) return p if p.exists() else None d = pathlib.Path(path) - # rugnux writes both .hkl and _01.hkl; .hkl is the stable one. + # rugnux writes the merged reflections as .hkl. for cand in (d / name / f"{name}.hkl", d / f"{name}.hkl"): if cand.exists(): return cand diff --git a/viewer/windows/JFJochProcessingJobsWindow.cpp b/viewer/windows/JFJochProcessingJobsWindow.cpp index 62906c8c..87247d06 100644 --- a/viewer/windows/JFJochProcessingJobsWindow.cpp +++ b/viewer/windows/JFJochProcessingJobsWindow.cpp @@ -233,8 +233,8 @@ int JFJochProcessingJobsWindow::askJob(const ReprocessingInputs &inputs, JobSpec refineRowLayout->addStretch(); // Rotation two-pass geometry post-refine: a first pass post-refines the detector distance / beam / - // cell + rotation axis from the whole sweep, then re-integrates at that geometry (the refined pass is the - // canonical _* output, the header-geometry pass is kept as _01_*). Rotation-only; + // cell + rotation axis from the whole sweep, then re-integrates at that geometry (only the refined pass + // is written, as the canonical _* output). Rotation-only; // default on there, to match the rugnux CLI. const bool rotation_run = spec.mode == ProcessMode::FullAnalysis && inputs.experiment.GetIndexingSettings().GetRotationIndexing(); @@ -243,8 +243,8 @@ int JFJochProcessingJobsWindow::askJob(const ReprocessingInputs &inputs, JobSpec postrefine->setEnabled(rotation_run); postrefine->setToolTip( "Rotation only: a first pass post-refines the detector distance / beam centre / cell / rotation " - "axis from the whole sweep, then re-integrates at the refined geometry. The refined pass is the " - "canonical output; the header-geometry pass is kept as _01. Default on; a no-op for stills."); + "axis from the whole sweep, then re-integrates at the refined geometry. Only the refined pass is " + "written, as the canonical output. Default on; a no-op for stills."); // Beam-stop shadow: a projection of a few frames shows where the stop and its holder shadow the // detector; those pixels go into the mask (bit 9) so nothing behind them is integrated. Cheap and