release: 1.0.0-rc.166

The changelog entry for this release is six lines, not the fifty-four the
development log had accumulated: what a user of rc.166 gets is native miniCBF
input, masters from other facilities opening, the beam centre measured on every
run, a 2theta-swung detector placed where the file says, the unmerged MTZ and a
P1 merge written by default, and a substantially reworked symmetry
determination. The per-change detail is in the commits.

update_version.sh regenerates the three clients from broker/jfjoch_api.yaml, so
this also carries the one API description that had drifted from the generated
code - fft_high_resolution_A, which sizes the FFT's projection histogram and
does not filter spots by resolution. Everything else in the regenerated tree is
the version string.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N

docs/CHANGELOG.md

@@ -1,71 +1,14 @@
 # Changelog
 ## 1.0.0
 
-* The space group is carried as its full name rather than its number - through the analysis, in `-S`, and in `/entry/sample/space_group` - so a setting a number cannot name (`P 1 1 2`, `R 3:R`) is no longer lost.
-* The space-group search names the setting the data support, so a screw or a 2-fold on the a or c axis is reported as such instead of costing the crystal its space group.
-* The run report carries `SPACE_GROUP_NAME` beside `SPACE_GROUP_NUMBER`, since the number alone does not say which axes a group's symmetry lies on.
-* A run whose cell metric admits more rotational symmetry than the space group it adopted says so, naming both, so a symmetry the intensities were too weak to confirm is visible rather than silent.
-* Where the two space-group search merges disagree, the point group is taken from whichever supports the larger one and its systematic absences are judged on all observations, so a symmetry only the well-measured reflections show is no longer discarded and its screw axes are still read from the full data.
-* A lattice keeps the symmetry it has whatever orientation it was refined in: the cell reduction now judges a structurally-zero scalar product against the size of the cell rather than against a fixed tolerance eight decades smaller, so a body-centred tetragonal lattice is no longer read as C-centred monoclinic on most rotations of itself.
-* A centred monoclinic lattice that reduces to the mI form keeps its centring: the character naming that reduced form was missing from the table, so such a cell was reported as triclinic.
-* The primitive cell of an R- or H-centred lattice is that lattice: the change of basis was applied without the transposition the convention needs, which left the volume right and the cell wrong, and reached the path that re-seats a lattice into a space group fixed by hand.
-* A space-group candidate whose predicted absences were never measurable no longer outranks one whose absences are genuinely dead - an unmeasured zone was worth about 690 nats per reflection, which inverted the ranking outright.
-* A CBF with no detector header is refused instead of being opened with a pixel size of zero and every pixel called saturated, and a header number that does not parse is reported as a malformed header.
 ### 1.0.0-rc.166
-* `rugnux` reads a PILATUS miniCBF rotation sweep natively, with no conversion: naming any frame - or the directory holding it - processes the whole sweep that frame's template belongs to. `--mode scale` still needs a `_process.h5`.
-* A master whose companion `_meta.h5` was not kept still opens: the fields an Eiger master links into that file are treated as absent rather than as a read error, and a run that finds no saturation value anywhere says so and carries on instead of refusing the dataset.
-* `jfjoch_viewer` opens PILATUS miniCBF sweeps - naming any frame opens the whole sweep - and can run a processing job on one.
-* A miniCBF sweep takes its rotation axis from the goniometer angles the header states, instead of assuming the axis every such file was previously assumed to have.
-* A miniCBF sweep takes the mounting from the imgCIF axis table its header carries - which laboratory direction the image's columns and rows run along, and which the spindle turns about - and, where there is no table, from a `+SLOW` on the `Oscillation_axis` line, which says the spindle runs along the image's slow direction.
-* Naming a miniCBF frame with no directory - a frame in the working directory - finds its sweep instead of reporting that no images were found.
-* An NXmx master written outside the DECTRIS toolchain opens: lengths are taken in the unit the file declares rather than assumed to be metres, the image size comes from the image array's own shape, the detector distance is looked for in `NXinstrument` as well, and each data file's images are read at the path the master's own link names.
-* A master written by beamlines that store single values as one-element arrays, compose their images as a virtual dataset over the master itself, or keep them outside `/entry/data/data`, now opens.
-* A master that states its goniometer axis directions in an NXmx `transformations` group but keeps the angles in the legacy `goniometer` group now opens, and takes the axis direction from what the file states rather than assuming it.
-* A DECTRIS firmware 1.x master opens: the detector distance, the saturation value and the goniometer axes are read under their pre-NXmx names where the NXmx ones are absent. Such a sweep was previously read as stills, silently.
-* A master that links to data files but yields no images is an error, instead of a warning and a successful exit.
-* Images compressed with the HDF Group's plain LZ4 filter (HDF5 filter 32004) are read, and an unsupported filter is named in the message instead of the run failing later as though the crystal had not diffracted.
-* The stored image's orientation is taken from the `fast_pixel_direction` / `slow_pixel_direction` vectors an NXmx file states, instead of every detector being assumed to be mounted the way this system mounts its own.
-* A detector swung out on a 2theta arm is placed where it stands, from the `depends_on` transformation chain of an NXmx master or the `Detector_2theta` line of a miniCBF header.
-* A detector whose stored image is mirrored in Y or mounted at a multiple of 90 degrees can be described as such, in the detector configuration or with `--detector-mirror-y` / `--detector-quarter-turns`.
-* `--beam-center-check`, on by default, measures the beam centre from the isotropy of the scattered background on every run, reports how far the file's value is from it and how right this geometry needs it to be, and indexes a second first pass at the measured centre to see whether the two give the same lattice. The measured centre is adopted only where the file's indexes nothing and the measured one indexes a majority; a disagreement between two passes that both worked is reported and left undecided.
-* `--beam-center-search`, on by default, steps the beam centre a pixel at a time out to 12 px along each detector axis after a first pass that indexes fewer than half the validation frames, and keeps the first rung that indexes a majority; `=off` disables it.
-* Where neither the file's beam centre nor the measured one indexes anything, the rotation-axis sign is tried again at the measured centre.
-* A run reports, before it reads a frame, when the file's beam centre lands on a masked pixel, when the beam-stop pre-scan finds no shadowed pixels at all, and when the centre looks typed rather than measured.
-* Two candidate cells whose primitive volumes are related by a small integer or by the square root of three are named as axis harmonics of one another, with the note that the beam centre is what decides between them.
-* After a rotation first pass that indexes poorly, the opposite rotation-axis sign is tried and whichever indexes more validation frames is kept.
-* The rotation first pass refines twelve candidate lattices rather than four.
-* A candidate cell whose three rows are coplanar is rejected before refinement, instead of producing a not-a-number Jacobian and several hundred lines of solver output.
-* When the directions an FFT search shortlists all lie in one plane, one further transform is spent along the plane normal, so a crystal with a very long axis can index.
-* `--fft-min-unit-cell` sets the shortest cell axis the FFT will consider, and giving a cell with `-C` lowers it automatically, so a small-molecule cell is no longer forced onto a supercell.
-* The FFT search can reach past 500 A: on rotation data `-C` raises the longest cell axis it looks for to cover the cell given, and `fft_max_unit_cell_A` is accepted up to 1200 A. The default is unchanged at 500 A, so nothing pays for the wider range unless it is asked for.
-* The FFT peak picker's background window is slid inward at the ends of the spectrum instead of being truncated, so a peak near either end - where the longest cells sit - is judged on a full-width background.
-* The lattice search rejects a Bravais character whose implied cell angle is geometrically impossible.
-* The lattice search retries its Niggli type-boundary supplement for each of the three cell angles, not only for beta.
-* Systematic-absence evidence for a screw axis is scored per axial row rather than pooled, so an unmeasured row no longer vetoes a confirmed one.
-* Lattice centring is chosen by the same likelihood the screw test uses, rather than by counting absences.
-* The space group is determined after the second pass rather than the first, on the geometry the run refined rather than the geometry it started with.
-* The de-novo space-group search ignores the frames whose fitted per-frame scale came out below a tenth of the run median; the production merge still keeps every frame.
-* A point-group promotion that fails the merge chi-squared test needs a smaller rise in its merge's systematic error to be rescued from it, so a merohedrally twinned crystal is kept in its true lower symmetry.
-* Each scaling pass of a de-novo rotation run runs the number of scaling iterations it was asked for, instead of continuing the previous pass's. A run given its space group with `-S` is unaffected.
-* `--min-indexed-spots` sets how many spots must lie on a lattice for a frame to count as indexed, and `--rot3` gives the third detector rotation alongside `--rot1` and `--rot2`.
-* A run whose indexer could not run at all - an exhausted GPU, most often - reports that, instead of reporting that no lattice was found and advising the beam centre be checked.
-* A run that exhausts GPU or host memory stops with an error instead of skipping the images it could not process and finishing as if they had never been there.
-* `rugnux` measures its own spot budget on rotation data instead of keeping the strongest 1000 spots of every image: the first pass reads how deep into an image's spot list the spots still lie on the lattice it found, and the run keeps that many. `--max-spots` still pins the budget, and stills are unchanged.
-* Self-calibrating spot detection intersects its per-resolution-ring threshold with the local signal-to-noise test again, instead of replacing it, so the brightest reflections of a strongly diffracting crystal are no longer detected as oversized blobs and discarded.
-* The self-calibrating threshold no longer steps where it switches from the exact Poisson tail to a normal approximation.
-* A spot may hold up to 200 connected pixels rather than 50, the old bound having acted as an intensity ceiling under the self-calibrating threshold.
-* A spot larger than 50 pixels must also be compact, filling a fifth of the square its bounding box fits inside, so that an ice arc, a cosmic-ray track or a lit detector row is refused however many pixels it has.
-* Eight further hexagonal-ice bands between 1.472 and 1.170 A are recognised, so a detector reaching past 1.5 A no longer leaves those rings in the data. Ice handling still only applies where a run trips the ice gate.
-* Spot finding no longer throws away a whole image when it holds many strong pixels: the limit follows the detector (one pixel in 64) instead of standing at the 65535 that suited a 4-megapixel detector.
-* The connected-component search is linear in the strong pixels rather than quadratic in how many of them a detector line holds; the spots it finds are unchanged.
-* The spot-finding resolution estimate is no longer capped at the corner of the detector, so a crystal that diffracts past the edge is reported as reaching past it, and a run whose estimate is finer than what was merged is a detector-limited run.
-* `/entry/MX/strongPixels` and the strong-pixel plot are filled on the CPU/GPU analysis path, not only behind the FPGA, so an image the spot finder gave up on can be told from one that did not diffract.
-* `rugnux` writes the unmerged MTZ `<prefix>_unmerged.mtz` by default; `--no-export-unmerged` skips it.
-* Every rotation run that determines its own space group also writes `<prefix>_P1.mtz`, the same observations merged in P1, so a wrong space group can be re-merged, re-solved or re-refined without processing the images again; `--no-p1-crosscheck` declines it, and a run given `-S` writes nothing because its space group's centring absences were never integrated.
-* The rugnux results report carries `JFJOCH_DATASET_SETTINGS=`, the geometry the run integrated at written as one line of JSON in the form `jfjoch_broker` takes it, so a refined beam centre and distance can be carried back to the instrument.
-* `rugnux --mode calibration --no-refine-tilt`, and the "Refine detector tilt" tick box in the viewer's calibration panel, fit the beam centre and the distance with the detector tilt held fixed, for a calibration meant for a program that cannot express a tilted detector.
-* In `jfjoch_viewer`, Alt and the mouse wheel step through the dataset one image at a time.
-* The API description of `fft_high_resolution_A` says what the setting does: it sizes the FFT's projection histogram and does not filter spots by resolution.
+
+* `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion.
+* Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants.
+* `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing.
+* A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed.
+* `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing.
+* Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences.
 
 ### 1.0.0-rc.165
 This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

docs/THIRD_PARTY_NOTICES.md

@@ -53,6 +53,12 @@ These are copied into the source tree (see the path) rather than fetched.
 | [LZ4](https://github.com/lz4/lz4) | `compression/lz4/` | Yann Collet | BSD-2-Clause | [lz4.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/lz4.txt) |
 | [HLS arbitrary-precision types](https://github.com/Xilinx/HLS_arbitrary_Precision_Types) | `fpga/include/` | Xilinx, Inc. | Apache-2.0 | [xilinx-hls-headers.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/xilinx-hls-headers.txt) |
 | [GEMMI](https://github.com/project-gemmi/gemmi) | `gemmi_gph/` | Global Phasing Ltd. | MPL-2.0 | [gemmi.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/gemmi.txt) |
+| [PEGTL](https://github.com/taocpp/PEGTL) | `gemmi_gph/gemmi/third_party/tao/` | Dr. Colin Hirsch and Daniel Frey | MIT | [pegtl.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/pegtl.txt) |
+| [sajson](https://github.com/chadaustin/sajson) | `gemmi_gph/gemmi/third_party/sajson.h` | Chad Austin | MIT | [sajson.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/sajson.txt) |
+| [fast_float](https://github.com/fastfloat/fast_float) | `gemmi_gph/gemmi/third_party/fast_float.h` | The fast_float authors (Daniel Lemire et al.) | Apache-2.0 OR MIT OR BSL-1.0 | [fast-float.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/fast-float.txt) |
+| [half](https://half.sourceforge.net/) | `gemmi_gph/gemmi/third_party/half.hpp` | Christian Rau | MIT | [half.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/half.txt) |
+| [pocketfft](https://github.com/mreineck/pocketfft) | `gemmi_gph/gemmi/third_party/pocketfft_hdronly.h` | Max-Planck-Society; Peter Bell; MIT (FFTW-derived parts) | BSD-3-Clause | [pocketfft.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/pocketfft.txt) |
+| [tinydir](https://github.com/cxong/tinydir) | `gemmi_gph/gemmi/third_party/tinydir.h` | Cong Xu, Lautis Sun, Baudouin Feildel, Andargor | BSD-2-Clause | [tinydir.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/tinydir.txt) |
 | [traccc (ACTS)](https://github.com/acts-project/traccc) | `image_analysis/spot_finding/StrongPixelSet.cpp`, `SpotExtractorGPU.cu` | CERN, for the benefit of the ACTS project | MPL-2.0 | [traccc.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/traccc.txt) |
 | [xbflash.qspi](https://github.com/Xilinx/XRT) | `tools/xbflash.qspi/` | Xilinx / AMD | Apache-2.0 | [xbflash-qspi.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/xbflash-qspi.txt) |
 | [wingetopt](https://github.com/alex85k/wingetopt) | `tools/wingetopt/` | Todd C. Miller; The NetBSD Foundation | ISC AND BSD-2-Clause | [wingetopt.txt](https://gitea.psi.ch/mx/jungfraujoch/src/branch/main/licenses/wingetopt.txt) |
@@ -83,6 +89,12 @@ served frontend, so the shipped web UI carries its own attribution.
 * **MPL-2.0** (Eigen, GEMMI, libzmq, traccc): file-level copyleft. GEMMI is vendored in `gemmi_gph/`
   in trimmed form; libzmq is fetched at build time; Eigen is provided externally (header-only). The
   corresponding source is available from each project upstream.
+* **GEMMI's own bundled third-party headers** — PEGTL, sajson, fast_float, half, pocketfft and
+  tinydir, all under `gemmi_gph/gemmi/third_party/` — are listed separately above rather than being
+  absorbed into GEMMI's row: they are other authors' code under other licences (MIT, BSD and
+  Apache/MIT/BSL), and GEMMI's MPL-2.0 does not speak for them. Their terms are carried in the
+  headers themselves rather than in `LICENSE` files, so the `licenses/*.txt` copies are kept by hand;
+  only PEGTL ships a `LICENSE`, which `COLLECT.sh` copies.
 * **traccc** is the one entry that is not a vendored directory. Its sparse connected-component
   labelling enters two otherwise first-party files: `StrongPixelSet.cpp` adapts the SparseCCL source,
   and `SpotExtractorGPU.cu` follows the design of its GPU counterpart. MPL-2.0 is file-level, so both

docs/conf.py

@@ -9,7 +9,7 @@
 project = 'Jungfraujoch'
 copyright = '2024, Paul Scherrer Institute'
 author = 'Filip Leonarski'
-release = '1.0.0-rc.165'
+release = '1.0.0-rc.166'
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

docs/python_client/README.md

@@ -22,8 +22,8 @@ communicate through network calls or other mechanisms.
 
 This Python package is automatically generated by the [OpenAPI Generator](https://openapi-generator.tech) project:
 
-- API version: 1.0.0-rc.165
-- Package version: 1.0.0-rc.165
+- API version: 1.0.0-rc.166
+- Package version: 1.0.0-rc.166
 - Generator version: 7.20.0
 - Build package: org.openapitools.codegen.languages.PythonClientCodegen
 

docs/python_client/docs/IndexingSettings.md

@@ -9,7 +9,7 @@ Name | Type | Description | Notes
 **algorithm** | [**IndexingAlgorithm**](IndexingAlgorithm.md) |  | [default to IndexingAlgorithm.FFBIDX]
 **fft_max_unit_cell_a** | **float** | Largest unit cell to be indexed by FFT algorithm; parameter value affects execution time of FFT | [default to 250]
 **fft_min_unit_cell_a** | **float** | Smallest unit cell to be indexed by FFT algorithm; parameter value affects execution time of FFT | [default to 10.0]
-**fft_high_resolution_a** | **float** | Highest resolution of spots used for FFT algorithm; parameter value affects execution time of FFT. There is also correlation between smallest unit cell and max resolution, which need to be checked for very small systems.  | [default to 2.0]
+**fft_high_resolution_a** | **float** | Sets how finely the FFT samples each search direction: it fixes the extent of the projection histogram, and with it the transform size, so it affects execution time and memory of FFT. It does NOT filter spots - every spot is projected whatever its resolution, so raising it does not make the transform see a coarser subset of the data. There is also correlation between smallest unit cell and this value, which need to be checked for very small systems.  | [default to 2.0]
 **fft_num_vectors** | **int** | Number of search directions for the FFT algorithm; parameter value affects execution time of FFT. | [default to 16384]
 **tolerance** | **float** | Acceptance tolerance for spots after the indexing run - the larger the number, the more spots will be accepted | 
 **thread_count** | **int** | Thread count for indexing algorithm |