From b8fa8e67d50016fa487ae44ebb77f18e85093135 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 2 Sep 2026 17:41:17 +0200 Subject: [PATCH] docs: the pages catch up with the last day on rc166 An audit of docs/ against the code at HEAD, concentrating on what landed after the previous documentation audit: the balanced half-set split and CCanom, the declared-range completeness denominator, the one rule for a quantity nobody measured, and the PONI the calibration now refuses to write. Five statements the code had made false: - The rotation merge's CCref column shows a dash, not "nan". - Two pages still promised unweighted 2Fo-Fc / Fo-Fc maps; they have been sigma_A-weighted since the rigid-body work, and one of the two sat six lines from a paragraph that said so. - Inserting the CCanom prose into the SigAno paragraph left the sentence about the PDBx items and the SigAno column with CCanom as its subject, so it read as a claim about a quantity that is in neither. CCanom is also rotation-only and is not in the mmCIF, which nothing said. - Two cross-references did not resolve - a heading that was renumbered, and a slug spelled without the hyphen MyST puts in "TCP/IP". Added where the behaviour is new and a user meets it: the report's contract for a quantity a run did not measure - no key, and a dash in the table, which is not the same claim as a measured zero; the half-set rule behind CC1/2, which is why a CUDA and a non-CUDA build now agree on it and on CCanom; and the second reason a calibration writes no .poni, a detector whose stored image is mirrored or quarter-turned, which the PONI format cannot state. The rc.166 changelog stays a release note. One entry is widened from the shell table to the rule it is a case of, and two are added for output that was wrong rather than merely undocumented: FITTED_RESOLUTION was the P1 cross-check's, and the unmerged MTZ carried the reference setting where the merged file carried the adopted one. Sphinx builds clean with -W on the pinned docs/requirements.txt, and every intra-doc anchor resolves against the generated HTML. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 4 +++- docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 6 +++--- docs/JFJOCH_WRITER.md | 2 +- docs/RUGNUX.md | 2 +- docs/RUGNUX_ADVANCED.md | 4 ++-- docs/RUGNUX_CALIBRATION.md | 8 ++++++++ docs/RUGNUX_REPORT.md | 10 +++++++++- docs/TOOLS.md | 2 +- 8 files changed, 28 insertions(+), 10 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index b3ac3d6dc..1ee27f80c 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -16,7 +16,9 @@ * The HDF5 master records `direct_beam_x`/`direct_beam_y` - where the undeflected beam lands, sent on the CBOR start message too - the beam size at the sample as `incident_beam_size` from the new `dataset_settings` `beam_size_x_um`/`beam_size_y_um`, and `/entry/MX/peakCountUnfiltered`; `dataset_settings` accepts any `smargon.chi_deg`, which was restricted to 0-90 degrees. * Reported completeness counts the reflections the beam stop, a detector mask or the low-resolution limit kept out of the merge as missing: the denominator, and the resolution shells it is binned into, now span the run's declared resolution range rather than the range of the reflections that survived, so the innermost shell boundary moves and its numbers are not comparable with those of an earlier release. * The rugnux results report carries `CC_ANOM` beside `SIGANO`, overall and per shell: the anomalous difference measured from one half of the observations correlated against the same difference from the other half, which says whether there is an anomalous signal to phase on without depending on the error model. Reported on Friedel-merged runs too; it matches AIMLESS and phenix, and XDS's similarly named `Anomal Corr` is a different quantity. -* A merging-statistics shell prints `-` where the shell could not measure a quantity, in place of `nan`, and `CC_ANOM` is omitted from the report rather than written as a placeholder when no Bijvoet pair could be split in both hands. +* A quantity a rugnux run did not measure is left out of the results report altogether instead of being written as `nan` - `SIGANO=` on a Friedel-merged run, which is the default, is the case a script meets first - and a merging-statistics shell prints `-` in its place. +* `FITTED_RESOLUTION=` in the results report is the resolution fit of the run's own merge rather than of the P1 cross-check, and the unmerged MTZ is written in the space-group setting the run adopted rather than in that setting's reference one. +* `rugnux --mode calibration` refuses to write a `.poni` for a detector whose stored image is mirrored or turned by a multiple of 90 degrees, the format having no field for it. * CC1/2 is computed on half-sets of equal size, so every reflection measured more than once contributes to it, as in XDS and phenix.merging_statistics, and a CPU-only build now reports the same value as a CUDA one; reported CC1/2 values move slightly, most where multiplicity is low. * The rugnux manual is reorganised into task pages with a run overview and worked phenix / REFMAC5 / Phaser / SHELXC/D/E / POINTLESS-AIMLESS / careless examples, and the HDF5 and API documentation say how a grid scan records the angle its spindle stood at: the goniometer axis with a step of 0. diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index bbb21f1fe..5c3d0ca14 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -271,17 +271,17 @@ Per-shell and overall merging statistics are computed on corrected intensities, - number of observations and of unique reflections, and multiplicity, - mean $I/\sigma(I)$, - $R_\mathrm{meas}$ (the redundancy-independent Diederichs–Karplus form) from within‑HKL deviations, -- $\mathrm{CC}_{1/2}$ (half-set correlation) and, when a reference dataset is supplied, $\mathrm{CC}_\mathrm{ref}$, +- $\mathrm{CC}_{1/2}$, correlating two half-sets of **equal size**: an observation's half is the parity of its rank among its own reflection's observations, ordered by a key built from the raw Miller index and the peak frame. Every reflection measured more than once therefore contributes (a hash of the image alone leaves $2^{1-n}$ of the multiplicity-$n$ reflections entirely in one half, with no second mean to correlate), and because a rank is a property of the set rather than of the order it is walked in, a CUDA build and a `JFJOCH_USE_CUDA=OFF` build report the same $\mathrm{CC}_{1/2}$ and the same $\mathrm{CC}_\mathrm{anom}$. It is the split cctbx's `compute_cc_one_half` — and `phenix.merging_statistics` through it — uses. The stills path balances its halves sequentially instead, in image order. When a reference dataset is supplied, $\mathrm{CC}_\mathrm{ref}$ is reported beside it, - completeness against the reflections the cell and symmetry can give over that same declared range, so low-resolution terms lost to the beam stop, to a detector mask or to the low-resolution limit itself count as missing instead of leaving the denominator along with the data, -- the anomalous signal-to-noise $\mathrm{SigAno}$ (below). +- the anomalous signal-to-noise $\mathrm{SigAno}$ and the half-set anomalous correlation $\mathrm{CC}_\mathrm{anom}$ (below). The error model is refined as $\sigma_\mathrm{corr}^2 = a\,\sigma^2 + (b\,\langle I\rangle)^2$, with $a$ set by the scatter of weak (counting-limited) reflections and $b$ the intensity-proportional systematic scatter of the strong ones. On the **rotation** path, **ISa** is the asymptotic ($I\to\infty$) signal-to-noise — by definition the reproducibility limit of the strongest reflections (Diederichs, *Acta Cryst.* **D66** (2010) 733) — and is read directly from the strong symmetry equivalents as the counting-subtracted fractional scatter of well-measured reflection groups (a robust median over strong groups; the $I/\sigma$ threshold is relaxed on weak or radiation-damaged data that has few strong reflections), rather than as $1/b$ of the whole-range fit, whose $b$ is raised slightly by an intermediate-intensity excess and so understates the limit. The asymptotic value is **report-only** — nothing downstream reads it, and the merged $\sigma$ is not floored at $b|I|$ (that floor was removed). The per-observation $\sigma_\mathrm{corr}$ (the merge weights) uses the whole-range $a,b$. The **stills** path has no asymptotic estimate and reports $\mathrm{ISa}=1/b$ directly. $a$ and $b$ are **reported in XDS's convention**, which is $\sigma^2 = a(\sigma_0^2 + b I^2)$ with $\mathrm{ISa}=1/\sqrt{ab}$, so the printed pair can be read straight against a `CORRECT.LP`. The internal fit keeps the form above; only the report converts, as $b_\mathrm{XDS} = b^2/a$. Note that $a$ is the same in both conventions and that the two ISa expressions are the same number, $1/\sqrt{a\cdot b^2/a} = 1/b$ — so the rotation log prints **two** ISa, the whole-range $1/b$ (XDS's meaning) and the strong-reflection asymptote beside it, which can only ever be the more optimistic of the two. The mmCIF follows the same split: `_reflns.jfjoch_diffrn_ISa` is the whole-range value, directly comparable with a `CORRECT.LP`, and the asymptote is written separately as `_reflns.jfjoch_diffrn_ISa_asymptotic`, with `_reflns.jfjoch_error_model_a` and `_b` alongside so the number can be re-derived. Note that a file written before this change carries the *asymptote* under the plain `ISa` name. A third, unrelated $b$ appears in the space-group search (§13.1); it is fitted with the $\sigma^2$ coefficient held at 1 and its gate constants are calibrated in that convention. -**Anomalous signal-to-noise (SigAno).** The strength of the anomalous signal is reported per shell and overall as $\mathrm{SigAno}=\langle|\Delta I|\rangle / \langle\sigma(\Delta I)\rangle$, where $\Delta I = I(+)-I(-)$ over acentric reflections measured in both Bijvoet hands and $\sigma(\Delta I)=\sqrt{\sigma_+^2+\sigma_-^2}$. It is computed from the **full-multiplicity** inverse-variance $I(+)/I(-)$ split (the same one written to the output), i.e. from all observations rather than a half-set. For pure noise $\mathrm{SigAno}$ approaches the half-normal value $\sqrt{2/\pi}\approx0.8$, and it rises above $1$ once a real anomalous difference is present. A half-set anomalous correlation ($\mathrm{CC}_\mathrm{anom}$) is reported beside it: $\Delta I$ is formed once per half-set and the two are correlated over the acentric pairs where **both** hands split into two non-empty halves, per shell and overall as one correlation rather than a mean of shells. Unlike SigAno it is not a ratio against the error model, so an optimistic $\sigma$ cannot inflate it. It has no floor either: subtracting the two Bijvoet hands cancels the large common intensity that keeps $\mathrm{CC}_{1/2}$ non-negative, so on data with little anomalous signal and about two observations per mate it goes strongly negative. That is a property of the statistic and is reported as measured. It agrees with AIMLESS's `CCanom` and `phenix.merging_statistics`' `cc_anom`; XDS's `Anomal Corr` is a different quantity and is not comparable with it. It is emitted only when an anomalous split was made, using the standard PDBx items `_reflns.pdbx_absDiff_over_sigma_anomalous` (overall) and `_reflns_shell.pdbx_absDiff_over_sigma_anomalous` (per shell), and appears as the `SigAno` column of the printed merge-statistics table. +**Anomalous signal-to-noise (SigAno).** The strength of the anomalous signal is reported per shell and overall as $\mathrm{SigAno}=\langle|\Delta I|\rangle / \langle\sigma(\Delta I)\rangle$, where $\Delta I = I(+)-I(-)$ over acentric reflections measured in both Bijvoet hands and $\sigma(\Delta I)=\sqrt{\sigma_+^2+\sigma_-^2}$. It is computed from the **full-multiplicity** inverse-variance $I(+)/I(-)$ split (the same one written to the output), i.e. from all observations rather than a half-set. For pure noise $\mathrm{SigAno}$ approaches the half-normal value $\sqrt{2/\pi}\approx0.8$, and it rises above $1$ once a real anomalous difference is present. A half-set anomalous correlation ($\mathrm{CC}_\mathrm{anom}$) is reported beside it: $\Delta I$ is formed once per half-set and the two are correlated over the acentric pairs where **both** hands split into two non-empty halves, per shell and overall as one correlation rather than a mean of shells. Unlike SigAno it is not a ratio against the error model, so an optimistic $\sigma$ cannot inflate it. It has no floor either: subtracting the two Bijvoet hands cancels the large common intensity that keeps $\mathrm{CC}_{1/2}$ non-negative, so on data with little anomalous signal and about two observations per mate it goes strongly negative. That is a property of the statistic and is reported as measured. It agrees with AIMLESS's `CCanom` and `phenix.merging_statistics`' `cc_anom`; XDS's `Anomal Corr` is a **different quantity and is not comparable with it** — on the same observations it reads two to three times higher in the low shells. $\mathrm{CC}_\mathrm{anom}$ is a rotation-path statistic (the stills merge forms no half-set anomalous difference), and it is reported in the `CCanom` column of the printed merge-statistics table and as `CC_ANOM=` in the results report — not in the mmCIF. SigAno is emitted only when an anomalous split was made, using the standard PDBx items `_reflns.pdbx_absDiff_over_sigma_anomalous` (overall) and `_reflns_shell.pdbx_absDiff_over_sigma_anomalous` (per shell), and appears as the `SigAno` column of the same table. Where either could not be measured at all — a Friedel-merged run that split no Bijvoet pair, a shell too thin to split one in both hands — the table prints `-` and the results report writes no key, which is not the same statement as a value measured to be zero. ### 10.6 Rotation datasets: combining partials into fulls (3D integration) diff --git a/docs/JFJOCH_WRITER.md b/docs/JFJOCH_WRITER.md index 30452e564..9fde52d17 100644 --- a/docs/JFJOCH_WRITER.md +++ b/docs/JFJOCH_WRITER.md @@ -39,7 +39,7 @@ the writer notification socket — a ZeroMQ address the broker passes in the STA number, run name and number, processed image count, throughput, and on failure an error string. That is how the broker learns that a writer could not write. On the TCP/IP image stream, failures additionally come back in-band as negative acknowledgements -(see [Data streams](IMAGE_STREAM.md#tcpip-image-stream)). +(see [Data streams](IMAGE_STREAM.md#tcp-ip-image-stream)). To stop a writer, send it `SIGINT`, `SIGQUIT`, `SIGTERM` or `SIGHUP`: it closes the HDF5 files it is writing and exits. This is only for the case where the broker was terminated or disconnected — it is diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index cda136b31..c19febc56 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -40,7 +40,7 @@ rugnux -o myrun dataset_master.h5 # resolves the indexing ambiguity, and hands over its R-free set rugnux -o myrun -z reference.mtz dataset_master.h5 -# 3. with a known structure: R-work / R-free and 2Fo-Fc / Fo-Fc maps on top of the merge +# 3. with a known structure: R-work / R-free and sigma_A-weighted 2mFo-DFc / mFo-DFc maps rugnux -o myrun --model model.pdb dataset_master.h5 # 4. with the space group and the cell pinned (-S takes either spelling: P43212 or 96) diff --git a/docs/RUGNUX_ADVANCED.md b/docs/RUGNUX_ADVANCED.md index d25faa933..24daa9691 100644 --- a/docs/RUGNUX_ADVANCED.md +++ b/docs/RUGNUX_ADVANCED.md @@ -25,7 +25,7 @@ starts, and used for four things: is scored on the same free reflections. - **It reports CCref**, the correlation of the merged intensities against the reference, in the statistics table. Stills only — the rotation merge never scores itself against the - reference, and its table shows `nan` in that column. + reference, and its table shows `-` in that column. A reference is **not** a scale anchor. Both workflows scale against their own data — scaling images against a foreign dataset injects that dataset's systematics — so `-z` never puts the reference's @@ -143,7 +143,7 @@ cell or no usable space group, does not fail the run: it is logged, and the resu maps cannot be mistaken for one that was never given `--model`. Either way the results report carries a **`10. MODEL VALIDATION`** section: `R_WORK=` / `R_FREE=` -with their reflection counts, the bulk-solvent and overall scale parameters, the mean 2Fo-Fc density +with their reflection counts, the bulk-solvent and overall scale parameters, the mean 2mFo−DFc density at the atom centres, the reindexing operators the written reflections were brought into the model's frame with, and `MAPS_PREFIX=`; or `MODEL_VALIDATION= NOT_PERFORMED` with `MODEL_VALIDATION_REASON=` when the model could not be used. A run given no `--model` has no such diff --git a/docs/RUGNUX_CALIBRATION.md b/docs/RUGNUX_CALIBRATION.md index 8d94546db..27f2b9623 100644 --- a/docs/RUGNUX_CALIBRATION.md +++ b/docs/RUGNUX_CALIBRATION.md @@ -73,6 +73,14 @@ either case rugnux prints the reason, writes the `.json` with `converged` false the detector is and has no field in which to say that it does not know. A declined tilt over a header that states *no* tilt is not this — reporting no tilt is then exactly what the fit measured. +A `.poni` is refused for a second reason, whatever the fit found: a detector whose **image +orientation** is not the identity — the stored image mirrored in Y, or turned by a multiple of 90° +about the beam. That comes either from `--detector-mirror-y` / `--detector-quarter-turns` or from the +file itself (a PILATUS miniCBF axis table, an NXmx module's pixel directions). A PONI states the +detector in five numbers — two offsets, a distance and three rotations — and has no field for how the +image is stored, so one written here would describe a different geometry from the one that was +measured. The run says so and exits non-zero. + `--no-refine-tilt` holds rot1/rot2 where the header put them and fits only the centre and the distance. The tilt is real and worth measuring, but a program that has nowhere to put one — XDS takes a detector normal to the beam — is better given a geometry that was measured with the tilt diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 5eab7e6ee..2dcb84bfe 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -37,6 +37,14 @@ structure a script can consume without parsing prose. (10.1 deg, scale 0.12 and CC 0.30 of the run, 2% scaled)`. `grep '^WARNING:'` finds every one. - **Section banners** (`***…***` around a numbered title) delimiting the blocks. +**A quantity the run did not measure writes no key at all**, and the fixed-width tables print `-` in +its place. There is one rule and no placeholders — no `nan`, and no `0.0%` that reads as a measured +total failure — so a consumer must treat an absent key as *not measured* rather than assume every key +it knows about is present. A measured value always prints, including a negative one. The keys a +script meets this on first are `SIGANO=` and `CC_ANOM=`, which a Friedel-merged run — the default — +does not write because it splits no Bijvoet pair; `COMPLETENESS=`, `MULTIPLICITY=`, `I_OVER_SIGMA=`, +`R_MEAS=`, `CC_HALF=` and `WILSON_B=` follow the same rule. + `REPORT_VERSION=` is the format's own version. Key names, table columns and the reason vocabulary below are an interface other software may depend on: they do not change without that number moving. Adding a key does not move it — a consumer that greps for what it needs is unaffected by one more @@ -140,7 +148,7 @@ CC-to-merge relative to the run median; `INDEXED` — the fraction of the range' scaled at all. Every range also appears as a `WARNING:` sentence in section 11. The same finding is written **per image** into the `_process.h5` as `/entry/MX/sweepQuality`, when -one is written — see [HDF5](HDF5.md#41-entrymx--spot-finding-and-indexing-cxi-style). +one is written — see [HDF5](HDF5.md#entry-mx-spot-finding-and-indexing-cxi-style). ## Diffraction anisotropy diff --git a/docs/TOOLS.md b/docs/TOOLS.md index 4688a342a..c1900a4a7 100644 --- a/docs/TOOLS.md +++ b/docs/TOOLS.md @@ -13,7 +13,7 @@ merging, reflection files. Merging is on by default (`--no-merge` disables it). run does: `mx` (the above, the default), `azint` (only azimuthal integration, no spot finding/indexing), `scale` (re-scale/merge the already-integrated reflections in a `_process.h5` without re-integrating) or `calibration` (detector geometry from a calibrant's powder rings, written -as a `.poni` file). See [rugnux](RUGNUX.md). +as `.poni` and `.json`). See [rugnux](RUGNUX.md). `rugnux` installs on its own, as the `rugnux` package or as a standalone archive — see [Installing rugnux](RUGNUX_INSTALL.md).