Jungfraujoch/docs/RUGNUX_REPORT.md

# The results report

```{contents} On this page
:local:
:depth: 2
```

`<prefix>_report.txt` records **what the run determined**, next to the reflection files. It is
written on every `--mode mx` and `--mode scale` run that has an output prefix — there is no option
to enable or disable it. Two cases follow from that:

- An **empty output prefix** (`-o ""`, the "compute the statistics, persist nothing" mode) writes
  nothing, the report included.
- **`--no-merge`** still writes a report. It determined an indexing and a geometry result, and those
  are recorded; the merging section then says `MERGE= NOT_PERFORMED` rather than being omitted, so
  the absence is a statement and not something a reader has to infer.

The report is never allowed to fail a run: if it cannot be written (unwritable path, full disk) the
failure is logged as a warning and the run finishes normally.

**Exit status.** Rugnux exits 0 when the run completed — everything it determined, declined to
determine (`CANNOT_DETERMINE`) or warned about is in the report — and non-zero when the run
stopped: unreadable input, no usable lattice, a `-S` group the indexed lattice cannot host, an
indexer that could not run. The reason goes to the terminal, and the report may not exist in that
case — so a script branches on the exit status first and greps the report second.

## Format

The model is XDS's `CORRECT.LP`: prose and tables a crystallographer reads top to bottom, with a
structure a script can consume without parsing prose. Every line is one of three kinds — a
`KEY= value` data line, a `#` comment, or blank — so `grep -v '^#'` leaves the data alone
(since `REPORT_VERSION= 8`; before that, comment lines had no prefix).

- **`KEY= value` assignment lines.** Every number worth extracting is one, so a consumer gets it with
  a single `grep '^ISA= '` and never has to read a sentence. Key names are stable.
- **`#` comment lines** — everything else: the prose, the section banners, and the fixed-width
  tables (stable header row; the resolution shells, the space-group candidates, the sweep-quality
  ranges).
- **`# WARNING:` lines**, one per finding, in plain English: `# WARNING: Frames 500-600 out of beam
  (10.1 deg, scale 0.12 and CC 0.30 of the run, 2% scaled)`. `grep '^# WARNING:'` finds every one.

The worked examples on this page are shown with the leading `#` stripped for readability; in the
file itself every such line starts with `#`.

**A quantity the run did not measure writes no key at all**, and the fixed-width tables print `-` in
its place. There is one rule and no placeholders — no `nan`, and no `0.0%` that reads as a measured
total failure — so a consumer must treat an absent key as *not measured* rather than assume every key
it knows about is present. A measured value always prints, including a negative one. The keys a
script meets this on first are `SIGANO=` and `CC_ANOM=`: a rotation merge measures them whether or
not `-A` was given, but where no Bijvoet pair could be split in both hands — a stills merge without
`-A`, or too few pairs — the quantity does not exist and the key is absent; `COMPLETENESS=`,
`MULTIPLICITY=`, `I_OVER_SIGMA=`, `R_MEAS=`, `CC_HALF=` and `WILSON_B=` follow the same rule.

The last blocks before `END OF REPORT` are the authorship and the acknowledgement: who wrote
rugnux, its licence (GPLv3 — free to use for academic institutions and commercial companies alike)
and where releases are published, then the credit to the X-ray research community whose methods
rugnux implements and the open-source projects it is built on — both credited in
`ACKNOWLEDGEMENT.md` beside `LICENSE` and `THIRD_PARTY_NOTICES.md` in the installed package.
rugnux prints the acknowledgement at startup as well.

`REPORT_VERSION=` is the format's own version. Key names, table columns and the reason vocabulary
below are an interface other software may depend on: they do not change without that number moving.
Adding a key does not move it — a consumer that greps for what it needs is unaffected by one more
line. It counts releases, not changes: it moves at most once per release, however many format
changes that release carries, because a reader only ever meets the format that was released.

The header block above section 1 records **how the result was produced**: `RUGNUX_VERSION=` and
`RUGNUX_DOWNLOAD=` (the release page of exactly that version), `RUGNUX_GIT=`, `BUILD_CXX_FLAGS=`,
`DATE=`, `INPUT_FILE=` and `OUTPUT_PREFIX=`, plus

- **`RUGNUX_GIT=`** — the commit the binary was built from, stamped at build time so it cannot go
  stale in a reconfigured tree; a `-dirty` suffix marks a build from uncommitted changes.
- **`BUILD_CXX_FLAGS=`** — the compiler flags of the build (`NONE` for a plain configure). Two
  builds of one commit can differ by flags alone, and `-march` moves the CPU-bound results, so a
  comparison of two reports starts here.

- **`COMMAND_LINE=`** — the invocation as one shell-ready line, arguments containing spaces quoted.
- **`WALL_TIME=`** — the whole invocation in seconds. It covers everything the process did, opening
  the file and setting up included, so it is a little larger than the `Processing time` printed on
  stdout, which starts once the analysis does.
- **`GPU_COUNT=`** and **`GPU=`** — how many GPUs were visible and what they are, e.g.
  `GPU= 4x NVIDIA A100-SXM4-80GB`; several models on one machine are listed as separate groups.
  `GPU_COUNT= 0` appears on its own, with no `GPU=` line, when nothing was visible — which is the
  first thing to check when a run took far longer than expected. Rugnux prints the same line at
  startup, before the run, so a missing GPU can be caught while there is still time to stop.

Rates, per-image costs and progress remain on stdout only.

Sections, in order: the `SUMMARY`, then `1. DATA SET AND GEOMETRY`, `2. CRYSTAL`,
`3. MERGED DATA`, `4. DIAGNOSTICS`, and `5. MODEL VALIDATION` only with `--model`
(see [The summary](#the-summary-and-what-the-run-decided) below). The numbering is contiguous, and
the fifth section appearing renumbers nothing; a stage that did not run states that inside its
section — `MERGE= NOT_PERFORMED` — rather than the section disappearing.

**`SPOT_RESOLUTION_ESTIMATE=`** in section 1 is how far the merged data are expected to reach, read
off the found spots alone — no lattice, no integration, no merge — so it is there on a run that never
merges, and on a run that does it can be read against `INCLUDE_RESOLUTION_RANGE` in section 3. It is a
prediction, good to about 0.2 Å on rotation data; nothing is cut on it. It is **not** limited to what
the detector records: where it reads finer than the high-resolution end of `INCLUDE_RESOLUTION_RANGE`,
the crystal diffracts past the corner and the run was detector-limited.

**`JFJOCH_DATASET_SETTINGS=`** in section 1 is the geometry the run integrated at — on a rotation run
the post-refined one — written as the object `jfjoch_broker` takes it in: the four required properties
of `dataset_settings` in `broker/jfjoch_api.yaml`, joined by the three `poni_rot*_rad` angles whenever
any of them is non-zero (a body without them states a flat detector), on one line of valid JSON, so a refined beam centre
and distance can go back to the instrument for the next collection without anyone retyping them.

```
JFJOCH_DATASET_SETTINGS= {"beam_x_pxl": 2078.24, "beam_y_pxl": 2233.92, "detector_distance_mm": 190.311, "incident_energy_keV": 12.4000}
```

```bash
grep '^JFJOCH_DATASET_SETTINGS=' out_report.txt | cut -d' ' -f2- > geometry.json
```

**Which pass.** A rotation run integrates twice — once at the geometry in the input file, then again
at the post-refined geometry — and can integrate a third time if a guard rejects the second pass.
There is **one** report, for the pass that became the canonical output, and `PASS=` /
`PASS_DECISION=` (`--developer`) in section 1 say which pass that is and on what evidence, so no number in the file
is ambiguous about which geometry produced it.

**Not in the report:** timing, frame rates, thread counts, per-image progress and library banners.
Those are process, not result, and stay on stdout.

## The summary, and what the run decided

The file opens with a **`SUMMARY`** section, above everything it summarises. It exists because the
report used to have no evaluative line anywhere until its last section: a run that produced garbage
and a run that produced a textbook data set read identically for their first three hundred lines.

- **`VERDICT=`** is a closed vocabulary — `OK`, `WARNINGS`, `UNUSABLE`, `FAILED`. `FAILED` means no
  lattice was determined or the run was cancelled; `UNUSABLE` means the data merged but carry no
  usable signal; `WARNINGS` means something else needs attention; `OK` means nothing did. It is
  decided from the warnings the rest of the report produced, so it introduces no new analysis and
  cannot disagree with the sections below it.
- **`VERDICT_TEXT=`** is one to three sentences of free text saying the same thing in English.
- **`PATHOLOGY_FLAGS=`** is the *type* of each condition that fired, from a closed vocabulary, so a
  consumer switches on a code rather than parsing a sentence: `NO_LATTICE`, `INDEXING_AMBIGUITY`,
  `SYMMETRY_AMBIGUITY`, `CENTERING_UNTESTED`, `UNUSABLE_MERGE`, `LOW_COMPLETENESS`, `SWEEP_GAPS`,
  `GONIO_SCALE`, `SPINDLE_CAP`, `ANISOTROPY`, `TWINNING`, `PSEUDO_TRANSLATION`,
  `LATTICE_TRANSLATION`, `MODEL_HAND`, `MODEL_NOT_VALIDATED`, `CANCELLED`, `RESOLUTION_FIT`,
  `FLIGHT_PATH`. `NONE` when nothing fired. A code appears if and only if its
  warning fired, so the flags and the `WARNING:` lines are two renderings of one list — the closed
  type for machinery, the open sentence for a person.
- **`WARNING_COUNT=`** and the **`WARNING:`** lines follow, in the same section. They are what they
  always were; they have moved from the bottom of the file to the top.

Then the numbered sections: **1. DATA SET AND GEOMETRY**, **2. CRYSTAL**,
**3. MERGED DATA**, **4. DIAGNOSTICS**, and **5. MODEL VALIDATION** only with `--model`.

## The reference-range table

`--report-resolution <dmin>[,<dmax>]` adds a second block of merging statistics to section 3, the
`REFRES_*` keys and a second shell table, over the resolution range it is given rather than the range
the run chose. It exists for comparison: another program's table is at that program's range, and
running rugnux *at* that range (`--scaling-high-resolution`) is not the same run — the cut moves, and
with it the symmetry decision and everything downstream of it. The reference table is instead the
same merged reflections binned again, so nothing is processed differently whether or not it is
asked for; the merged files and every decision are byte-for-byte the run's own.

- **`REFRES_RANGE=`** is the requested range (`dmax dmin`, as `INCLUDE_RESOLUTION_RANGE`; `INF` when
  the run has no low-resolution limit and none was given). **`REFRES_MEASURED_RANGE=`** is the
  coarsest and finest merged reflection that actually landed in it.
- **`REFRES_COMPLETENESS=`**, **`REFRES_MULTIPLICITY=`**, **`REFRES_I_OVER_SIGMA=`**,
  **`REFRES_R_MEAS=`**, **`REFRES_CC_HALF=`**, **`REFRES_SIGANO=`**, **`REFRES_CC_ANOM=`**,
  **`REFRES_UNIQUE_REFLECTIONS=`** and **`REFRES_TOTAL_OBSERVATIONS=`** are the overall numbers over
  that range, under the same absent-when-unmeasured rule as their section-3 namesakes.
  **`REFRES_ISA=`** is the error model refitted on the reflections of this table alone, in XDS's
  convention, so it reads against an `ISa` produced at that range; the table itself is merged under
  the run's own model.
- **The table holds only what the run kept.** Where the reference range is finer than the run's own
  limit, the shells past that limit are empty by the run's decision — it judged them to carry no
  signal and did not merge them — and are printed as `past the run's own limit of X A: not merged
  (N possible)` rather than as zeros; the shell the limit falls inside is marked. **`REFRES_SHELLS_PAST_LIMIT=`**
  counts those shells (0 when the range lies within the run's own), so a consumer can tell *not
  merged* from a measured zero. `REFRES_COMPLETENESS` counts their reflections as missing; every
  other overall number is over the shells the run reached.

The shells are equal steps in 1/d² between the two bounds, as XDS's are, so at XDS's
`INCLUDE_RESOLUTION_RANGE` the two tables read row for row.

## The developer report

`--developer` renders the same report in full. The default report carries what a person deciding
*keep or recollect* acts on; `--developer` adds the pipeline's own internals — the anisotropy
detection gate's parameters, the space-group operator and candidate tables, the model-fit null, the
sweep-quality internals, the twinning statistics measured before the search, and the long
explanatory passages — plus advisories about the cut's own behaviour that no user can act on.

Nothing is computed differently and nothing is lost by leaving the flag off: the report is built
once, in full, and the flag selects how much of it is written. Every key the default report writes,
`--developer` writes too.

## The space group, and the Sohncke answer beside it

The space group lives in section 2: `SPACE_GROUP_NAME=` / `SPACE_GROUP_NUMBER=`, with
`SPACE_GROUP_ALTERNATIVES=` naming the candidates the data could not separate — an enantiomorphic
partner among them — and, on a group that has such a partner, `SPACE_GROUP_ENANTIOMORPH=` saying
whether the hand is open (`UNDETERMINED`), asserted by the user (`GIVEN`), or taken from an accepted
model (`ASSUMED_FROM_MODEL`).

`SOHNCKE_SPACE_GROUP=` in section 2 is written on every run whose space group was determined by the
search; a run given its group with `-S` has no Sohncke candidate to name and omits the key. Where the
search found a glide plane, `SPACE_GROUP_NAME=` names the group with it and this names the best group
without - a crystal of chiral molecules, which is any protein, cannot have a glide plane or an
inversion centre, so a reader who knows their sample is a protein reads this key and needs no second
run. Where no glide was found the two keys read the same, deliberately: greppability is the point, and
a key that appears only sometimes has to be tested for before it can be read.

`SPACE_GROUP_ENANTIOMORPH=` in section 2 reads **`ASSUMED_FROM_MODEL`** when the hand written in the
files is the model's. Assumed, not determined: merged intensities cannot see the hand at all — |F| is
invariant under the change of hand — so an accepted model asserts it out of prior chemical knowledge.
It is only ever written where `MODEL_FIT= ACCEPTED`, and the anomalous difference map vetoes it
outright where the map says the model and the data are in opposite hands. (Before `REPORT_VERSION= 6`
this value was spelled `DETERMINED_FROM_MODEL` and was emitted whenever a model file merely parsed.)

## Sweep quality, the disposition, and their vocabularies

Section 4 lists the stretches of the sweep over which the crystal delivered much less than the rest
of the run — the feedback a beamline control system needs to tell an operator that a crystal should
be recentred or recollected — says what became of each of them, measures what keeping each one
costs the merged intensities, and drops the stretches that cost too much.

```
SWEEP_QUALITY_STATUS= COMPUTED
SWEEP_QUALITY_COUNT= 2
SWEEP_QUALITY_REASONS= no_diffraction crystal_out_of_beam weak_diffraction loss_of_centring radiation_damage inconsistent_with_merge
SWEEP_DISPOSITIONS= merged downgraded rejected
FRAMES_MERGED= 1663
FRAMES_DOWNGRADED= 101
FRAMES_REJECTED= 36
FRAMES_REJECTED_PCT= 2.00
ROTATION_REJECTED_DEG= 3.6
SWEEP_ROTATION= 180.0
FLUX_PEAK_TO_TROUGH= 1.03
SCALE_MODULATION_PEAK_TO_TROUGH= 1.00

  FIRST_IMAGE   LAST_IMAGE   N_IMAGES  ROTATION  REASON                   SEVERITY   SCALE      CC   INDEXED  DISPOSITION  DELTA_CC_HALF  DELTA_CC_HALF_SE
  -----------  -----------  ---------  --------  -----------------------  --------  ------  ------  --------  -----------  -------------  ----------------
          500          600        101      10.1  crystal_out_of_beam          0.83    0.12    0.30      0.02  downgraded         +0.0004            0.0031
          612          630         19       1.9  no_diffraction               0.98    0.02    0.00      0.00  rejected           -0.0481            0.0110
  -----------  -----------  ---------  --------  -----------------------  --------  ------  ------  --------  -----------  -------------  ----------------
```

Of those keys, `SWEEP_QUALITY_COUNT` and the five disposition keys (`FRAMES_MERGED`,
`FRAMES_DOWNGRADED`, `FRAMES_REJECTED`, `FRAMES_REJECTED_PCT`, `ROTATION_REJECTED_DEG`) are in the
default report; `SWEEP_QUALITY_STATUS`, `SWEEP_QUALITY_REASONS`, `SWEEP_DISPOSITIONS`,
`SWEEP_ROTATION`, `FLUX_PEAK_TO_TROUGH` and `SCALE_MODULATION_PEAK_TO_TROUGH` appear with
`--developer` (the default report states in prose whether the diagnostic ran).

The three frame counts **partition the sweep** — every processed image is exactly one of them and
they add up to the frame count — so `FRAMES_REJECTED_PCT` is the answer to "how much of this
experiment was useless". It is reported beside `ROTATION_REJECTED_DEG` on purpose: a percentage of
*frames* moves when the same experiment is re-sliced, and a percentage of the *rotation* does not.
The prose headline above the table states both.

`SWEEP_QUALITY_STATUS` distinguishes **`COMPUTED`** (the diagnostic ran; a count of 0 means the sweep
was clean throughout) from **`NOT_COMPUTED`** (it did not run — no scaling and merging, or stills
data). A consumer must not read a missing table or a zero count as "clean" without checking it.
`SWEEP_QUALITY_REASONS` lists the whole vocabulary this version can emit, so an unknown code is
distinguishable from a missing one.

| Reason code | Meaning |
|-------------|---------|
| `no_diffraction` | The range recorded essentially no diffraction from the indexed lattice. |
| `crystal_out_of_beam` | Frames were lost: over the range a per-image scale could be fitted far less often than over the run. |
| `weak_diffraction` | The frames all still index, but with much less intensity — the cause was not determined. |
| `loss_of_centring` | One cycle of modulation per revolution: the crystal is off the rotation axis. |
| `radiation_damage` | The range runs to the end of a sweep whose quality was already decaying. |
| `inconsistent_with_merge` | The frames diffract as the run does, but their intensities do not agree with it — the only evidence is `DELTA_CC_HALF`, so the cause is not named. |

The vocabulary is **closed and stable**: a code is never renamed, and never reused for a different
meaning. New codes are only ever added, and adding one moves `REPORT_VERSION` at the next release.

The columns are: `FIRST_IMAGE`/`LAST_IMAGE` — inclusive, in processed-image ordinals (the numbering
of `<prefix>_image.dat` and of every other per-image array `rugnux` writes; with `-s`/`--stride` the
source image is `start + ordinal * stride`); `ROTATION` — the width of the range in degrees;
`SEVERITY` — the fraction of the run's typical diffracting power missing over the range, 0 (as good
as the run) to 1 (nothing at all); `SCALE` and `CC` — the range's mean per-image scale and
CC-to-merge relative to the run median; `INDEXED` — the fraction of the range's frames that were
scaled at all; `DISPOSITION` — what became of it; `DELTA_CC_HALF` and `DELTA_CC_HALF_SE` — what
keeping it costs the merged intensities, and how precisely that is known. Every range also appears as
a `WARNING:` sentence in the SUMMARY, with the same cost in words. A range is split where its
disposition changes, so each row is wholly kept or wholly rejected.

### What the disposition means, and what decides it

| Disposition | Meaning |
|-------------|---------|
| `merged` | The frame's observations are in the merged data at their own weight. |
| `downgraded` | They are in the merged data, but over a stretch the run itself flagged, carried at the reduced weight the frame's own scale and sigmas give it. Nothing extra is subtracted: for weak-but-consistent data that reduced weight *is* the honest weight, and a second, invented per-frame weight would double-count with the σ's. |
| `rejected` | Nothing of the frame reached the merge — because ΔCC1/2 convicted it, because an earlier guard dropped a frame whose scale had collapsed to an unusable number, or because the frame recorded nothing to drop in the first place. To a user asking how much of the experiment was useless these are the same answer, and the `REASON` column separates them. |

`DELTA_CC_HALF` is **ΔCC1/2**: the overall CC1/2 of the merged data **with** the range minus the
CC1/2 **without** it, evaluated over the reflections the range touches. Negative means keeping the
range makes the merged intensities worse. It is computed in the σ-τ form — no random half-dataset
split, so the same input gives the same answer every run — with each reflection's error variance taken
from the observed scatter of its own observations, not from the error model's σ's (a bad stretch claims
the same σ's as a good one, so an error-model estimate would read a stretch that adds noise as one that
adds precision). It is a CC1/2 over **the range's own reflections**, not over the whole dataset — a
range that touches a few hundred reflections can carry a large ΔCC1/2 without the dataset's headline
CC1/2 moving by anything like as much. `DELTA_CC_HALF_SE` is the standard error of a CC1/2 on that
many reflections, in the same units, and a ΔCC1/2 smaller than it says nothing.

A range is rejected only where ΔCC1/2 is **both well below the rest of this run's own batches and
several standard errors below zero**, and only where **every frame in it is one the per-image channels
call worse than the run's typical frame** — the scale and the CC to the merge, frame by frame and never
as an average over the stretch, because an average cannot tell a uniformly bad stretch from a healthy
arc lying beside a dead one. All three are needed: a healthy crystal merges at
CC1/2 ≈ 0.999, where a harm of 0.001 in CC is already many standard errors, so significance alone
convicts frames on clean data whose removal moves nothing; and ΔCC1/2 is itself measured against
the merge, so removing whichever frames disagree with it most improves every agreement statistic
whether or not anything was wrong with them — the decision has to be triggered by a channel that owes
nothing to the merge, and only then confirmed by what the merge does. The test runs last, after the per-frame scale, the
decay slope and the per-batch relative-*B* have been fitted, so it judges corrected data. The decision
is taken over 10° batches — the same batches as the radiation-damage curve — and over those batches
doubled, and doubled again, up to a quarter of the sweep: a defect much longer than a batch is
invisible one batch at a time, because each batch inside it is judged against a merge that still
contains the rest of the defect. Where the harm lies is then settled finely: each edge of the
convicted stretch is slid frame by frame with the *whole* stretch re-measured at every position, so
the range is reported where it actually lies rather than at the batch grid, and the reflection count
the verdict rests on never shrinks with the edge. Each edge is then pulled back off any frame the
per-image channels call normal, and what is left has to carry the verdict again on its own: a stretch
that cannot be taken without healthy frames holds more than one thing and is not removed at all. A rejected
stretch is never narrower than one rocking event, because the partials of one event are combined into
the same intensities and inside it no frame can be judged apart from its neighbours. Never more than
a quarter of the sweep is removed.

**What ΔCC1/2 cannot do**, because the report must not imply otherwise:

- it says nothing about the **cause**: a shutter fault and a crystal that slipped have the identical
  signature, both integrating background, so the cause comes from the `REASON` column and never from
  the ΔCC1/2 itself;
- a **second lattice entering** is invisible to it: those spots were never integrated, so they are not
  in the merged intensities it measures;
- a **centring drift that is pure attenuation** reads ≈ 0. That is the right answer, not a blind spot:
  the data are weak but consistent, the σ's already say so, and their ΔCC1/2 is the evidence that
  discarding them would cost completeness for nothing;
- it is attributed to the frame carrying a rocking event's **peak** partial, so it cannot resolve a
  single frame: a stretch narrower than one rocking event is never rejected, and the edges of a
  rejected stretch are soft to within half an event;
- `loss_of_centring` needs ≥ 350° of sweep to be named at all. On a 90° sweep the same drift is still
  *detected*, only as `crystal_out_of_beam` or `weak_diffraction` — "cause not determined" here means
  this sweep cannot determine it, not that it is undeterminable.

The same finding is written **per image** into the `_process.h5` as `/entry/MX/sweepQuality` and
`/entry/MX/frameDisposition`, when one is written — see
[HDF5](HDF5.md#41-entrymx--spot-finding-and-indexing-cxi-style).

`SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` and `SPINDLE_SYMMETRY_AXIS_ORDER=` (`--developer`) in section 4 say how the crystal
sat on the goniometer: the angle between the spindle and the nearest symmetry axis, and that axis's
order. They are descriptive: neither convicts nor clears the mounting on its own, because an aligned
axis of any order maps the sweep's blind cone onto itself while an axis near perpendicular does the
same only when it is a lone 2-fold, and only the nearest axis is reported. (New in
`REPORT_VERSION= 7`.)

`SPINDLE_LOST_UNIQUE_FRACTION=` in section 4 is the exact verdict the angle cannot give: the fraction
(0-1, so 0.0300 means 3%) of unique reflections, to this run's resolution limit, that the mounting
made unmeasurable - the part of the sweep's blind double cone that no operator of the measured point
group maps onto measured territory, computed in the crystal's actual indexed orientation. 0.0000
means the mounting cost nothing; the run warns when the group recovers less than half of the cone's
content. The same number is written to the master file as `/entry/MX/spindleLostUniqueFraction`, so a
pipeline can read it from either output without parsing prose. Written on every rotation run that
determined a space group and merged reflections.

## Powder contamination

A crystalline phase other than the crystal, diffracting as rings among its reflections — hexagonal
ice, a shower of microcrystals, a salt out of the cryoprotectant. It is measured on **every** run, in
the pre-scan, from the spots found there, and reported whether or not anything acted on it: a user
whose crystal sat in a powder is told so even where the run indexed perfectly well. Only hexagonal ice
has rings that can be named in advance, so `POWDER_RINGS_A` is what *this* sample showed.

```
POWDER_RINGS_DETECTED= TRUE
POWDER_RING_COUNT= 24
POWDER_SPOT_FRACTION= 0.412
POWDER_RINGS_SEPARABLE_TO= 2.31
POWDER_RINGS_A= 3.897 3.671 3.447 ...
POWDER_EXCLUDED_FROM_INDEXING= TRUE
POWDER_INDEXING_D_MIN= 4.91
```

`POWDER_RINGS_DETECTED` is written on every merging run and is `FALSE` on nearly all of them; the
rest of the keys appear only where rings were found. `POWDER_SPOT_FRACTION` is the share of the
pre-scan's spots the rings hold *over* the smooth fall-off around them — what the contaminant
contributes, not what happens to lie in a ring band. `POWDER_RINGS_SEPARABLE_TO` is the resolution
past which the rings crowd together too tightly to be told apart, and so the finest an indexing pass
can be asked to trust on such a pattern; it is absent where they stay separable over the whole range,
which is the ordinary case. `POWDER_EXCLUDED_FROM_INDEXING` says whether the run needed them
left out to index at all (with `POWDER_INDEXING_D_MIN` the resolution the retried first pass used).
Rings are detected far more often than they are excluded: exclusion happens only where a first pass
found no usable lattice. The measurement is described in
[CPU/GPU data analysis ▸ Resolution and ice-ring handling](CPU_DATA_ANALYSIS_IMAGE.md#33-resolution-and-ice-ring-handling).

## Translational pseudo-symmetry

Two copies of the contents of the asymmetric unit related by a pure translation that is not a lattice
vector. It is the pathology that most reliably breaks molecular replacement, because the modulation it
puts on the intensities is not in the search model. Section 4 reports it beside twinning, because the
two interact. The algorithm is in
[CPU/GPU data analysis ▸ Twinning and translational pseudo-symmetry](CPU_DATA_ANALYSIS_DECISIONS.md#132-twinning-check-and-translational-pseudo-symmetry).

**`TNCS_DETECTED=`** is `TRUE`, `FALSE`, `INCONCLUSIVE` or `NOT_MEASURED`. The last two are not
`FALSE`: `NOT_MEASURED` means the merge has too few reflections in 20–5 Å to compute the statistic at
all, `INCONCLUSIVE` means the Patterson was measured but too few acentric reflections remain to test
whether the vector it names modulates the intensities. Neither is a statement that the crystal has no
pseudo-symmetry.

`TRUE` requires **both** of two tests, because either alone over-calls by about a factor of two:

- **`TNCS_PATTERSON_PEAK_PCT=`** — the largest off-origin peak of the native Patterson, as a
  percentage of the origin peak, counting only peaks farther than 15 Å from any origin-equivalent
  lattice point. **`TNCS_PATTERSON_PEAK_Z=`** scores it against **`TNCS_PATTERSON_NULL_PCT=`**, the
  same map recomputed with the intensities permuted within resolution shells. The null is per dataset
  and not a table: the noise floor of this statistic runs from about 1% on a large merge to about 18%
  on a small one, so no fixed percentage separates the two populations.
- **`TNCS_MODULATION=`** — the ratio of the strongest to the weakest bin mean of ⟨E²⟩ over the phase
  frac(**h**·**u**), with **u** the refined peak vector (**`TNCS_VECTOR=`**, fractional, and
  **`TNCS_VECTOR_LENGTH=`** in Å). **`TNCS_MODULATION_NULL=`** is the same search started from random
  vectors, so the contrast the search itself can manufacture is measured rather than assumed.

The vector is good to about 0.05 fractional. It is a starting point for a program that refines it,
not a refined result: the refinement maximises the modulation, not the accuracy of the vector.

**`TNCS_PSEUDO_CENTRED=`** and **`TNCS_SUBLATTICE=`** are separate claims and are deliberately not
merged. A vector that is a rational translation `1/q` of the cell means the crystal is pseudo-centred;
the cell itself is not in question, because the suppressed class is weak rather than absent and a
smaller cell would contradict it. Only `TNCS_SUBLATTICE= NEAR_EXTINCT_CLASS` — the suppressed class
almost gone — says the reported cell may be a supercell.

**`UNDECLARED_LATTICE_TRANSLATION=`** is a different finding, and is reported instead of a
pseudo-symmetry rather than as one. The merged data are *exactly* invariant under the translation, and
a translation the data are exactly invariant under is a lattice vector by definition — so the centring
or the cell is wrong, not the packing. It is what a centred lattice merged in P1 looks like, which
`--mode scale` on a file with no space group produces by design. The pseudo-symmetry search continues
underneath it, so a real pseudo-translation sitting under an undeclared centring is still found.

**`L_TEST_VS_TNCS=`** says how the twinning L-test beside it coped. A pseudo-translation **u** biases
⟨|L|⟩ upwards unless the partner reflection at **h** + **s** shares its class, which happens exactly
when **s**·**u** is an integer; a half-integer **u** — a pseudo-centering — is preserved by the
ordinary axis step of 2 and reads `UNAFFECTED`. Where it is not, the steps are restricted to those
that do preserve the class (`REPAIRED`), and where no step does, `UNREADABLE` says the statistic was
dropped from the twin verdict **in both directions**: it can no longer indicate a twin, and it can no
longer be read as proof that there is none. The second moment then decides alone.

## Diffraction anisotropy

Section 4 also reports how much the fall-off with resolution depends on **direction**, and whether that is
established above the data set's own systematic error. It runs automatically on every merging run —
there is no flag — and it is a **description only**: no intensity is corrected, no reflection is
removed on a directional criterion, and the merged data and the written reflection files do not
depend on direction at all. The algorithm is in
[CPU/GPU data analysis ▸ Diffraction anisotropy](CPU_DATA_ANALYSIS_DECISIONS.md#135-diffraction-anisotropy).

Two different quantities are reported and they are not interchangeable. `ANISOTROPY_DELTA_B` is a
*rate* — the range of the principal components of the anisotropy tensor, on the ordinary
crystallographic *B* scale, so it is directly comparable with phenix.xtriage's `B_cart`, ctruncate's
anisotropic *B* and AIMLESS's anisotropic Δ*B*. `ANISOTROPY_D_MIN_PRINCIPAL` is where the signal
actually *runs out* along each principal direction. A crystal can have a large Δ*B* and almost no
spread in directional limit, or the reverse.

| key | meaning |
|---|---|
| `ANISOTROPY_VERDICT` | `DETECTED` \| `NOT_DETECTED` \| `CANNOT_DETERMINE` |
| `ANISOTROPY_FREE_DIRECTIONS` | Deviatoric directions the Laue class allows — 5 triclinic, 3 monoclinic, 2 orthorhombic, 1 tetragonal/trigonal/hexagonal, **0 cubic** |
| `ANISOTROPY_DELTA_B` | The anisotropic Δ*B* (Ų), fitted on intensities with nothing dropped |
| `ANISOTROPY_DELTA_B_LINEAR` | The part of it that follows exp(−½ **s**ᵀ*B***s**). **This is the number the verdict is gated on**, and the report says which of the two it is quoting |
| `ANISOTROPY_PRINCIPAL_B` | The three principal components, relative to the weakest |
| `ANISOTROPY_D_MIN_PRINCIPAL` | Diffraction limit (Å) along each principal direction — where ⟨I/σ(I)⟩ in a 20° cone about it falls through 2 |
| `ANISOTROPY_D_MIN_CENSORED` | One flag per direction. `1` means ⟨I/σ(I)⟩ never fell through 2, so the limit is the **edge of the measured data**, a bound and not a measurement. The prose marks it with a `<` |
| `ANISOTROPY_D_MIN_SPREAD` | Range of the three limits — itself a lower bound if any is censored |
| `ANISOTROPY_SHAPE` | `LINEAR` (a real Debye–Waller *B*) \| `FLAT` (the deficit does not follow a *B* at all, so Δ*B* may be an **under**-estimate) \| `CONVEX` (grows faster than a *B* can) \| `UNDETERMINED` (the verdict moved on rebinning) |
| `ANISOTROPY_FLOOR`, `ANISOTROPY_SIGNIFICANCE` | The data set's own systematic-error floor (Ų) and Δ*B*<sub>linear</sub> over it. Banded: below 2 not established, 2–3.5 marginal, above 3.5 established, above 5 strong |
| `ANISOTROPY_DETECTION_LIMIT` | The smallest Δ*B* that could have been established on these data. It is set by systematic error, not by counting, so it does **not** improve with more reflections or a longer exposure |
| `ANISOTROPY_N_OBSERVATIONS`, `ANISOTROPY_FORBIDDEN_Z`, `ANISOTROPY_SIGMA_SYSTEMATIC` | The unmerged observations the floor was measured on, that measurement against its own counting noise, and the floor before the counting part is added back |

The default report carries `ANISOTROPY_VERDICT`, `ANISOTROPY_DELTA_B`, `ANISOTROPY_D_MIN_PRINCIPAL`
and `ANISOTROPY_D_MIN_SPREAD`; the rest of this table — the detection gate's own parameters — is
written with `--developer`.

**`CANNOT_DETERMINE` is a real answer, not an evasion.** The verdict is not measured against counting
statistics — real data carry systematic error far larger than that, and gating on counting error
reports anisotropy on data sets that have none. Instead the data set measures its own systematic
error in the tensor directions its Laue class *forbids*, where the true value is exactly zero
whatever the crystal is. Where that measurement cannot be made, the run says so and gives the
reason: a **triclinic** Laue class (no forbidden direction exists), an observed rotation under about
90°, merged data at the noise floor, a scale model carrying no dose term
(`--no-scaling-corrections`), or no unmerged observations. A **cubic** Laue class is different again
— symmetry forces Δ*B* to be exactly zero, and the run says that rather than reporting a measurement.

Where anisotropy is detected and the directional limits differ by more than 0.5 Å, a `WARNING:` line
says so, since refinement and map interpretation should allow for it.


## Model validation

Section 5 appears only with `--model`. It reports the supplied model against the merged data —
R-factors, maps, anomalous sites — and, separately, whether the data accepted the model at all.

The two are not the same question, and the report keeps them apart. **The R-factors, the maps and the
rigid-body placement describe the model**: they are computed and reported whatever the answer, because
a model that does not belong to this crystal still has an R against it, and that is the negative
result. **`MODEL_FIT=` is the answer**, and it is what governs whether the model was allowed to change
anything about the written reflections.

There is no threshold on R behind it. What a model that explains nothing reaches against a given data
set depends on its atom count and B-factors as much as on the data, so the same model is refitted —
and re-placed as a rigid body, exactly as the real one is — from `MODEL_FIT_NULL_REPLICATES` random
orientations about its own centroid, and `MODEL_FIT_SIGMA` is how far the real fit sits above that
distribution. The statistic is R-work, not R-free - not because nothing is refined against the working
set (the placement's six parameters are), but because every null replicate is placed the same way, so
what they buy is bought on both sides and cancels; and it is decided on an order of magnitude more
reflections than R-free.

That null is **only built where the model claims one of the two things it could change** — the
enantiomorph, or an indexing other than the one the data were merged in. A model already in the data's
space group on a crystal with no merohedral ambiguity, which is the isomorphous case a screening
campaign is made of, claims neither: `MODEL_FIT= NOT_TESTED`, `MODEL_DECISIONS_TAKEN= NONE`, and the
run does not pay for a null that would gate nothing. **`NOT_TESTED` is not `REJECTED`** — it says the
question was never put, not that the data answered it badly — and the three values are distinguishable
by grepping the one key. The `MODEL_FIT_NULL_*` and `MODEL_FIT_SIGMA` keys are absent in that case,
since there is no null to report; `R_WORK`, `R_FREE`, the maps and the rigid-body shift are all there
as usual.

| key | meaning |
|---|---|
| `MODEL_VALIDATION` | `PERFORMED` \| `NOT_PERFORMED` (with `MODEL_VALIDATION_REASON`, and no R-factors) |
| `MODEL_FIT` | `ACCEPTED` \| `REJECTED` \| `NOT_TESTED` — whether the model may decide anything, or had nothing to decide |
| `MODEL_FIT_STATISTIC` | What the verdict was taken on; `R_WORK` |
| `MODEL_FIT_VALUE`, `MODEL_FIT_NULL_MEAN`, `MODEL_FIT_NULL_SD`, `MODEL_FIT_NULL_REPLICATES` | The real fit, and the null of the same model in random orientations. Absent when `NOT_TESTED` |
| `MODEL_FIT_SIGMA` | The real fit above that null, in its standard deviations. Signed. Absent when `NOT_TESTED` |
| `MODEL_DECISIONS_TAKEN` | `NONE` \| `ENANTIOMORPH` \| `INDEXING` \| `ENANTIOMORPH+INDEXING` |
| `MODEL_ENANTIOMORPH_ADOPTED`, `MODEL_INDEXING_OPERATOR` | The two decisions individually; `x,y,z` is no reindexing |
| `MODEL_INDEXING_MARGIN`, `MODEL_INDEXING_MARGIN_NULL`, `MODEL_INDEXING_MARGIN_SIGMA` | Present only where a merohedral ambiguity was probed. The winner's lead over the runner-up in R-free, against the lead a random placement of the same model produces |

The null's own numbers — `MODEL_FIT_STATISTIC`, `MODEL_FIT_VALUE`, the `MODEL_FIT_NULL_*` keys and
the `MODEL_INDEXING_MARGIN*` keys — are written with `--developer`; the default report carries the
verdict (`MODEL_FIT`, `MODEL_FIT_SIGMA`, `MODEL_DECISIONS_TAKEN` and the two decisions).

### CC(model, data)

Beside the R-factors, section 5 carries the **correlation of the merged intensities with the placed,
scaled model**, |*F*<sub>model</sub>|², by resolution shell. The shells are the merge table's own, so
a row here can be read straight across from that shell's CC1/2 and R<sub>meas</sub> in
section 3. It is a correlation of *intensities*, like CC1/2 and CCref beside it, and the
observed value is the merged intensity itself rather than the French–Wilson |*F*|² the R-factors use —
that amplitude is a posterior mean under a Wilson prior, which pulls a weak reflection towards its
shell mean and would show up as correlation in exactly the outer shells this number is read in.

Nothing here was refined against these reflections — the model is placed and scaled with eleven
parameters — so there is no work/free distinction to draw: the correlation is unbiased on **all** the
reflections of a shell, not only the few hundred free ones, and `SIGMA` is correspondingly sharp.

| key | meaning |
|---|---|
| `CC_MODEL_OVERALL`, `CC_MODEL_REFLECTIONS` | The correlation over every reflection in the table, and how many — the `N` column sums to it. Like any overall correlation it is shell-weighted and can take any value between the best shell and the worst; the table is what to read |
| `CC_MODEL_CONFIRMED_TO_D_MIN` | The finest shell whose correlation reaches 3 σ, or `NONE`. A **lower bound** on the useful resolution |
| the `D_MIN / CC_MODEL / N / SIGMA` table | Per shell: the correlation, the reflections it was formed on, and how far above zero it sits (Fisher's transform, `atanh(CC)·√(N−3)`) |

**Read it in one direction only.** A shell whose correlation is significantly above zero carries
signal — a model cannot agree by accident with measurements it was never fitted to — so
`CC_MODEL_CONFIRMED_TO_D_MIN` is evidence for keeping *more* data. A shell whose correlation is near
zero says nothing about the data: the model may be incomplete, in the wrong hand, or simply wrong for
this crystal, and cutting on it would be cutting because the model is poor. Nothing in the pipeline
acts on these numbers; they are reported and no more. This is the same asymmetry cryo-EM works under,
where the half-map FSC sets the resolution and the model–map FSC only validates it.

A *significantly negative* correlation in a shell is worth chasing rather than ignoring: it cannot be
signal, so it points at a systematic error — an indexing the model disagrees with, or an outer shell
the scaling has mistreated.

`MODEL_DECISIONS_TAKEN= NONE` — whether the model was rejected or never tested — means the reflection
files are **byte for byte** what a run with no model would have written — same space group, same indexing, same `.mtz`, `.cif`, `.hkl` and
`_unmerged.mtz`. A rejected model is therefore safe to try: it costs the null's compute and changes
nothing else.