diff --git a/docs/EXTERNAL_TEST_DATA_STILLS.md b/docs/EXTERNAL_TEST_DATA_STILLS.md
index 490d4d97c..b68d40972 100644
--- a/docs/EXTERNAL_TEST_DATA_STILLS.md
+++ b/docs/EXTERNAL_TEST_DATA_STILLS.md
@@ -37,6 +37,15 @@ Every set of the manifest is in the table and every set in the table is in the m
same CXIDB entry, PDB entry, space group and dmin; `tools/battery/doc_check.py` checks
this. The beamline of `pal90` is the CXIDB entry's (NCI); the PDB entry 6IRK records CSI.
+## How the battery runs them
+
+Each set is one rugnux command, the way a serial user runs: the deposited cell and space group given
+(`-C`, `-S`), the wavelength where the files do not state it, `--crystfel-geometry` for the AGIPD and
+JUNGFRAU sets (the geometry file of the entry; the SACLA, PAL and LCLS files are placed without one),
+and the deposited model with `--model`, so that the report carries its R-free against the merge. The
+exact arguments are the `args` of each set in `tools/battery/still_manifest.json`; the
+[stills section of the tutorial](RUGNUX_TUTORIAL.md#still--serial-data) explains them.
+
## The files we use
Paths are relative to the still data root of the site config (`tools/battery/README.md`). CXIDB
@@ -69,8 +78,9 @@ that line differs.
**The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV,
68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The
`_int16` sets are copies with the same layout in which `/data/data` holds whole photons (keV divided
-by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller; rugnux reads them
-with `--gain 1` and merges them to the same numbers. Make them with
+by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller; the battery runs
+them with the same arguments as the float32 originals (rugnux measures the gain from the frames), and
+they merge to the same numbers. Make them with
`tools/battery/still/float32_to_int16.py ` (h5py and hdf5plugin),
into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV, keeping the tarball's
directory tree), named `_int16.h5`. One MbCO dark run (`run_000238`) holds no frames and has no copy.
diff --git a/docs/RUGNUX_BENCHMARKING.md b/docs/RUGNUX_BENCHMARKING.md
index e6785bd7e..0175538d7 100644
--- a/docs/RUGNUX_BENCHMARKING.md
+++ b/docs/RUGNUX_BENCHMARKING.md
@@ -26,6 +26,13 @@ the old ones were retired.
and also checks that the answers are right. It answers *how fast, on this machine*. It takes
minutes, and needs nothing on the machine but four open datasets.
+Serial stills have a battery of their own, `tools/battery/still_battery.py`, on the public datasets
+of [External test data: serial crystallography](EXTERNAL_TEST_DATA_STILLS.md). It shares the site
+config, the verbs and the run-directory layout, but scores nothing about the lattice or the symmetry
+(a serial user gives the cell and the space group): each row says what the merge is worth (R_split,
+the deposited model's R-free against it beside the deposition's own) and what it cost. The README's
+*The still battery* describes it.
+
The two are kept apart on purpose. A battery run reads hundreds of gigabytes once, so its wall clock
mostly measures the disk and the page cache (see [the page-cache trap](#the-page-cache)). Use the
battery to judge correctness and the timing benchmark, run under controlled conditions, to judge
@@ -189,9 +196,11 @@ noise, and one that reproduces is a real change.
## The REFMAC model check
-R-free from `rugnux --model` is a **trend number**. It is a rigid-body placement scored on rugnux's
+R-free from `rugnux --model` is a **trend number**. `R_FREE=` is the placed model scored on rugnux's
own free set, with a deliberately simple scale model, so it shows how a set moves between runs, but
-it cannot be compared with the deposition's R-free. `--model-check` adds an independent number that
+it cannot be compared with the deposition's R-free. rugnux then refines the placed model as well
+(`REFINE_R_FREE=`, recorded as `rfree_refined` in `results.json`, reported and never scored), still
+on its own free set, so that number is not comparable with the deposition's either. `--model-check` adds an independent number that
can be. It runs `tools/battery/model_check.py` on every open-arm set with a deposited X-ray model,
after rugnux has finished, so it cannot influence the processing. It needs CCP4: the runner sources
`$CCP4_SETUP`, by default `/opt/xtal/ccp4-9/bin/ccp4.setup-sh`, and calls the `refmac5` found there
@@ -288,7 +297,7 @@ $B timing --rugnux $R [--benchmark standard|inhouse] [--threads N] [--label L] [
The benchmark runs a few sets. Each starts from a model that the run has to place by molecular
replacement, so a single run times every stage: reading, pre-scan, indexing, integration, scaling,
-the space-group search, MR and model validation. It also checks that every answer is right.
+the space-group search, MR, model validation and the refinement of the placed model. It also checks that every answer is right.
`--benchmark standard`, the default, is the one to run on a new machine. It uses open data only:
three EIGER 16M rotation sets of ordinary size and quality, and a virus capsid as the large case.
@@ -574,6 +583,8 @@ section. For a rotation run the steps are, in order:
| Analyses | twinning, anisotropy and the other report-only analyses |
| Model validation | `--model` only: structure factors, scale fits, rigid-body placement, maps |
| Molecular replacement | only where the rigid body did not place the model (always, in the timing benchmark) |
+| Refinement | the placed model refined against the merge (wherever a model is placed, unless `--no-refine`) |
+| Doubled-cell hypothesis | only where an index-2 superstructure qualifies: the doubled cell integrated and merged as well ([Results report](RUGNUX_REPORT.md#index-2-superstructure)) |
| Writing files | MTZ, mmCIF, HKL, report |
A step that is slower on one machine than on another points at the code, and a step that is slower
diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md
index 38d3fb6ab..d88d3e004 100644
--- a/docs/RUGNUX_REPORT.md
+++ b/docs/RUGNUX_REPORT.md
@@ -119,10 +119,12 @@ The header block above section 1 records **how the result was produced**: `RUGNU
Rates, per-image costs and progress remain on stdout only.
-Sections, in order: the `SUMMARY`, then `1. DATA SET AND GEOMETRY`, `2. CRYSTAL`,
-`3. MERGED DATA`, `4. DIAGNOSTICS` and `5. MODEL VALIDATION` only with `--model` or `--mr` - which
-also holds molecular replacement's keys where it ran ([below](#molecular-replacement))
-(see [The summary](#the-summary-and-what-the-run-decided) below). The numbering is contiguous, and
+Sections, in order: the `SUMMARY` ([below](#the-summary-and-what-the-run-decided)), then
+`1. DATA SET AND GEOMETRY`, `2. CRYSTAL`, `3. MERGED DATA`, `4. DIAGNOSTICS` and, only where a model
+was given (`--model`, `--mr`, or a model file among the inputs), `5. MODEL VALIDATION` - which also
+holds the keys of [molecular replacement](#molecular-replacement) and of the
+[refinement](#refinement-refine_) where they ran. A `--mode refine` report has one section,
+`1. REFINEMENT`. The numbering is contiguous, and
the fifth section appearing renumbers nothing; a stage that did not run states that inside its
section — `MERGE= NOT_PERFORMED` — rather than the section disappearing.
@@ -207,6 +209,29 @@ is ambiguous about which geometry produced it.
**Not in the report:** timing, frame rates, thread counts, per-image progress and library banners.
Those are process, not result, and stay on stdout.
+## The headline keys
+
+The keys most scripts read, by section; the rest of this page explains the ones that need it.
+
+| key | meaning |
+|---|---|
+| `EXPERIMENT_TYPE` | `ROTATION` or `STILLS` (section 1) |
+| `IMAGES_PROCESSED`, `WAVELENGTH` | Images the run read, and the wavelength (Å) |
+| `OSCILLATION_RANGE`, `STARTING_ANGLE`, `ROTATION_AXIS` | Rotation only: the per-image rotation the run integrated at (deg: the file's, scaled where the run adopted a goniometer rotation scale), the first image's angle and the axis |
+| `DETECTOR_DISTANCE`, `BEAM_CENTRE`, `DETECTOR_TILT` | The geometry the run integrated at (mm; the point of normal incidence in pixels; the three PONI rotations in degrees) - on a rotation run the post-refined one, not the header's |
+| `DIRECT_BEAM` | Where the direct beam meets the detector under that tilt (pixels) |
+| `LATTICE_FOUND`, `UNIT_CELL_CONSTANTS`, `CELL_ESD` | Section 2: whether a lattice was found, the cell (a b c α β γ, Å and degrees) and its e.s.d.s ([below](#cell-esds)) |
+| `SPACE_GROUP_NAME`, `SPACE_GROUP_NUMBER` | The space group the files are written in ([below](#the-space-group-and-the-sohncke-answer-beside-it)) |
+| `MOSAICITY_DEG` | Rotation only ([below](#mosaicity)) |
+| `MERGE` | Section 3: `PERFORMED` or `NOT_PERFORMED` (`--no-merge`) |
+| `INCLUDE_RESOLUTION_RANGE` | `dmax dmin` of the written reflections (Å), as XDS's keyword |
+| `FITTED_RESOLUTION` | Where the CC1/2 fall-off crosses the `--resolution-cc-target`; absent where the curve is not a fall-off (a `RESOLUTION_FIT` warning says why) |
+| `UNIQUE_REFLECTIONS`, `TOTAL_OBSERVATIONS`, `COMPLETENESS`, `MULTIPLICITY` | Over the written range; completeness in percent |
+| `I_OVER_SIGMA`, `R_MEAS`, `CC_HALF`, `WILSON_B` | The overall merging statistics; the shell table beside them is in the comment lines |
+| `SIGANO`, `CC_ANOM`, `FRIEDELS_LAW` | The anomalous signal, and whether Friedel pairs were merged (`FALSE` with `-A`) |
+| `ERROR_MODEL_A`, `ERROR_MODEL_B`, `ISA` | The error model in XDS's convention and ISa = 1/sqrt(a·b); `ISA= undetermined` where b is zero within its error, so 1/b would be noise |
+| `R_WORK`, `R_FREE` | Section 5: the model as placed against these data ([below](#model-validation)); the refined model's are `REFINE_R_WORK`/`REFINE_R_FREE` |
+
## The summary, and what the run decided
The file opens with a **`SUMMARY`** section, above everything it summarises. It exists because the
@@ -239,7 +264,8 @@ and a run that produced a textbook data set read identically for their first thr
admits alternative indexing: one sweep is indexed consistently, so it matters only against other data;
the operators are **`INDEXING_AMBIGUITY_OPERATORS=`** in section 2). On serial stills the same ambiguity
mixes the hands in the merge. Where the space group was given (`-S`), the crystals are made
- consistent from the data alone and section 2 says so: **`INDEXING_AMBIGUITY_RESOLVED= data`**,
+ consistent from the data alone - by their own cells where those decide the indexing, and by the
+ intensities otherwise - and section 2 says so: **`INDEXING_AMBIGUITY_RESOLVED= data`**,
**`AMBIGUITY_CRYSTALS=`** (the crystals that took part), **`AMBIGUITY_REINDEXED=`** (of them, the ones
moved), **`AMBIGUITY_SEPARATION_SIGMA=`** (per crystal, how far its chosen indexing beats the best other
one in correlation with all the other crystals, in sigma; the median) and
@@ -257,7 +283,7 @@ and a run that produced a textbook data set read identically for their first thr
always were; they have moved from the bottom of the file to the top.
Then the numbered sections: **1. DATA SET AND GEOMETRY**, **2. CRYSTAL**,
-**3. MERGED DATA**, **4. DIAGNOSTICS**, and **5. MODEL VALIDATION** only with `--model`.
+**3. MERGED DATA**, **4. DIAGNOSTICS**, and **5. MODEL VALIDATION** only where a model was given.
## The reference-range table
@@ -327,8 +353,8 @@ per-zone screw table.
reindexing operator from the axes the space group was determined on (CCP4 style, `h,k,l` where
nothing moved) and what chose the setting - `STANDARD`, `CELL` (`-C`), `SPACE_GROUP` (a non-standard
`-S`), `MODEL` (a `--model` that fits) or `REFERENCE` (`-z`). `SPACE_GROUP_NAME`, the cell and every
-file the run writes are in that setting (see RUGNUX_ADVANCED.md, "The setting the files are written
-in").
+file the run writes are in that setting (see
+[The setting the files are written in](RUGNUX_ADVANCED.md#the-setting-the-files-are-written-in)).
`SOHNCKE_SPACE_GROUP=` in section 2 is written on every run whose space group was determined by the
search; a run given its group with `-S` has no Sohncke candidate to name and omits the key. Where the
@@ -744,7 +770,9 @@ scatter of their cells, over √blocks, is **`CELL_ESD_BLOCKS=`** (where at leas
fitted; **`CELL_ESD_BLOCK_COUNT=`** says how many). `CELL_ESD=` is, per parameter, the larger of the
two. **`CELL_ESD_REFLECTIONS=`** is the number of distinct reflections used and
**`CELL_ESD_THETA_RANGE_DEG=`** their θ range, written as `_cell_measurement.reflns_used`,
-`theta_min` and `theta_max` in the mmCIF.
+`theta_min` and `theta_max` in the mmCIF. With `--developer`, **`CELL_ESD_OBSERVATIONS=`** is the
+number of observations behind them and **`CELL_DSPACING_FIT=`** the cell the d-spacing fit itself
+returned.
Neither estimate includes what is common to every observation - the wavelength and the detector
distance scale the whole cell - so the e.s.d. is a precision, not an accuracy. Stills runs have no
@@ -908,7 +936,8 @@ says so, since refinement and map interpretation should allow for it.
## Model validation
-Section 5 appears only with `--model`. It reports the supplied model against the merged data —
+Section 5 appears only where a model was given (`--model`, `--mr`, or a model file among the
+inputs). It reports the supplied model against the merged data —
R-factors, maps, anomalous sites — and, separately, whether the data accepted the model at all.
The two are not the same question, and the report keeps them apart. **The R-factors, the maps and the
@@ -983,7 +1012,8 @@ between those two runs cannot be read; `R_MODEL_SHELL_SCALED` can.
Both R values are over **all** the reflections, not the free 5%. Nothing here is refined — the
coordinates, the B-factors and the occupancies are the model's own, and only the scale and the
-placement are fitted — so work and free estimate the same quantity (they differ by a median 0.001
+placement are fitted (the refinement that follows is reported apart, under
+[`REFINE_*`](#refinement-refine_)) — so work and free estimate the same quantity (they differ by a median 0.001
over our corpus) and the split buys no cross-validation while costing a factor of √20 in precision.
None of this touches the maps: the map coefficients and `R_WORK` / `R_FREE` are