Jungfraujoch/docs/RUGNUX_BENCHMARKING.md

# Measuring Rugnux: validation and timing

This page is for anyone who has to say how well, or how fast, `rugnux` works on a machine: a
colleague with a new GPU server, a laptop, a Mac, an ARM box, or an agent running there on someone's
behalf. It explains the two measurements the project uses, how to run them, how to read what they
print, and how to take timings that can be compared with someone else's. `tools/battery/README.md`
in the source tree is the detailed guide to the runner itself: every option, the manifests, the
scoring rules and the `results.json` schema. This page summarises it and does not replace it.

```{contents} On this page
:local:
:depth: 2
```

## Two measurements, one runner

Everything goes through one script, `tools/battery/battery.py`. Do not write your own loop over
datasets: numbers made by ad-hoc scripts have never been comparable with each other, which is why
the old ones were retired.

- **The validation battery** (`battery.py run`) processes a fixed list of rotation datasets and
  scores each answer against an independent reference. It answers *is the result right, and as
  good as the reference*. A full run takes hours.
- **The timing benchmark** (`battery.py timing`) processes four sets, each started from a model
  that has to be placed by molecular replacement, so one short run exercises the whole pipeline
  and also checks that the answers are right. It answers *how fast, on this machine*. It takes
  minutes, and needs nothing on the machine but four open datasets.

The two are kept apart on purpose. A battery run reads hundreds of gigabytes once, so its wall clock
mostly measures the disk and the page cache (see [the page-cache trap](#the-page-cache)). Use the
battery to judge correctness and the timing benchmark, run under controlled conditions, to judge
speed.

### Setting up a machine

The runner needs Python 3.9 or newer with `gemmi` and `numpy` (`pip install gemmi numpy`), a
`rugnux` binary, the datasets, and a **site config**: a small JSON file naming this machine's data
roots, where runs go, the GPU lock and the persisted baselines. `tools/battery/site.example.json` is
the template. The runner looks for the config in `--site PATH`, then in `$JFJOCH_BATTERY_SITE`,
then in `tools/battery/site.json`, which git ignores. `--site` is an option of the script itself,
so it goes **before** the subcommand:

```
python3 tools/battery/battery.py --site /path/to/site.json run --rugnux build/rugnux/rugnux --tier smoke
```

A relative path in the site config (data roots, runs root, lock file, caches) is taken relative to
the site config's own directory, so a config shipped beside its data works wherever it is unpacked.
A `pdb_cache` copied from another machine can be used as it is: the coordinates are looked up where
the cache now is, not where its `<entry>.json` files say they were first fetched.

The data layout (`<data root>/<set>/...`) and where to get each arm's data are in the README's
*Getting the data*. The open arm is public (each dataset's DOI is in
[External test data](EXTERNAL_TEST_DATA.md)); the in-house data come from the maintainers. The
timing benchmark needs only four open dataset directories and the smallest site config there is
([below](#what-a-machine-needs)); models, references and expected answers are all in the repository.

## The validation battery

### Arms and references

The battery has three **arms**, each scored against a reference that does not come from `rugnux`:

| arm | datasets | scored against |
|---|---|---|
| open | public PDB depositions of raw diffraction data, plus a few published small-molecule sets | the deposition: its space group, cell and resolution |
| inhouse | standard test crystals (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), small-molecule standards, and no-crystal controls | XDS, from the `CORRECT.LP` stored beside each dataset |
| private | user data | XDS, as in-house |

Each set runs **once**, with one command per arm. The open arm runs
`rugnux -o p --no-export-unmerged --model <deposited coordinates> <input>`. The XDS arms run
`rugnux -o p --no-export-unmerged [-A] --report-resolution <XDS d_min>,<XDS d_max> [--model <reference model>] <input>`,
with `-A` where XDS merged with Friedel's law off. Neither `--model` nor `--report-resolution`
changes the processing: the merged reflections, the resolution cut and every decision are the same
with or without them (this has been checked bit for bit). So one run gives both what a user gets
and the extra report sections that compare it with the reference.

### Verdicts

Every set gets one verdict. Scoring asks four questions in order, and the first that fails decides:
did it run, is the lattice right, is the symmetry right, and, on the XDS arms, is the merge as good
as XDS's?

- **pass**: the lattice matches the reference (Niggli-reduced primitive cells: volume ratio 0.95 to
  1.05, reduced edges within 2%) and so does the symmetry. On the open arm the space group is
  compared; on the XDS arms only the point group, because XDS never tests a screw axis. A group
  that the intensities cannot tell apart from the reference (same Laue class, same systematic
  absences) passes as `indistinguishable_alternative`. A row whose manifest lists a second
  defensible answer can pass as `accepted_alternative`. The report counts both kinds separately.
- **fail**, with a `cause`: wrong lattice (`lattice_halved`, `lattice_doubled`, ...), wrong
  symmetry (`sym_under`, `sym_over`, `sym_screw`, ...), a reader or indexing failure, a timeout.
- **crash**: rugnux was killed by a signal, its log shows `Processing failed`, a GPU fault, an
  uncaught exception, or a non-zero sanitizer summary. A crash fails the set whatever its report
  says.
- **unscored**: a question the data cannot answer. Either there is no reference, or the hand of an
  enantiomorphic pair is in question, or a screw axis lies on a row the sweep never measured (and
  rugnux said so). This is not a pass, but it is not counted as a failure either.
- **not_run**: the input is missing.

A no-crystal control passes only when rugnux reports no lattice. **Data quality is reported, not
scored**: resolution, CC1/2, R_meas, ISa, completeness and the model R-factors are shown next to
the reference's values for a human to judge.

### The like-for-like table

On the XDS arms, rugnux keeps its own resolution cut and the reference does not change it.
`--report-resolution` adds a second statistics table (the `REFRES_*` keys of the report) made from
the same merge but binned over XDS's own range. That gives two separate answers. The own numbers say
what a user gets. The reference-range numbers say whether rugnux's merge is better than XDS's over
the same reflections. Where rugnux's cut is coarser than XDS's, the shells past it were never
merged, so the completeness there measures **coverage**, not a drop in quality. The report marks
those rows.

On the open arm the deposition gives only a space group, a cell and a resolution, so the data
quality is judged from outside: R-free of the deposited model, from `rugnux --model` (a trend number,
see below) and from the REFMAC check, and a per-shell correlation of the merge with the model beside
the depositor's own data (always on, needs no CCP4).

### Running it

```
B="python3 tools/battery/battery.py --site /path/to/site.json"
R=build/rugnux/rugnux

$B run --rugnux $R --only 5reo,lyso_ref       # named sets
$B run --rugnux $R --tier smoke               # readers, symmetry classes, known hard cases
$B run --rugnux $R --tier ci                  # the routine regression check CI runs on every push
$B run --rugnux $R                            # the full battery: open + inhouse
$B run --rugnux $R --gpulock --model-check    # queued behind the site's GPU lock, with REFMAC
```

`--threads N` passes `-N` to rugnux, `--extra "<args>"` adds rugnux arguments to every set,
`--label` names the run, and `--baseline RUN|none` picks the run the report is compared against.
`--gpulock` puts every rugnux behind the command the site config names as `gpulock` (at PSI, a
`flock` wrapper shared by every GPU job on the machine). A run made that way is marked
"timing NOT a reference", and so is a run that found another process on the GPU before a set.

`run` exits with status 1 when any set crashed or a ci-tier set failed. Other failures and every
metric are reported, not judged.

### The run directory

Each run gets a new directory `<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>/`, which is never
reused and is made read-only when the run finishes. It holds:

- `bin/rugnux`: a copy of the exact binary that ran;
- `manifest.json`: the options, the binary's version and sha256, the build flags (read from the
  `CMakeCache.txt` above the binary, or from `--build-dir`), the host and its hardware, start and
  end;
- `results.json`: one row per set, with the verdict, the cause, every number and the exact command;
- `report.md` and `report.html`: the report;
- `work/<arm>/<set>/`: what rugnux wrote, `p_report.txt`, `p_log.txt`, `p.mtz` and the rest, plus
  `model_check/` when the REFMAC check ran.

The report opens with the build, host and hardware and whether timing counts. Then come any crashes,
a summary per arm (pass, fail, crash, unscored, not run, median resolution gain, ISa, R_meas, time),
tables per population, the like-for-like table and, if there is a baseline, every set that moved
beyond noise.

### Comparing runs

```
$B list                                   # runs, their state, the baselines
$B compare RUN_A RUN_B [--all]            # every set whose verdict or numbers moved beyond noise
$B compare RUN_A RUN_B --rerun-changed    # rerun the changed sets with A's binary
```

Verdicts are scored again whenever a run is read, with today's rules and today's references, so the
two sides of a comparison are always scored the same way. The noise thresholds are a first guess.
`--rerun-changed` measures the noise directly: a set that moves again under the same binary is
noise, and one that reproduces is a real change.

### Rules

- **Never start a full battery (no `--only`, no `--tier`) on a shared machine without the owner's
  go-ahead.** It takes hours and everything else on the machine competes with it. A full run takes
  the site lock, so two cannot overlap.
- **Judge a change against a persisted baseline run, never against a remembered number.** The site
  config's `baseline` names a complete run that the owner has blessed, and every new run is
  reported against it. Totals remembered from earlier runs have been off by a factor of three on
  the same sets.
- **Keep the manifests unchanged between the two runs you compare.** A manifest change is committed
  on its own, and the baseline is run again with it.
- **Pool fixes.** Run the full battery once on a branch that carries several fixes. For a single
  fix, run the sets it touches with `--only`, plus the smoke tier.
- **The private arm never leaves the machine.** Its names, paths, cells and statistics go into no
  commit, issue, public report or pack sent elsewhere. A private run gets its own directory and
  report (`report_PRIVATE.*`), and the runner refuses to write it into the repository.

## The REFMAC model check

R-free from `rugnux --model` is a **trend number**. It is a rigid-body placement scored on rugnux's
own free set, with a deliberately simple scale model, so it shows how a set moves between runs, but
it cannot be compared with the deposition's R-free. `--model-check` adds an independent number that
can be. It runs `tools/battery/model_check.py` on every open-arm set with a deposited X-ray model,
after rugnux has finished, so it cannot influence the processing. It needs CCP4: the runner sources
`$CCP4_SETUP`, by default `/opt/xtal/ccp4-9/bin/ccp4.setup-sh`, and calls the `refmac5` found there
(at PSI, CCP4 9.0 with REFMAC 5.8.0431).

### What it does

For each set the check takes `p.mtz` (its F, SIGF and FreeR_flag columns) and the deposited entry
(the mmCIF and, when deposited, the structure factors, downloaded from the RCSB into the site's
`pdb_cache`). Then it:

1. **Reads the published R-work, R-free and d_min** from the entry's `_refine` record, and skips
   entries that are not X-ray, have no deposited R-free, hold several models, or have more than
   200 000 atoms. Strict-NCS copies the entry only describes by operators are generated.
2. **Finds the model's setting.** Every change of basis that maps the data's lattice onto the
   model's is tried; their Niggli cells are compared loosely, 5% on edges and 3° on angles. That
   covers an alternative setting, a centred-versus-primitive description and the merohedral indexing
   ambiguity. Each candidate is scored by the R between the reindexed amplitudes and the model's
   |Fcalc| (gemmi, after an overall scale and B) on a mid-resolution shell, and the best one wins.
3. **Re-expresses the data in the deposited space group.** The merge is expanded to P1 with
   rugnux's own group, reindexed, reduced to the asymmetric unit of the deposited group
   (equivalents averaged with 1/σ² weights) and given the **deposited cell**. A space group rugnux
   called too high just splits equivalents apart here, and one it called too low averages them.
4. **Has REFMAC compute R-work and R-free of the model as deposited**, at
   `d_min_used = max(rugnux's d_min, deposited d_min)`, the lower of the two resolutions. The run is
   REFMAC's rigid-body mode with one cycle, and the R-factors are those REFMAC prints for that cycle,
   before it applies any shift. REFMAC's default bulk solvent and overall anisotropic scale are
   fitted, and nothing else. Rigid-body mode reads no restraint dictionary, so ligands, waters and
   hydrogens stay in. Only atoms of unknown element (UNX), which stop REFMAC and scatter nothing,
   are removed. Nothing is refined, by design. The data were moved into the model's frame, so a
   rigid body has nothing to correct, and at low resolution REFMAC's rigid body can walk a correct
   model away.
5. **Scores the depositor's own structure factors the same way**, as the baseline. The depositor's
   data go through the same choice of setting, because they can sit on the other branch of an
   indexing ambiguity. Amplitudes are `F_meas`, else the mean of F(+) and F(−), else intensities
   converted by French-Wilson (ctruncate). Their free set comes from `_refln.status` or
   `pdbx_r_free_flag`. The depositor's free flags are also carried over onto rugnux's reflections.
6. **Uses twin refinement where the entry declares twinning** (`_pdbx_reflns_twin`), for both data
   sets, because the published R-free is then a twinned one. The untwinned numbers are kept as well.
7. **Second protocol, refined on a shared free set.** The same deposited model is refined against
   each data set at that set's own resolution: 10 cycles of restrained refinement, automatic weight,
   isotropic B for every atom, twin refinement where declared. A ligand that the monomer library
   describes with different atoms is renamed so that REFMAC builds its restraints from the model.
   The free set is the depositor's free reflections that are present in **both** data sets.
   Depositor free reflections outside that intersection are dropped from both, so neither refinement
   uses a free reflection as a work reflection.

### The numbers

| field in `results.json` | data | free set | read it as |
|---|---|---|---|
| `refmac_rfree` | rugnux | rugnux's | close to a work R: most of rugnux's free reflections were work reflections in the depositor's refinement |
| `refmac_rfree_depflags` | rugnux | the depositor's | **the fair first-cycle number for rugnux's data** |
| `refmac_rfree_depdata` | the depositor's | the depositor's | the baseline: same model, same protocol, the data the model was refined against |
| `refmac_rfree_ratio` | | | `refmac_rfree_depflags / refmac_rfree_depdata`; below 1 means rugnux's data fit the model better |
| `refmac_refined_rfree_shared` | rugnux | shared | **the refined number for rugnux's data** |
| `refmac_refined_rfree_shared_depdata` | the depositor's | shared | the same protocol on the depositor's data |
| `refmac_refined_ratio` | | | their ratio; below 1 means rugnux's data give the better refined model |

`refmac_status` and `refmac_reason` say why a set has no numbers: not X-ray, too large, no
deposited free set, a download that failed. The same applies to `refmac_refined_reason` for the
second protocol. `python3 tools/battery/model_check.py <p.mtz> <pdb id> --workdir DIR` runs the
check on one merge and prints every field, including the reindexing operators chosen for both data
sets. `battery.py recheck RUN` reruns it on a finished run's merges without running rugnux again.

### Limits

- **The comparison favours the deposition.** The model was refined against the depositor's data, so
  those data start with an advantage. A ratio near 1 is a good result for rugnux.
- **`refmac_rfree_depdata` is not the published R-free.** It comes from a different program (REFMAC,
  where the deposition often used PHENIX, whose resolution-binned scaling absorbs a falloff that
  REFMAC's overall scale cannot) and from no refinement at all.
- **The data are given the deposited cell.** A real cell difference between the two crystals is not
  modelled, and costs R. The check records it as `cell_diff_pct`.
- **The first-cycle numbers use the common resolution limit**, so they say nothing about the shells
  only one data set reaches. The refined protocol uses each set's own range, but its free set is
  limited to what both data sets measured.
- **The setting is chosen by agreement with the model.** That is right for this purpose, but it
  means the check cannot referee rugnux's own indexing choice where the model is ambiguous.

### Cost

The check is opt-in because it is the expensive part. In a battery run on 2026-10-16, an open-arm
set took about 2.5 minutes with the check, against well under a minute of rugnux. The
refined protocol can take up to an hour on a large entry. Across a full battery, REFMAC accounts for
most of the time, so add `--model-check` only when the R-free figures are the point, for example in
a release battery or for a paper.

## The timing benchmark

```
$B timing --rugnux $R [--benchmark standard|inhouse] [--threads N] [--label L] [--baseline RUN|none] [--extra=<rugnux args>] [--gpulock]
```

The benchmark runs a few sets. Each starts from a model that the run has to place by molecular
replacement, so a single run times every stage: reading, pre-scan, indexing, integration, scaling,
the space-group search, MR and model validation. It also checks that every answer is right.

`--benchmark standard`, the default, is the one to run on a new machine. It uses open data only:
three EIGER 16M rotation sets of ordinary size and quality, and a virus capsid as the large case.

| set | start model (`--model`) | what makes it hard |
|---|---|---|
| `9zlo` | the AlphaFold DB prediction of UreE (*Proteus mirabilis*), committed in `tools/battery/models/` | two copies to place, P2₁2₁2₁, 2.0 Å |
| `9s02` | the AlphaFold DB prediction of human PYCR1, also in `models/` | five copies, P2₁2₁2, 1.65 Å |
| `9gjx` | the AlphaFold DB prediction of a *Bacillus licheniformis* nitroreductase, also in `models/` | six copies, P2₁, 2.4 Å, the largest of the three datasets (37 GB) |
| `4nwv` | the deposited 4NWV (a virus capsid, I222, 45 chains once strict NCS is expanded), turned 30° and shifted 33 Å by a fixed transform | the largest: too far off for the rigid body, so MR places the whole asymmetric unit |

`--benchmark inhouse` is the benchmark as first set up, kept so that earlier runs stay comparable:
the predictions of horse myoglobin (`myob_x10sa`, a prediction without heme) and bovine cytochrome c
(`cytc_x10sa`: three copies related by a pseudo-translation; the data alone cannot tell P3₁21 from
P3₂21, so the model must decide the hand), and the same `4nwv`. Its two in-house datasets come from
the maintainers.

The predictions are committed next to their predicted-aligned-error files, which rugnux uses to cut
them into rigid domains. The deposited entries the placements are checked against are committed too
(`tools/battery/models/pdb/`), and the moved 4NWV is made at run time from the committed 4NWV with
the transform in `tools/battery/timing.json`. A timing run needs no `pdb_cache` and no network.

### What a machine needs

Copy these directories under one open data root (sizes as unpacked; please cite the DOIs):

| directory | size | source |
|---|---:|---|
| `9zlo/` | 11 GB | Zenodo [10.5281/zenodo.18652652](https://doi.org/10.5281/zenodo.18652652) |
| `9s02/` | 11 GB | MX Raw Data Repository [10.60884/NRNGS4](https://doi.org/10.60884/NRNGS4) |
| `9gjx/` | 37 GB | IRRMC [10.18430/M39GJX](https://doi.org/10.18430/M39GJX) |
| `4nwv/` | 16 GB | SBGrid Data Bank [10.15785/sbgrid/9](https://doi.org/10.15785/sbgrid/9) |

The paths inside each directory are the `input`s of `tools/battery/open.json` (for example
`9gjx/BlNTR_9gjx/data/SC6_1_master.h5`), and [External test data](EXTERNAL_TEST_DATA.md) says what
each archive holds. The site config needs only the runs root and that data root
(`tools/battery/site.timing.example.json`):

```json
{
 "runs_root": "/path/to/runs",
 "arms": {
  "open": {"data_root": "/path/to/open"}
 }
}
```

and the run is

```
python3 tools/battery/battery.py --site /path/to/site.json timing --rugnux /path/to/rugnux
```

Python 3.9 or newer with `gemmi` and `numpy` is all the runner needs.

### The checks

A set passes only when every check holds. The thresholds are in `timing.json`.

| check | passes when | why |
|---|---|---|
| `space_group` | the battery's own verdict on lattice and space group passes (the hand excepted) | the data's own determination is right |
| `placed` | `MODEL_PLACEMENT` is `MR` | molecular replacement placed the model, and the R-factors, maps and model written are of that placement |
| `mr_group` | `MR_SPACE_GROUP` is the deposited group, hand included (P 21 21 21, P 21 21 2, P 1 21 1 and I 2 2 2 for `standard`; P 1 21 1 and P 32 2 1 for the two in-house sets) | the model settles the hand, so this is where a wrong enantiomorph shows |
| `r_free` | `R_FREE` ≤ 0.50 for the predictions, ≤ 0.35 for 4NWV | a random structure gives 0.586 |
| `confirmed` | `CC_MODEL_CONFIRMED_TO_D_MIN` is not `NONE` | some shell's CC(model, data) clears rugnux's empirical null |
| `placement_cc` | the placed `p_model.pdb` and the reference model have correlated intensities, ≥ 0.30 | see below |

`placement_cc` is the one check that does **not** depend on rugnux's own scores. It computes |F|²
from the placed model and from the battery's reference model for the set, each in its own cell and
space group. It normalises each resolution shell to its mean, from 10 to 4 Å, and takes the best
correlation over the changes of basis the two lattices allow. |F| is the same for every allowed
origin and every symmetry copy, so no superposition is needed. A right placement correlates
strongly (measured 0.55 to 0.96). A wrong placement gives 0.01 to 0.04, and the wrong hand gives
about 0.2. This catches a placement that rugnux reports as right but is not.

### Reading the output

The console prints one block per set: the verdict, rugnux's wall time and the MR time, one line per
check (`ok` or `BAD`, the value, the limit), and the step times. For example:

```
4nwv         pass  wall   167.2 s  MR 52.0 s
    ok  space_group   I 2 2 2 (want I 2 2 2)
    ok  placed        MR (want MR)
    ...
    Reading input 0.2, Pre-scan 4.6, Indexing 5.5, Integration (geometry pre-pass) 11.8, ...
```

The run directory is an ordinary one, so `list`, `table`, `compare` and `keep` all work on it. Its
`report.md` opens with a **Timing benchmark** section: the checks per set, then a table with one row
per step and one column per set. Each `results.json` row carries `bench_verdict`, `bench_reason`,
`bench_checks`, `mr_seconds` and `steps`, the seconds per step. `wall_s` is rugnux's own
`WALL_TIME`, so time spent waiting for a GPU queue is not in it. The run exits with status 1 when a
set fails a check or crashes. A wrong answer is a failure even when the time is good.

If the site config names a `baseline_timing` for the benchmark (`{"standard": RUN, "inhouse": RUN}`,
earlier timing runs on this machine), the report is compared against it. Otherwise pass
`--baseline RUN`. Compare timing runs only from the same machine and the same benchmark.

### CPU only

```
$B timing --rugnux $R --extra=--no-gpu
```

This runs a CUDA build as if the machine had no GPU: CUDA is never touched and every step takes its
CPU path. Write the option with `=`, because `--extra --no-gpu` makes Python's argument parser take
`--no-gpu` for an option of its own. Several options go in quotes: `--extra="--no-gpu -N 8"`. A CPU
run and a GPU run agree to rounding, not bit for bit, so their last digits can differ.

## Taking timings that mean something

A wall-clock number is only useful if someone else can reproduce it and if two numbers can be
compared. Each lesson below was learned by getting a timing wrong first.

### Build like CI

The build system sets no `-march` and no LTO, so that each site can choose. CI and the released
binaries are built with `-march=x86-64-v3 -flto=auto` on Linux x86 and `/arch:AVX` with MSVC on
Windows. A plain `cmake -DCMAKE_BUILD_TYPE=Release` build is a baseline binary that nobody ships. GPU
work runs at the same speed in it, but the CPU-bound phases (first-pass indexing, scaling and
merging) are **about 26% slower**. A profile of that build also points at the wrong code: Eigen has
no vectorised `round` below SSE4.1. For any CPU-side measurement, build with at least
`-DCMAKE_CXX_FLAGS="-march=x86-64-v3" -DCMAKE_C_FLAGS="-march=x86-64-v3"`. On ARM (a GB10 or a
Grace) and on Apple Silicon, leave `-march` out: the default target already has NEON and a
vectorised round. The runner records the flags in `manifest.json`, so always state them.

### Do not time the first run after a build

The first runs straight after a build are not representative. On a Windows laptop on 2026-10-14,
the runs started immediately after a 500-target build were 1.3 to 2 times slower in every phase:
GPU image loop, file reads, everything. Rerun back to back, the same binaries matched to within a
few percent. Let the machine settle (the build's own I/O, antivirus scans of new binaries, a hot
CPU on a laptop), then make a throw-away run first.

### A quiet machine

On a shared machine, another job's load makes wall-clock numbers and parallel-efficiency numbers
meaningless: they measure the other job. Profiles that report **shares of cycles** (`perf`, Nsight)
still show where the time goes, but not how long it took. If the numbers are the point, take the
machine to yourself. Check `uptime` and `nvidia-smi` before and after, and say what else was running.
On a laptop, plug it in and keep it in the same power mode: on 2026-10-16 two back-to-back runs of
4nwv on the same binary on a Windows laptop took 167 and 206 s, with the growth spread over every
CPU- and GPU-heavy step. That is thermals and power, not code. Treat such a spread as the machine's
noise.

### Interleave: ABAB, not AB

One run of the old binary against one of the new says almost nothing. Run them **interleaved**, old,
new, old, new, on the same two or three sets, in one session on a quiet machine, and compare the
pairs. Slow drift (the page cache, thermals, another user) then affects both sides alike. Keep the
old binary for this; the run directory's `bin/rugnux` is one. Several minutes of ABAB have settled
questions that a whole battery's totals had answered wrongly, more than once.

### The page cache

The battery and the benchmark read every image from disk. Whether those files are still in the
operating system's page cache can change a set's time several-fold, with every result identical. On
the PSI workstation, whose data disk is a 7200 rpm HDD, one 10 GB set took 15 s warm and 111 s cold.
A whole battery once swung from 11 to 19 minutes only because a large job beforehand had pushed its
data out of the cache. The signs are:

- the slowdown scales with each set's size on disk, not with its content;
- it can be bimodal: the sets still cached are unchanged to the second, while the evicted ones are
  several times slower;
- divide a set's extra seconds by its size: if the result is the disk's read speed, it is the disk;
- rugnux reads the frames in each of its passes, and on a machine that can hold the set in RAM the
  later passes are always warm. A real regression slows them too, while a cold read slows only the
  first pass (compare the `Integration (geometry pre-pass)` step with `Integration (refined
  geometry)` in the log).

Either time warm runs (one throw-away run first, and enough RAM to hold the set) or time cold ones
deliberately (on Linux, drop the file from the cache with `posix_fadvise(POSIX_FADV_DONTNEED)` or
`echo 1 > /proc/sys/vm/drop_caches`). Either way, **say which**. A machine whose RAM is about the
size of a set cannot keep it warm, so on a 16 GB Mac a 16 GB set always reads cold.

### The GPU lock

On a machine where several people or agents share the GPUs, the site's `gpulock` serialises the
jobs. `battery.py --gpulock` takes the lock for each rugnux separately, so other jobs can run
between sets, and the run is marked "timing NOT a reference". For a timing session, take the lock
**once** around the whole session instead, so that nothing else gets in between your A and B runs:

```
flock /data/tmp/gpulock bash -c "
  $B timing --rugnux old/rugnux --label A1
  $B timing --rugnux new/rugnux --label B1
  $B timing --rugnux old/rugnux --label A2
  $B timing --rugnux new/rugnux --label B2"
```

(`/data/tmp/gpulock` is the lock file at PSI; use your site's.)

The runner still checks for other processes on the GPU before each set (`gpu_others` in
`results.json`). A machine without such a lock, a laptop for example, sets `"gpulock": null` and runs
without `--gpulock`.

### Report the hardware and the conditions

`manifest.json` records the CPU, cores and threads, RAM and GPUs on Linux. On Windows and macOS it
may say "not recorded", so write the hardware down yourself. A timing means nothing without: the
machine (CPU, RAM, GPU and its memory, driver), the OS, the binary (git hash, sha256, build flags,
compiler and CUDA version), `-N` if set, warm or cold data, what else was running, mains or battery
power, and which runs are the first after a build.

### Windows

- Run the runner as `python -X utf8 tools\battery\battery.py ...`, because the report and the
  console table are UTF-8. Give `--rugnux` the path of `rugnux.exe`. The run directory gets its own
  copy of the binary, so any DLL it needs must be found on `PATH`.
- **WDDM can quietly page GPU memory to system RAM.** A consumer or laptop card under Windows' WDDM
  driver model does not fail an allocation that exceeds the card. It spills to "shared GPU memory"
  in system RAM, and the kernels touching it slow down several-fold with no error. rugnux decides
  between GPU and CPU from the card's total memory (80% for scaling and the model fits), but memory
  held from earlier phases and the driver's own reserve count too. During a run, watch Task
  Manager's **Dedicated** and **Shared GPU memory** for the process: if the shared figure grows, the
  timing measures paging. On 2026-10-15 an 8 GB laptop card spent five times longer in model
  validation for this reason alone.
- Windows has no `flock` and no process groups. The timing benchmark does not need them.

### macOS

- The macOS build has no CUDA. Bragg integration, the model density and the molecular-replacement
  rotation function run on the Apple GPU through Metal. Everything else runs on the CPU, so a Mac
  timing is mostly a CPU-bound timing. Molecular replacement in particular takes much longer than on
  a CUDA machine, as its translation function and rigid body stay on the CPU.
- Memory is unified: CPU and GPU share it, and the OS compresses memory before it swaps. A 16 GB Mac
  can run the benchmark, but its biggest set comes close to the limit, so report both measures
  below.

### Memory peaks

The battery does not record memory, so measure it next to a timing when it matters. Each row's
`cmd` in `results.json` is the exact command a set ran, so you can rerun that command by hand under
a tool that measures:

- **Linux:** `/usr/bin/time -v <cmd>` reports *Maximum resident set size*, which is `VmHWM` in
  `/proc/<pid>/status` while the process runs. RSS is not the same as live data: glibc keeps freed
  memory in its arenas, and pinned host buffers for the GPU count too. On one large set about
  0.5 GB of the peak was memory freed but not yet returned to the system, and up to 2.4 GB on an
  older build.
- **macOS:** `/usr/bin/time -l <cmd>` reports both the maximum RSS and the *peak memory footprint*,
  which includes compressed memory and GPU buffers and can be much larger. On 2026-10-16, 4nwv
  showed 9.1 GiB of RSS against about 22 GiB of footprint.
- **Windows:** sample the process's peak working set (PowerShell
  `(Get-Process rugnux).PeakWorkingSet64`) while it runs, and Task Manager's dedicated and shared
  GPU memory.
- **GPU:** `nvidia-smi --query-gpu=memory.used --format=csv -lms 500` while the set runs. rugnux's
  log also states each GPU-or-CPU decision with its numbers, for example
  `Model validation: ... 4.42 GB held of a 6.84 GB budget, 80% of the card's 8.55 GB` and
  `... scaling on the CPU instead`. Quote those lines; they explain most surprises between machines.

[Installing Rugnux ▸ Memory](RUGNUX_INSTALL.md#memory) explains what sets the peak.

### Per-step times: the TIMING lines

rugnux writes its full log to `<prefix>_log.txt` (`p_log.txt` in the battery) at debug level, and
every timed section adds a line when it ends:

```
[2026-10-16 07:42:45.430] [rugnux] [debug] TIMING step Integration (geometry pre-pass), 500 images 11.800
[2026-10-16 07:42:52.230] [Rugnux] [debug] TIMING merge P1 4.697
```

The `TIMING step ...` lines are the steps shown on the console, and they add up to the run's wall
time. These are what the benchmark tabulates. The other `TIMING ...` lines are nested sections
inside a step (`rsm ...` sections inside `merge ...`, everything in a pass before its
`pass (...)` line). `tools/rugnux_timing.py p_log.txt` tabulates them, and `--all` lists every
section. For a rotation run the steps are, in order:

| step | covers |
|---|---|
| Reading input | opening the master file, geometry and mask |
| Pre-scan | a projection of a sample of frames: beam stop, background beam centre, spot widths, defective pixels |
| Indexing | first-pass spot finding and lattice search on the first part of the sweep |
| Integration (geometry pre-pass), N images | integrating the first part of the sweep at the file's geometry |
| Scaling and post-refinement | scaling that pre-pass and post-refining the detector geometry, cell and axis from it |
| Metric-symmetry check, Geometry check | only where needed: re-integration of a promoted lattice class, more geometry rounds |
| Indexing at the refined geometry | re-indexing with the refined geometry |
| Integration (refined geometry), N images | the whole sweep, every frame |
| Scaling and merging | combining partials, scaling, the space-group search, the resolution cut, merging |
| Analyses | twinning, anisotropy and the other report-only analyses |
| Model validation | `--model` only: structure factors, scale fits, rigid-body placement, maps |
| Molecular replacement | only where the rigid body did not place the model (always, in the timing benchmark) |
| Writing files | MTZ, mmCIF, HKL, report |

A step that is slower on one machine than on another points at the code, and a step that is slower
everywhere at once points at the conditions. Send the whole step table, not only the total.

### NVTX ranges for Nsight Systems

A CUDA build compiled with the NVTX headers (found next to `nvcc`; otherwise the configure says
`NVTX3 headers not found - NVTX ranges compiled out`) opens an NVTX range for every console step
(`Step <n> <title>`) and for every `TIMING` section, with the same names. An Nsight Systems trace
therefore shows the pipeline by name above the CUDA kernels and copies:

```
nsys profile --trace=cuda,nvtx,osrt -o 4nwv_trace <the set's cmd from results.json>
```

Without a profiler attached, the ranges cost nothing.

## What to send back

A remote measurement is only useful if the owner can compare it with the others. Send one archive
(a "pack") per session, named after the machine, the binary and the date, for example
`rugnux_<machine>_<git>_<YYYYMMDD>.tar.gz`, laid out like this:

```
BRIEF.md
runs/<n>_<what>/report.md          # unedited, as the runner wrote them
runs/<n>_<what>/report.html
runs/<n>_<what>/results.json
runs/<n>_<what>/manifest.json
runs/<n>_<what>/SOURCE_RUN.txt     # the run directory's original path
runs/<n>_<what>/logs/<set>/{p_log.txt,p_report.txt,run.log}
timing/                            # step tables, ABAB comparisons
memory/                            # peak RSS / footprint / GPU memory samples, with how they were taken
build/                             # the exact configure command (or CMakeCache.txt flags)
```

`BRIEF.md` comes first and should be short: the host and hardware, the OS, the binary (git, sha256,
build flags, compiler, CUDA), the data (which pack, warm or cold), the conditions (quiet or shared,
power, first run after a build), and then the results. Give a table per run, the step table of the
slowest set, what changed against the previous pack from the same machine, and what is not
explained yet together with the experiment that would explain it. Number the runs in the order they
were made, and say which ones are not representative and why (`1_` straight after the build, for
example).

Leave out the maps, MTZ files and binaries unless they are asked for, since they are large and the
logs and reports carry the numbers. Never edit a `report.md` or `results.json`: comment in the
brief instead. If you had to patch the runner to make it work, include the patch and say so. A pack
never contains anything from the private arm.