Jungfraujoch/docs/RUGNUX_BENCHMARKING.md

Measuring Rugnux: validation and timing

This page is for anyone who has to say how well, or how fast, rugnux works on a machine: a colleague with a new GPU server, a laptop, a Mac, an ARM box, or an agent running there on someone's behalf. It explains the two measurements the project uses, how to run them, how to read what they print, and how to take timings that can be compared with someone else's. tools/battery/README.md in the source tree is the detailed guide to the runner itself: every option, the manifests, the scoring rules and the results.json schema. This page summarises it and does not replace it.

:local:
:depth: 2

Two measurements, one runner

Everything goes through one script, tools/battery/battery.py. Do not write your own loop over datasets: numbers made by ad-hoc scripts have never been comparable with each other, which is why the old ones were retired.

• The validation battery (battery.py run) processes a fixed list of rotation datasets and scores each answer against an independent reference. It answers is the result right, and as good as the reference. A full run takes hours.
• The timing benchmark (battery.py timing) processes four sets, each started from a model that has to be placed by molecular replacement, so one short run exercises the whole pipeline and also checks that the answers are right. It answers how fast, on this machine. It takes minutes, and needs nothing on the machine but four open datasets.

The two are kept apart on purpose. A battery run reads hundreds of gigabytes once, so its wall clock mostly measures the disk and the page cache (see the page-cache trap). Use the battery to judge correctness and the timing benchmark, run under controlled conditions, to judge speed.

Setting up a machine

The runner needs Python 3.9 or newer with gemmi and numpy (pip install gemmi numpy), a rugnux binary, the datasets, and a site config: a small JSON file naming this machine's data roots, where runs go, the GPU lock and the persisted baselines. tools/battery/site.example.json is the template. The runner looks for the config in --site PATH, then in $JFJOCH_BATTERY_SITE, then in tools/battery/site.json, which git ignores. --site is an option of the script itself, so it goes before the subcommand:

python3 tools/battery/battery.py --site /path/to/site.json run --rugnux build/rugnux/rugnux --tier smoke

A relative path in the site config (data roots, runs root, lock file, caches) is taken relative to the site config's own directory, so a config shipped beside its data works wherever it is unpacked. A pdb_cache copied from another machine can be used as it is: the coordinates are looked up where the cache now is, not where its <entry>.json files say they were first fetched.

The data layout (<data root>/<set>/...) and where to get each arm's data are in the README's Getting the data. The open arm is public (each dataset's DOI is in External test data); the in-house data come from the maintainers. The timing benchmark needs only four open dataset directories and the smallest site config there is (below); models, references and expected answers are all in the repository.

The validation battery

Arms and references

The battery has three arms, each scored against a reference that does not come from rugnux:

arm	datasets	scored against
open	public PDB depositions of raw diffraction data, plus a few published small-molecule sets	the deposition: its space group, cell and resolution
inhouse	standard test crystals (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), small-molecule standards, and no-crystal controls	XDS, from the CORRECT.LP stored beside each dataset
private	user data	XDS, as in-house

Each set runs once, with one command per arm. The open arm runs rugnux -o p --no-export-unmerged --model <deposited coordinates> <input>. The XDS arms run rugnux -o p --no-export-unmerged [-A] --report-resolution <XDS d_min>,<XDS d_max> [--model <reference model>] <input>, with -A where XDS merged with Friedel's law off. Neither --model nor --report-resolution changes the processing: the merged reflections, the resolution cut and every decision are the same with or without them (this has been checked bit for bit). So one run gives both what a user gets and the extra report sections that compare it with the reference.

Verdicts

Every set gets one verdict. Scoring asks four questions in order, and the first that fails decides: did it run, is the lattice right, is the symmetry right, and, on the XDS arms, is the merge as good as XDS's?

• pass: the lattice matches the reference (Niggli-reduced primitive cells: volume ratio 0.95 to 1.05, reduced edges within 2%) and so does the symmetry. On the open arm the space group is compared; on the XDS arms only the point group, because XDS never tests a screw axis. A group that the intensities cannot tell apart from the reference (same Laue class, same systematic absences) passes as indistinguishable_alternative. A row whose manifest lists a second defensible answer can pass as accepted_alternative. The report counts both kinds separately.
• fail, with a cause: wrong lattice (lattice_halved, lattice_doubled, ...), wrong symmetry (sym_under, sym_over, sym_screw, ...), a reader or indexing failure, a timeout.
• crash: rugnux was killed by a signal, its log shows Processing failed, a GPU fault, an uncaught exception, or a non-zero sanitizer summary. A crash fails the set whatever its report says.
• unscored: a question the data cannot answer. Either there is no reference, or the hand of an enantiomorphic pair is in question, or a screw axis lies on a row the sweep never measured (and rugnux said so). This is not a pass, but it is not counted as a failure either.
• not_run: the input is missing.

A no-crystal control passes only when rugnux reports no lattice. Data quality is reported, not scored: resolution, CC1/2, R_meas, ISa, completeness and the model R-factors are shown next to the reference's values for a human to judge.

The like-for-like table

On the XDS arms, rugnux keeps its own resolution cut and the reference does not change it. --report-resolution adds a second statistics table (the REFRES_* keys of the report) made from the same merge but binned over XDS's own range. That gives two separate answers. The own numbers say what a user gets. The reference-range numbers say whether rugnux's merge is better than XDS's over the same reflections. Where rugnux's cut is coarser than XDS's, the shells past it were never merged, so the completeness there measures coverage, not a drop in quality. The report marks those rows.

On the open arm the deposition gives only a space group, a cell and a resolution, so the data quality is judged from outside: R-free of the deposited model, from rugnux --model (a trend number, see below) and from the REFMAC check, and a per-shell correlation of the merge with the model beside the depositor's own data (always on, needs no CCP4).

Running it

B="python3 tools/battery/battery.py --site /path/to/site.json"
R=build/rugnux/rugnux

$B run --rugnux $R --only 5reo,lyso_ref       # named sets
$B run --rugnux $R --tier smoke               # readers, symmetry classes, known hard cases
$B run --rugnux $R --tier ci                  # the routine regression check CI runs on every push
$B run --rugnux $R                            # the full battery: open + inhouse
$B run --rugnux $R --gpulock --model-check    # queued behind the site's GPU lock, with REFMAC

--threads N passes -N to rugnux, --extra "<args>" adds rugnux arguments to every set, --label names the run, and --baseline RUN|none picks the run the report is compared against. --gpulock puts every rugnux behind the command the site config names as gpulock (at PSI, a flock wrapper shared by every GPU job on the machine). A run made that way is marked "timing NOT a reference", and so is a run that found another process on the GPU before a set.

run exits with status 1 when any set crashed or a ci-tier set failed. Other failures and every metric are reported, not judged.

The run directory

Each run gets a new directory <runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>/, which is never reused and is made read-only when the run finishes. It holds:

• bin/rugnux: a copy of the exact binary that ran;
• manifest.json: the options, the binary's version and sha256, the build flags (read from the CMakeCache.txt above the binary, or from --build-dir), the host and its hardware, start and end;
• results.json: one row per set, with the verdict, the cause, every number and the exact command;
• report.md and report.html: the report;
• work/<arm>/<set>/: what rugnux wrote, p_report.txt, p_log.txt, p.mtz and the rest, plus model_check/ when the REFMAC check ran.

The report opens with the build, host and hardware and whether timing counts. Then come any crashes, a summary per arm (pass, fail, crash, unscored, not run, median resolution gain, ISa, R_meas, time), tables per population, the like-for-like table and, if there is a baseline, every set that moved beyond noise.

Comparing runs

$B list                                   # runs, their state, the baselines
$B compare RUN_A RUN_B [--all]            # every set whose verdict or numbers moved beyond noise
$B compare RUN_A RUN_B --rerun-changed    # rerun the changed sets with A's binary

Verdicts are scored again whenever a run is read, with today's rules and today's references, so the two sides of a comparison are always scored the same way. The noise thresholds are a first guess. --rerun-changed measures the noise directly: a set that moves again under the same binary is noise, and one that reproduces is a real change.

Rules

• Never start a full battery (no --only, no --tier) on a shared machine without the owner's go-ahead. It takes hours and everything else on the machine competes with it. A full run takes the site lock, so two cannot overlap.
• Judge a change against a persisted baseline run, never against a remembered number. The site config's baseline names a complete run that the owner has blessed, and every new run is reported against it. Totals remembered from earlier runs have been off by a factor of three on the same sets.
• Keep the manifests unchanged between the two runs you compare. A manifest change is committed on its own, and the baseline is run again with it.
• Pool fixes. Run the full battery once on a branch that carries several fixes. For a single fix, run the sets it touches with --only, plus the smoke tier.
• The private arm never leaves the machine. Its names, paths, cells and statistics go into no commit, issue, public report or pack sent elsewhere. A private run gets its own directory and report (report_PRIVATE.*), and the runner refuses to write it into the repository.

The REFMAC model check

R-free from rugnux --model is a trend number. It is a rigid-body placement scored on rugnux's own free set, with a deliberately simple scale model, so it shows how a set moves between runs, but it cannot be compared with the deposition's R-free. --model-check adds an independent number that can be. It runs tools/battery/model_check.py on every open-arm set with a deposited X-ray model, after rugnux has finished, so it cannot influence the processing. It needs CCP4: the runner sources $CCP4_SETUP, by default /opt/xtal/ccp4-9/bin/ccp4.setup-sh, and calls the refmac5 found there (at PSI, CCP4 9.0 with REFMAC 5.8.0431).

What it does

For each set the check takes p.mtz (its F, SIGF and FreeR_flag columns) and the deposited entry (the mmCIF and, when deposited, the structure factors, downloaded from the RCSB into the site's pdb_cache). Then it:

1. Reads the published R-work, R-free and d_min from the entry's _refine record, and skips entries that are not X-ray, have no deposited R-free, hold several models, or have more than 200 000 atoms. Strict-NCS copies the entry only describes by operators are generated.
2. Finds the model's setting. Every change of basis that maps the data's lattice onto the model's is tried; their Niggli cells are compared loosely, 5% on edges and 3° on angles. That covers an alternative setting, a centred-versus-primitive description and the merohedral indexing ambiguity. Each candidate is scored by the R between the reindexed amplitudes and the model's |Fcalc| (gemmi, after an overall scale and B) on a mid-resolution shell, and the best one wins.
3. Re-expresses the data in the deposited space group. The merge is expanded to P1 with rugnux's own group, reindexed, reduced to the asymmetric unit of the deposited group (equivalents averaged with 1/σ² weights) and given the deposited cell. A space group rugnux called too high just splits equivalents apart here, and one it called too low averages them.
4. Has REFMAC compute R-work and R-free of the model as deposited, at d_min_used = max(rugnux's d_min, deposited d_min), the lower of the two resolutions. The run is REFMAC's rigid-body mode with one cycle, and the R-factors are those REFMAC prints for that cycle, before it applies any shift. REFMAC's default bulk solvent and overall anisotropic scale are fitted, and nothing else. Rigid-body mode reads no restraint dictionary, so ligands, waters and hydrogens stay in. Only atoms of unknown element (UNX), which stop REFMAC and scatter nothing, are removed. Nothing is refined, by design. The data were moved into the model's frame, so a rigid body has nothing to correct, and at low resolution REFMAC's rigid body can walk a correct model away.
5. Scores the depositor's own structure factors the same way, as the baseline. The depositor's data go through the same choice of setting, because they can sit on the other branch of an indexing ambiguity. Amplitudes are F_meas, else the mean of F(+) and F(−), else intensities converted by French-Wilson (ctruncate). Their free set comes from _refln.status or pdbx_r_free_flag. The depositor's free flags are also carried over onto rugnux's reflections.
6. Uses twin refinement where the entry declares twinning (_pdbx_reflns_twin), for both data sets, because the published R-free is then a twinned one. The untwinned numbers are kept as well.
7. Second protocol, refined on a shared free set. The same deposited model is refined against each data set at that set's own resolution: 10 cycles of restrained refinement, automatic weight, isotropic B for every atom, twin refinement where declared. A ligand that the monomer library describes with different atoms is renamed so that REFMAC builds its restraints from the model. The free set is the depositor's free reflections that are present in both data sets. Depositor free reflections outside that intersection are dropped from both, so neither refinement uses a free reflection as a work reflection.

The numbers

field in results.json	data	free set	read it as
refmac_rfree	rugnux	rugnux's	close to a work R: most of rugnux's free reflections were work reflections in the depositor's refinement
refmac_rfree_depflags	rugnux	the depositor's	the fair first-cycle number for rugnux's data
refmac_rfree_depdata	the depositor's	the depositor's	the baseline: same model, same protocol, the data the model was refined against
refmac_rfree_ratio			refmac_rfree_depflags / refmac_rfree_depdata; below 1 means rugnux's data fit the model better
refmac_refined_rfree_shared	rugnux	shared	the refined number for rugnux's data
refmac_refined_rfree_shared_depdata	the depositor's	shared	the same protocol on the depositor's data
refmac_refined_ratio			their ratio; below 1 means rugnux's data give the better refined model

refmac_status and refmac_reason say why a set has no numbers: not X-ray, too large, no deposited free set, a download that failed. The same applies to refmac_refined_reason for the second protocol. python3 tools/battery/model_check.py <p.mtz> <pdb id> --workdir DIR runs the check on one merge and prints every field, including the reindexing operators chosen for both data sets. battery.py recheck RUN reruns it on a finished run's merges without running rugnux again.

Limits

• The comparison favours the deposition. The model was refined against the depositor's data, so those data start with an advantage. A ratio near 1 is a good result for rugnux.
• refmac_rfree_depdata is not the published R-free. It comes from a different program (REFMAC, where the deposition often used PHENIX, whose resolution-binned scaling absorbs a falloff that REFMAC's overall scale cannot) and from no refinement at all.
• The data are given the deposited cell. A real cell difference between the two crystals is not modelled, and costs R. The check records it as cell_diff_pct.
• The first-cycle numbers use the common resolution limit, so they say nothing about the shells only one data set reaches. The refined protocol uses each set's own range, but its free set is limited to what both data sets measured.
• The setting is chosen by agreement with the model. That is right for this purpose, but it means the check cannot referee rugnux's own indexing choice where the model is ambiguous.

Cost

The check is opt-in because it is the expensive part. In a battery run on 2026-10-16, an open-arm set took about 2.5 minutes with the check, against well under a minute of rugnux. The refined protocol can take up to an hour on a large entry. Across a full battery, REFMAC accounts for most of the time, so add --model-check only when the R-free figures are the point, for example in a release battery or for a paper.

The timing benchmark

$B timing --rugnux $R [--benchmark standard|inhouse] [--threads N] [--label L] [--baseline RUN|none] [--extra=<rugnux args>] [--gpulock]

The benchmark runs a few sets. Each starts from a model that the run has to place by molecular replacement, so a single run times every stage: reading, pre-scan, indexing, integration, scaling, the space-group search, MR and model validation. It also checks that every answer is right.

--benchmark standard, the default, is the one to run on a new machine. It uses open data only: three EIGER 16M rotation sets of ordinary size and quality, and a virus capsid as the large case.

set	start model (--model)	what makes it hard
9zlo	the AlphaFold DB prediction of UreE (Proteus mirabilis), committed in tools/battery/models/	two copies to place, P2₁2₁2₁, 2.0 Å
9s02	the AlphaFold DB prediction of human PYCR1, also in models/	five copies, P2₁2₁2, 1.65 Å
9gjx	the AlphaFold DB prediction of a Bacillus licheniformis nitroreductase, also in models/	six copies, P2₁, 2.4 Å, the largest of the three datasets (37 GB)
4nwv	the deposited 4NWV (a virus capsid, I222, 45 chains once strict NCS is expanded), turned 30° and shifted 33 Å by a fixed transform	the largest: too far off for the rigid body, so MR places the whole asymmetric unit

--benchmark inhouse is the benchmark as first set up, kept so that earlier runs stay comparable: the predictions of horse myoglobin (myob_x10sa, a prediction without heme) and bovine cytochrome c (cytc_x10sa: three copies related by a pseudo-translation; the data alone cannot tell P3₁21 from P3₂21, so the model must decide the hand), and the same 4nwv. Its two in-house datasets come from the maintainers.

The predictions are committed next to their predicted-aligned-error files, which rugnux uses to cut them into rigid domains. The deposited entries the placements are checked against are committed too (tools/battery/models/pdb/), and the moved 4NWV is made at run time from the committed 4NWV with the transform in tools/battery/timing.json. A timing run needs no pdb_cache and no network.

What a machine needs

Copy these directories under one open data root (sizes as unpacked; please cite the DOIs):

directory	size	source
9zlo/	11 GB	Zenodo 10.5281/zenodo.18652652
9s02/	11 GB	MX Raw Data Repository 10.60884/NRNGS4
9gjx/	37 GB	IRRMC 10.18430/M39GJX
4nwv/	16 GB	SBGrid Data Bank 10.15785/sbgrid/9

The paths inside each directory are the inputs of tools/battery/open.json (for example 9gjx/BlNTR_9gjx/data/SC6_1_master.h5), and External test data says what each archive holds. The site config needs only the runs root and that data root (tools/battery/site.timing.example.json):

{
 "runs_root": "/path/to/runs",
 "arms": {
  "open": {"data_root": "/path/to/open"}
 }
}

and the run is

python3 tools/battery/battery.py --site /path/to/site.json timing --rugnux /path/to/rugnux

Python 3.9 or newer with gemmi and numpy is all the runner needs.

The checks

A set passes only when every check holds. The thresholds are in timing.json.

check	passes when	why
space_group	the battery's own verdict on lattice and space group passes (the hand excepted)	the data's own determination is right
placed	MODEL_PLACEMENT is MR	molecular replacement placed the model, and the R-factors, maps and model written are of that placement
mr_group	MR_SPACE_GROUP is the deposited group, hand included (P 21 21 21, P 21 21 2, P 1 21 1 and I 2 2 2 for standard; P 1 21 1 and P 32 2 1 for the two in-house sets)	the model settles the hand, so this is where a wrong enantiomorph shows
r_free	R_FREE ≤ 0.50 for the predictions, ≤ 0.35 for 4NWV	a random structure gives 0.586
confirmed	CC_MODEL_CONFIRMED_TO_D_MIN is not NONE	some shell's CC(model, data) clears rugnux's empirical null
placement_cc	the placed p_model.pdb and the reference model have correlated intensities, ≥ 0.30	see below

placement_cc is the one check that does not depend on rugnux's own scores. It computes |F|² from the placed model and from the battery's reference model for the set, each in its own cell and space group. It normalises each resolution shell to its mean, from 10 to 4 Å, and takes the best correlation over the changes of basis the two lattices allow. |F| is the same for every allowed origin and every symmetry copy, so no superposition is needed. A right placement correlates strongly (measured 0.55 to 0.96). A wrong placement gives 0.01 to 0.04, and the wrong hand gives about 0.2. This catches a placement that rugnux reports as right but is not.

Reading the output

The console prints one block per set: the verdict, rugnux's wall time and the MR time, one line per check (ok or BAD, the value, the limit), and the step times. For example:

4nwv         pass  wall   167.2 s  MR 52.0 s
    ok  space_group   I 2 2 2 (want I 2 2 2)
    ok  placed        MR (want MR)
    ...
    Reading input 0.2, Pre-scan 4.6, Indexing 5.5, Integration (geometry pre-pass) 11.8, ...

The run directory is an ordinary one, so list, table, compare and keep all work on it. Its report.md opens with a Timing benchmark section: the checks per set, then a table with one row per step and one column per set. Each results.json row carries bench_verdict, bench_reason, bench_checks, mr_seconds and steps, the seconds per step. wall_s is rugnux's own WALL_TIME, so time spent waiting for a GPU queue is not in it. The run exits with status 1 when a set fails a check or crashes. A wrong answer is a failure even when the time is good.

If the site config names a baseline_timing for the benchmark ({"standard": RUN, "inhouse": RUN}, earlier timing runs on this machine), the report is compared against it. Otherwise pass --baseline RUN. Compare timing runs only from the same machine and the same benchmark.

CPU only

$B timing --rugnux $R --extra=--no-gpu

This runs a CUDA build as if the machine had no GPU: CUDA is never touched and every step takes its CPU path. Write the option with =, because --extra --no-gpu makes Python's argument parser take --no-gpu for an option of its own. Several options go in quotes: --extra="--no-gpu -N 8". A CPU run and a GPU run agree to rounding, not bit for bit, so their last digits can differ.

Taking timings that mean something

A wall-clock number is only useful if someone else can reproduce it and if two numbers can be compared. Each lesson below was learned by getting a timing wrong first.

Build like CI

The build system sets no -march and no LTO, so that each site can choose. CI and the released binaries are built with -march=x86-64-v3 -flto=auto on Linux x86 and /arch:AVX with MSVC on Windows. A plain cmake -DCMAKE_BUILD_TYPE=Release build is a baseline binary that nobody ships. GPU work runs at the same speed in it, but the CPU-bound phases (first-pass indexing, scaling and merging) are about 26% slower. A profile of that build also points at the wrong code: Eigen has no vectorised round below SSE4.1. For any CPU-side measurement, build with at least -DCMAKE_CXX_FLAGS="-march=x86-64-v3" -DCMAKE_C_FLAGS="-march=x86-64-v3". On ARM (a GB10 or a Grace) and on Apple Silicon, leave -march out: the default target already has NEON and a vectorised round. The runner records the flags in manifest.json, so always state them.

Do not time the first run after a build

The first runs straight after a build are not representative. On a Windows laptop on 2026-10-14, the runs started immediately after a 500-target build were 1.3 to 2 times slower in every phase: GPU image loop, file reads, everything. Rerun back to back, the same binaries matched to within a few percent. Let the machine settle (the build's own I/O, antivirus scans of new binaries, a hot CPU on a laptop), then make a throw-away run first.

A quiet machine

On a shared machine, another job's load makes wall-clock numbers and parallel-efficiency numbers meaningless: they measure the other job. Profiles that report shares of cycles (perf, Nsight) still show where the time goes, but not how long it took. If the numbers are the point, take the machine to yourself. Check uptime and nvidia-smi before and after, and say what else was running. On a laptop, plug it in and keep it in the same power mode: on 2026-10-16 two back-to-back runs of 4nwv on the same binary on a Windows laptop took 167 and 206 s, with the growth spread over every CPU- and GPU-heavy step. That is thermals and power, not code. Treat such a spread as the machine's noise.

Interleave: ABAB, not AB

One run of the old binary against one of the new says almost nothing. Run them interleaved, old, new, old, new, on the same two or three sets, in one session on a quiet machine, and compare the pairs. Slow drift (the page cache, thermals, another user) then affects both sides alike. Keep the old binary for this; the run directory's bin/rugnux is one. Several minutes of ABAB have settled questions that a whole battery's totals had answered wrongly, more than once.

The page cache

The battery and the benchmark read every image from disk. Whether those files are still in the operating system's page cache can change a set's time several-fold, with every result identical. On the PSI workstation, whose data disk is a 7200 rpm HDD, one 10 GB set took 15 s warm and 111 s cold. A whole battery once swung from 11 to 19 minutes only because a large job beforehand had pushed its data out of the cache. The signs are:

• the slowdown scales with each set's size on disk, not with its content;
• it can be bimodal: the sets still cached are unchanged to the second, while the evicted ones are several times slower;
• divide a set's extra seconds by its size: if the result is the disk's read speed, it is the disk;
• rugnux reads the frames in each of its passes, and on a machine that can hold the set in RAM the later passes are always warm. A real regression slows them too, while a cold read slows only the first pass (compare the Integration (geometry pre-pass) step with Integration (refined geometry) in the log).

Either time warm runs (one throw-away run first, and enough RAM to hold the set) or time cold ones deliberately (on Linux, drop the file from the cache with posix_fadvise(POSIX_FADV_DONTNEED) or echo 1 > /proc/sys/vm/drop_caches). Either way, say which. A machine whose RAM is about the size of a set cannot keep it warm, so on a 16 GB Mac a 16 GB set always reads cold.

The GPU lock

On a machine where several people or agents share the GPUs, the site's gpulock serialises the jobs. battery.py --gpulock takes the lock for each rugnux separately, so other jobs can run between sets, and the run is marked "timing NOT a reference". For a timing session, take the lock once around the whole session instead, so that nothing else gets in between your A and B runs:

flock /data/tmp/gpulock bash -c "
  $B timing --rugnux old/rugnux --label A1
  $B timing --rugnux new/rugnux --label B1
  $B timing --rugnux old/rugnux --label A2
  $B timing --rugnux new/rugnux --label B2"

(/data/tmp/gpulock is the lock file at PSI; use your site's.)

The runner still checks for other processes on the GPU before each set (gpu_others in results.json). A machine without such a lock, a laptop for example, sets "gpulock": null and runs without --gpulock.

Report the hardware and the conditions

manifest.json records the CPU, cores and threads, RAM and GPUs on Linux. On Windows and macOS it may say "not recorded", so write the hardware down yourself. A timing means nothing without: the machine (CPU, RAM, GPU and its memory, driver), the OS, the binary (git hash, sha256, build flags, compiler and CUDA version), -N if set, warm or cold data, what else was running, mains or battery power, and which runs are the first after a build.

Windows

• Run the runner as python -X utf8 tools\battery\battery.py ..., because the report and the console table are UTF-8. Give --rugnux the path of rugnux.exe. The run directory gets its own copy of the binary, so any DLL it needs must be found on PATH.
• WDDM can quietly page GPU memory to system RAM. A consumer or laptop card under Windows' WDDM driver model does not fail an allocation that exceeds the card. It spills to "shared GPU memory" in system RAM, and the kernels touching it slow down several-fold with no error. rugnux decides between GPU and CPU from the card's total memory (80% for scaling and the model fits), but memory held from earlier phases and the driver's own reserve count too. During a run, watch Task Manager's Dedicated and Shared GPU memory for the process: if the shared figure grows, the timing measures paging. On 2026-10-15 an 8 GB laptop card spent five times longer in model validation for this reason alone.
• Windows has no flock and no process groups. The timing benchmark does not need them.

macOS

• The macOS build has no CUDA. Bragg integration, the model density and the molecular-replacement rotation function run on the Apple GPU through Metal. Everything else runs on the CPU, so a Mac timing is mostly a CPU-bound timing. Molecular replacement in particular takes much longer than on a CUDA machine, as its translation function and rigid body stay on the CPU.
• Memory is unified: CPU and GPU share it, and the OS compresses memory before it swaps. A 16 GB Mac can run the benchmark, but its biggest set comes close to the limit, so report both measures below.

Memory peaks

The battery does not record memory, so measure it next to a timing when it matters. Each row's cmd in results.json is the exact command a set ran, so you can rerun that command by hand under a tool that measures:

• Linux: /usr/bin/time -v <cmd> reports Maximum resident set size, which is VmHWM in /proc/<pid>/status while the process runs. RSS is not the same as live data: glibc keeps freed memory in its arenas, and pinned host buffers for the GPU count too. On one large set about 0.5 GB of the peak was memory freed but not yet returned to the system, and up to 2.4 GB on an older build.
• macOS: /usr/bin/time -l <cmd> reports both the maximum RSS and the peak memory footprint, which includes compressed memory and GPU buffers and can be much larger. On 2026-10-16, 4nwv showed 9.1 GiB of RSS against about 22 GiB of footprint.
• Windows: sample the process's peak working set (PowerShell (Get-Process rugnux).PeakWorkingSet64) while it runs, and Task Manager's dedicated and shared GPU memory.
• GPU: nvidia-smi --query-gpu=memory.used --format=csv -lms 500 while the set runs. rugnux's log also states each GPU-or-CPU decision with its numbers, for example Model validation: ... 4.42 GB held of a 6.84 GB budget, 80% of the card's 8.55 GB and ... scaling on the CPU instead. Quote those lines; they explain most surprises between machines.

Installing Rugnux ▸ Memory explains what sets the peak.

Per-step times: the TIMING lines

rugnux writes its full log to <prefix>_log.txt (p_log.txt in the battery) at debug level, and every timed section adds a line when it ends:

[2026-10-16 07:42:45.430] [rugnux] [debug] TIMING step Integration (geometry pre-pass), 500 images 11.800
[2026-10-16 07:42:52.230] [Rugnux] [debug] TIMING merge P1 4.697

The TIMING step ... lines are the steps shown on the console, and they add up to the run's wall time. These are what the benchmark tabulates. The other TIMING ... lines are nested sections inside a step (rsm ... sections inside merge ..., everything in a pass before its pass (...) line). tools/rugnux_timing.py p_log.txt tabulates them, and --all lists every section. For a rotation run the steps are, in order:

step	covers
Reading input	opening the master file, geometry and mask
Pre-scan	a projection of a sample of frames: beam stop, background beam centre, spot widths, defective pixels
Indexing	first-pass spot finding and lattice search on the first part of the sweep
Integration (geometry pre-pass), N images	integrating the first part of the sweep at the file's geometry
Scaling and post-refinement	scaling that pre-pass and post-refining the detector geometry, cell and axis from it
Metric-symmetry check, Geometry check	only where needed: re-integration of a promoted lattice class, more geometry rounds
Indexing at the refined geometry	re-indexing with the refined geometry
Integration (refined geometry), N images	the whole sweep, every frame
Scaling and merging	combining partials, scaling, the space-group search, the resolution cut, merging
Analyses	twinning, anisotropy and the other report-only analyses
Model validation	--model only: structure factors, scale fits, rigid-body placement, maps
Molecular replacement	only where the rigid body did not place the model (always, in the timing benchmark)
Writing files	MTZ, mmCIF, HKL, report

A step that is slower on one machine than on another points at the code, and a step that is slower everywhere at once points at the conditions. Send the whole step table, not only the total.

NVTX ranges for Nsight Systems

A CUDA build compiled with the NVTX headers (found next to nvcc; otherwise the configure says NVTX3 headers not found - NVTX ranges compiled out) opens an NVTX range for every console step (Step <n> <title>) and for every TIMING section, with the same names. An Nsight Systems trace therefore shows the pipeline by name above the CUDA kernels and copies:

nsys profile --trace=cuda,nvtx,osrt -o 4nwv_trace <the set's cmd from results.json>

Without a profiler attached, the ranges cost nothing.

What to send back

A remote measurement is only useful if the owner can compare it with the others. Send one archive (a "pack") per session, named after the machine, the binary and the date, for example rugnux_<machine>_<git>_<YYYYMMDD>.tar.gz, laid out like this:

BRIEF.md
runs/<n>_<what>/report.md          # unedited, as the runner wrote them
runs/<n>_<what>/report.html
runs/<n>_<what>/results.json
runs/<n>_<what>/manifest.json
runs/<n>_<what>/SOURCE_RUN.txt     # the run directory's original path
runs/<n>_<what>/logs/<set>/{p_log.txt,p_report.txt,run.log}
timing/                            # step tables, ABAB comparisons
memory/                            # peak RSS / footprint / GPU memory samples, with how they were taken
build/                             # the exact configure command (or CMakeCache.txt flags)

BRIEF.md comes first and should be short: the host and hardware, the OS, the binary (git, sha256, build flags, compiler, CUDA), the data (which pack, warm or cold), the conditions (quiet or shared, power, first run after a build), and then the results. Give a table per run, the step table of the slowest set, what changed against the previous pack from the same machine, and what is not explained yet together with the experiment that would explain it. Number the runs in the order they were made, and say which ones are not representative and why (1_ straight after the build, for example).

Leave out the maps, MTZ files and binaries unless they are asked for, since they are large and the logs and reports carry the numbers. Never edit a report.md or results.json: comment in the brief instead. If you had to patch the runner to make it work, include the patch and say so. A pack never contains anything from the private arm.