Commit Graph
1288 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 5.5 aec6f6aabc rugnux: background clip scaled by the CCD gain measured in the pre-scan
The background ring's high-side clip, mean + 4*sqrt(mean), assumes counted photons. A CCD reports
ADU: on a Rigaku Saturn a background pixel scatters 4-5x its mean, so the clip sat at ~2 sigma, cut
the upper tail of every ring and added a positive offset to every reflection, compressing the
intensity statistics (a false 422 on one twinned I41 crystal).

No header carries a usable gain (SMV states none, every marCCD file the writer default of 1 ADU per
photon), so it is measured: background_gain::MeasureFrame takes the variance per unit mean of the
difference of pixels two columns apart, per level, clipped, on every fourth row of up to 16 of the
pre-scan's frames; Gain is the pair-weighted median over the levels. The clip becomes
mean + n*sqrt(max(g,1)*mean) (BraggIntegrationSettings::bkg_pixel_gain, CPU and GPU engines).
Only for SMV/marCCD input (ProcessConfig::counts_in_adu, set from the file format); HDF5 and CBF
keep g = 1, the identity. One log line reports the value. No new option.

Measured g: Saturn 944+ 4.7-5.2, Saturn 944 2.7, A200 0.68, ADSC ~0.3, marCCD ~0.3 - so the
ADSC/marCCD/A200 sets keep the photon clip exactly.

Checks (reflection data via gemmi and p.hkl, against the rc175 battery outputs / base build):
identical on 11if, 7qis, 8a1a, myob_x10sa, cytc_x10sa (photon counting), 4bwl, 5epe (ADSC,
marCCD) and 6cee (A200). Saturn sets, battery baseline -> this change:
  3r6o  I4122 R_meas 19.3% ISa 4.4 <|L|> .321 -> I41 (correct) 12.1% 8.1 .384
  3mc4  R_meas 10.2 -> 9.8 %, ISa 12.1 -> 12.7, d_min 1.79 -> 1.75 A, <|L|> .360 -> .391
  5uth  12.6 -> 11.8 %, ISa 10.4 -> 10.1, <|L|> .426 -> .483, offset vs |Fc|^2 0.21 -> 0.00
  5vml  10.2 -> 9.6 %, ISa 11.0 -> 11.6, <|L|> .426 -> .475, offset 0.15 -> 0.05
  5cc8  8.1 -> 7.9 %, ISa 10.8 -> 11.2, <|L|> .469 -> .494
  3meb  14.5 -> 14.6 %, ISa 13.2 -> 13.2, <|L|> .442 -> .457
Space groups unchanged except 3r6o. Pre-scan cost: +0.5 s on a 9 Mpx ADSC sweep, within noise on
a 15 Mpx marCCD sweep, nothing on HDF5/CBF.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 21:18:12 +02:00
leonarski_fandClaude Opus 5.5 e9d4f22210 ResultReport: RUGNUX_GIT has no trailing space when the build knows no commit date
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 19:33:01 +02:00
leonarski_fandClaude Opus 5.5 5eba68432e Merge branch 'mmcif-provenance' into rc175: deposition-grade mmCIF, provenance in the reflection files, git hash for builds without git
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 19:32:50 +02:00
leonarski_fandClaude Opus 5.5 e1a3dcb9d4 Merge branch 'firstpass-split' into rc175: the first pass as named steps, no output change
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 19:25:12 +02:00
leonarski_fandClaude Opus 5.5 97cac203a6 rugnux: split FirstPassRotationIndexing into named steps
Pure refactor, no behaviour change. The 2460-line function with ~25 lambdas
becomes Rugnux::FirstPassRun (RugnuxFirstPass.h): the state the steps share
(spot cache, validation frames, schemes, committed spot-list state, the
speculative passes) as members, and one member function per step taking
and returning the standing lattice (FirstPass). FirstPassRotationIndexing is
now the ordered list of decisions:

  PrefetchCertainFrames, IndexSchemes (+ centre/short-axis speculation),
  AxisSignRescue, BeamCentreCheck (IndexAtMeasuredCentre,
  CentreWhenFileFails, CentreByPooledSpots, CentreWhenBothIndex,
  CentresDisagree), LongAxisRescue, AddEndWedge, ReleaseSpotEngines,
  RequireLattice, ShortAxisHypothesis, SubLatticeHypothesis, Pass1Fallback,
  last-resort FirstPassLadder, MeasureVerdict, RejectChanceLattice,
  AdoptLattice, StartLeftoverCensus, ReportTiltWalk, MeasureSpotBudget,
  Pass1SupercellGuard.

Files: RugnuxFirstPass.cpp (order + shared machinery: spots, scoring,
schemes, ladder), RugnuxFirstPassRescues.cpp, RugnuxFirstPassAccept.cpp.
Bodies are moved verbatim (Rugnux members reached through rx.); every log
line is unchanged. Literal duplicates folded: the two MAX_SPOT_KEYS
constants, and the feed-every-scheme / run-every-indexer loops of the
long-axis rescue and the short-axis fallback (now FeedSchemes/RunIndexing,
the same calls in the same order). The leftover census, which outlives the
pass, captures what it needs explicitly instead of [&].

Verified against the rc175 0b2dde214 binary on 16 sets (7kcn, 8v4o, 5nw5,
9rci, 6z9g, 9min, nothing_1/2, 7atg, myob/cytc_x10sa, 8a1a, kdp, insu low
ISa, 6p8j, 9qw8) and -N 8 on 7kcn/8v4o/9rci: p.mtz md5-identical, logs
identical as multisets apart from timings.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 19:24:40 +02:00
leonarski_fandClaude Opus 5.5 bc9f2caccb Merge branch 'report-work' into rc175: beam-centre warning, overloads and overlaps, XDS geometry keywords in the report
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 19:09:44 +02:00
leonarski_fandClaude Opus 5.5 c0bcf1a7a9 rugnux report: XDS geometry keywords as one report section, not p_XDS.INP
The refined geometry is no longer written as a whole p_XDS.INP (and the
report's XDS_INPUT= key is gone). Instead the report carries one section,
XDS GEOMETRY KEYWORDS, with the XDS.INP keywords that carry the geometry the
result was integrated at: ORGX/ORGY (PONI, 1-based), DETECTOR_DISTANCE, the
detector axes where the detector is tilted, INCIDENT_BEAM_DIRECTION with
X-RAY_WAVELENGTH, and ROTATION_AXIS in XDS's sense (rugnux's axis negated,
as verified with XDS IDXREF). No cell or space group: XDS determines those.
The lines carry a "# XDS.INP> " prefix, so sed lifts them ready to paste.

The same keywords were logged by LogXDSGeometry at the start of every pass,
with the axis in rugnux's sense; that printing is removed, so the report
section is the one place they are stated. Docs updated (RUGNUX_REPORT,
RUGNUX_INTEGRATION, RUGNUX, RUGNUX_TUTORIAL).

Output only; p.mtz md5-identical on 3 battery sets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 19:09:27 +02:00
leonarski_fandClaude Opus 5.5 447d8f8b01 GitInfo: a source tree without its own git history builds, reporting commit "unknown"
The commit is already re-stamped on every build (JFJochGitInfo runs
GitInfoStamp.cmake with cmake -P; configure_file rewrites GitInfo.cpp
only when the value changed). What was missing is the case with no git:
a downloaded tarball left the SHA empty, and one unpacked inside some
other repository would have reported that repository's HEAD. The stamp
now asks git only when the source directory is itself the top of a work
tree, and otherwise records commit "unknown" beside the version from
VERSION. The reflection files' git description drops its trailing date
when there is none.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 19:04:22 +02:00
leonarski_fandClaude Opus 5.5 7ed1544136 rugnux report: BEAM_CENTRE_OFF only beyond three times the tolerance
At one times BeamCenterNeed_pxl the warning fired on 107 of the 250 battery
sets - most headers are a few pixels off. It now fires beyond three times
the tolerance; between one and three times the summary's Beam centre line
states the offset and recommends correcting the beamline value, without a
warning. The background-only branch (runs that refined no centre) uses the
same bound.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 18:57:06 +02:00
leonarski_fandClaude Opus 5.5 4120b0daa8 Merged mmCIF: drop the scaled unmerged block
The unmerged observations already leave a run as the unmerged MTZ (and
the scaled fulls as the .hkl), and deposition does not come straight
from rugnux, so the _diffrn_refln block made the .cif up to ten times
larger for no consumer. The .cif is one data block again; the keys,
software rows, metadata, R_merge/R_pim/CC* and provenance stay.
ScaledFull loses the scale and frame it carried only for that block.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 18:51:56 +02:00
leonarski_fandClaude Opus 5.5 7624bbe12e battery: paper_plots.py draws the other figures when no run has REFMAC R-free
The REFMAC check is opt-in, so a battery run without --model-check has no
R-free pairs, and the axis range min()/max() over them crashed the script
before any figure was drawn. The R-free figure is now its own gnuplot block,
left out (with a line in summary.txt saying why) when there is nothing to
draw; d_min and the time figures are drawn as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5 244f3c0f9d rugnux: write the refined geometry as <prefix>_XDS.INP
A rotation run now writes the geometry it integrated at as an XDS input
beside the report: ORGX/ORGY (the PONI, 1-based), DETECTOR_DISTANCE, the
detector axes carrying the tilt, wavelength, the rotation axis the run
integrated at (ProcessResult::used_rotation_axis, after any sign rescue),
oscillation (with an adopted rotation scale), starting angle/frame, data
range, NX/NY/QX/QY, OVERLOAD, sensor thickness, space group and cell (left
commented where the group is not in its number's reference setting),
FRIEDEL'S_LAW per -A and the merged resolution range. HDF5 inputs get the
master file as an x_??????.h5 template and a LIB= line to point at the
reader plugin; a CBF series gets its ???? template and its own frame
numbers. The report names the file (XDS_INPUT=). Conventions per Kabsch
(2010) Acta Cryst. D66, 125-132.

The rotation axis goes out negated: rugnux turns the crystal by -phi about
its axis (IndexAndRefine), XDS by +phi about ROTATION_AXIS. Checked with
XDS IDXREF on two exports (one CBF, one HDF5 via durin): the negated axis
indexes 99.5% / 14% of the spots, rugnux's own sign 37% / 0.04%. On the
CBF set XDS then ran DEFPIX/INTEGRATE/CORRECT from the file unchanged
(apart from the job list): 99.8% complete, ISa 23.3, direct beam within
0.15 px and cell within 0.07 A of rugnux's.

Output only; no result changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5 2632b77eb5 rugnux report: count overloads and overlapping spots, warn with advice
Overloads: the integrated reflections whose signal disk held a saturated
pixel are counted on the integration outcome before the merge, over the
whole range and over the reflections coarser than 4 A, where they
concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An
OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution
reflections, and recommends attenuation / a shorter exposure. Measured: a
strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an
ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%.

Overlaps: both integration engines count the signal disks a nearer
neighbour's prediction takes pixels from (the overlap ownership the default
--overlap exclude already builds), and those left under --overlap-minpk of
their own - BraggIntegrationCounts::overlapped / overlapped_major (log line
"Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning
fires from 1% in the second and recommends a longer distance, finer slicing
or a smaller beam; on 10 battery sets it fires on the two long-axis ones
(8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both;
documented in RUGNUX_REPORT.md.

Counters only; nothing reads them. p.mtz md5-identical to the rc175
battery on 10 sets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5 5584460de0 Reflection files record their provenance: commit, command line, input file
The merged .mtz and the unmerged .mtz carry, after the existing "From
Rugnux <version>" record, HISTORY records with the git commit and date,
the command line, the input file (absolute path) and a cheap identity of
it - size and modification time, the master file's for HDF5, nothing
hashed - and the collection date. MTZ allows 30 records of 80
characters; a longer value runs on over records starting with '+', cut
so that no record ends in a blank a reader would strip.

The mmCIF names the commit in _software.description and the command
line, input file and identity under _jfjoch_provenance (the local prefix
the other private items use; no PDBx item holds them), the first two as
text fields so any character survives.

OutputProvenance is filled once in the CLI and travels in ProcessConfig
to every writer of a reflection file. Reflection data are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 17:52:43 +02:00
leonarski_fandClaude Opus 5.5 fc6b91555f Merged mmCIF: deposition keys, R_merge/R_pim/CC*, experiment metadata, scaled unmerged block
The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the
entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on
_reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/
_diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a
loop naming Rugnux for 'data reduction' and for 'data scaling'.

The experiment as the input states it: _diffrn.ambient_temp from the
sample temperature, the source's general class (from NXmx
/entry/source/type, which the HDF5 reader now reads), facility and
beamline, the detector make/model in _diffrn_detector.type (where
.detector used to carry it) and its class in .detector where the name
settles it, and the collection date from /entry/start_time.

R_merge and R_pim are formed beside R_meas from the same per-reflection
sums (stills and rotation merges) and written with CC* overall and per
shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the
_reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as
three new columns of the shell tables. R_pim following Weiss (2001),
CC* following Karplus & Diederichs (2012).

A second data block, data_sample_unmerged, carries the scaled unmerged
observations of a rotation merge as _diffrn_refln (measured index,
frame, scaled I and sigma, pdbx_scale_value) - the same records as the
.hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now
carries the scale and the frame. The unmerged MTZ stays unscaled.

Reflection data are unchanged: the merged _refln rows, the .mtz columns
and the .hkl are written as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 17:50:11 +02:00
leonarski_fandClaude Opus 5.5 f462047038 rugnux report: BEAM_CENTRE_OFF warning when the file's beam centre is wrong
The pre-scan's beam-centre check and the post-refinement both measure where
the direct beam lands, but the report only carried the centre the run used.
Section 1 now sets the file's direct beam (FILE_DIRECT_BEAM) beside the
background's (BACKGROUND_DIRECT_BEAM, with its sigma), the shift of the
refined centre from the file's (BEAM_CENTRE_SHIFT) and the tolerance the
geometry asks for (BEAM_CENTRE_TOLERANCE: BeamCenterNeed_pxl, the number the
log's check prints, here on the run's own cell). A shift above the tolerance
is a BEAM_CENTRE_OFF warning naming the file's, refined and background
centres, the difference in px and mm and the value to correct the beamline
to; a run that refined no centre is warned on the background alone where
its fit resolves the difference (3 sigma). The summary gains a Beam centre
line.

The refined centre, not the background, is the trigger wherever one exists:
on a battery set where XDS's refined direct beam and rugnux's agree to 1 px,
the background fit is 12 px away from both. Report-only; no result changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 17:07:22 +02:00
leonarski_fandClaude Opus 5.5 341c347a3f Merge branch 'docs-battery' into rc175: battery report of the pushed rc.175
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 16:38:41 +02:00
leonarski_fandClaude Opus 5.5 13badcfa39 Docs: battery report of the pushed rc.175, with commentary and figures
Full open and in-house battery on 0b2dde214 (run 20261009-1447_0b2dde_rc175-0b2dde2-full) against
the 10-07 battery on 9ced4c444, without the REFMAC check: the R-free figure and numbers are kept from
the rc.174 battery. Report regenerated from the run with the current battery tools (verdicts scored
by the current scorer, so KDP passes as an indistinguishable alternative), commentary written by
hand. 5ky6 and the low-ISa cubic insulin set go from pass to fail, accepted losses of taking the
pre-pass-2 decisions from the first part of the sweep; every other verdict is unchanged. The whole
battery took 4898 s against 5795 s, most of it model validation now on the GPU (278 s against
1024 s); without it the HDF5 sets are at the same speed. Time, zoomed time, in-house time and
resolution figures regenerated with paper_plots.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 16:38:23 +02:00
leonarski_fandClaude Opus 5.5 748033d001 Merge branch 'battery-sgequiv' into rc175: groups the intensities cannot tell apart score as a pass
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 16:27:41 +02:00
leonarski_fandClaude Opus 5.5 d28c8b8bb3 battery: same-absence groups of one Laue class pass as indistinguishable_alternative
sgequiv keyed space groups on their proper+improper rotation matrices, so groups with
identical merged intensities but different point groups (I41md vs I-42d, Pc vs P2/c,
Pna21 vs Pnma, P2 vs P2/m) were called a "REAL difference", contradicting its own
docstring, and score.py compared point groups literally on the XDS arms. A run that
wrote one member of such a pair and listed the other in SPACE_GROUP_ALTERNATIVES failed
as sym_other / sym_over.

sgequiv now keys each group, in every setting gemmi tabulates, on its Laue rotations
(operations plus their negatives), centring and absences; describe() separates the new
different-point-group pairs from the old same-point-group ones, and enantiomorph
handling is unchanged. score.py passes a row, on both arms, when some setting of the
reference has the same signature as our group in our own setting: verdict pass, cause
indistinguishable_alternative, the reason saying whether rugnux listed the reference.

Pass, not unscored: unlike an unmeasured screw axis, everything was measured and the
answer is as right as the data allow; only a model or a strong anomalous signal decides.
The separate cause keeps such rows countable. Being listed in SPACE_GROUP_ALTERNATIVES
alone is deliberately not enough: that list also carries the screw alternatives of an
undetermined axis, whose absences differ, and those stay under the screw rule. The
open arm's same-point-group UNDECIDABLE pass now also requires the setting check.

results.json gains sg_alternatives (rugnux's SPACE_GROUP_ALTERNATIVES). KDP joins the
ci tier. Re-scoring the open and in-house arms of the two latest full runs changes one
verdict in each: kdp_x10sa_20keV fail/sym_other -> pass/indistinguishable_alternative.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 16:27:23 +02:00
leonarski_fandClaude Opus 5.5 f914a116aa ci: the battery workflow is listed as "Validation Battery", next to "Build Packages"
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 15:39:54 +02:00
leonarski_fandClaude Opus 5.5 0b2dde2147 Merge branch 'battery-ci' into rc175: a ci tier, a closing per-set table and the battery workflow
Build Packages / Create release (push) Successful in 9s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 3m25s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 10m16s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 11m37s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 13m59s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 19m14s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 19m43s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 21m3s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 25m3s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 26m3s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 18m57s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 18m15s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 19m27s
Build Packages / Generate python client (push) Successful in 32s
Build Packages / Build documentation (push) Successful in 1m17s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 19m44s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 20m54s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 19m7s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 16m55s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 16m40s
Validation battery / Validation battery (push) Successful in 47m59s
Build Packages / Unit tests (push) Successful in 1h26m17s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:43:16 +02:00
leonarski_fandClaude Opus 5.5 366935fe26 battery: right-align the reference values of the run table too, so decimal points line up
Each numeric pair is now rugnux and reference both right-aligned to a fixed width around the "/",
with fixed decimals per column (d_min 2, ISa / CC1/2 / R_meas / R-free 1, time 0), so the slashes
and the decimal points of a column sit on one line; "-" takes the same width.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:39:20 +02:00
leonarski_fandClaude Opus 5.5 fcb9c94134 Merge branch 'gpusf' into rc175: model validation on the GPU, in parallel, on every card
Structure factors, maps and the null on the GPU (with gemmi's mask shrink step), the second
validation started on a forecast beside the first, change-of-basis candidates scored at the same
time, validation work spread over the visible cards, the merge's GPU engine freed before the
validation, a CUDA error failing the validation instead of restarting it on the CPU, and no
first-validation maps where the forecast one is kept. Output files md5-identical to the CPU path's
reflection files and placed model on 8sa8, 8xtg, 9ea5, 8tyy, 7qis and myob; maps within float
tolerance of the CPU path and bit-identical run to run and at -N 8. Validation time 8sa8 24.0 ->
7.4 s, 8xtg 18.4 -> 5.0 s, 9ea5 19.1 -> 3.0 s, 8tyy 47.7 -> 3.0 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:39:09 +02:00
leonarski_fandClaude Opus 5.5 c5970a76f4 ci: the battery workflow keeps the battery machine's own paths and mounts
The workflow runs on the battery machine, not on the development workstation, so the data roots
(/data/open, /data/inhouse), the whole /data/battery mount, the scratch directory
(/data/battery/scratch/<run id>), the kept reports (/data/battery/ci_reports) and the runner's
valid_volumes are back as that machine has them; the remark tying the image to one host's driver
is gone. The private arm stays out because the site config names only open and inhouse.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:38:43 +02:00
leonarski_fandClaude Opus 5.5 32e78823fb ModelValidation: the non-CUDA build compiles again
The multi-GPU placement (replicate threads pinned with pin_gpu) and the
second-validation memory rule read RigidBodyGPUPool and get_gpu_count(),
which a build without CUDA does not have; both are now inside
JFJOCH_USE_CUDA, where the pool can exist at all. Nothing changes in a
CUDA build. Non-CUDA rugnux and jfjoch_test build; [ModelValidation] 22
cases pass there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:38:20 +02:00
leonarski_fandClaude Opus 5.5 9beb8a7813 battery: align the run table's pairs on their slashes, the ticks in a column of their own
Each rugnux / reference pair is now rugnux right-aligned to a fixed width, "/", the reference
left-aligned to the same width ("-" for a missing value), so every slash of a column sits on one
line; the tick or cross follows the space-group pair in its own column.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:37:50 +02:00
leonarski_fandClaude Opus 5.5 9a1f729993 rugnux: do not make the first validation's maps where the forecast one is kept
Where the validation in the model's setting, started beside the first on
the forecast (b5980c9ba), is going to be kept, it writes the maps, the map
coefficients and the placed model on the written axes, and the first
validation's p_2fofc/p_fofc/p_anom.ccp4 and p_maps.mtz were overwritten
unread. ModelValidationSchedule::maps_superseded is asked once the
verdict, the setting and the indexing are decided, before any map is
made; the first validation then makes none (the map coefficients and the
MTZ are not built, the three map transforms not run, nothing written).
The answer is the same rule that keeps the forecast validation
afterwards (forecast_kept, now one lambda for both). The kept validation
also brings back the reflections it relabelled, free set included, so
relabel_output does not reindex and redraw them a second time. Where the
forecast is discarded or was never started, everything runs as before,
and modelpar's delete-before-rewrite stays for those cases. If the kept
validation then fails, a warning says no maps were written.

Log lines that disappear (first validation only, kept-forecast runs):
"mean 2mFo-DFc density at atom centres = ...", "anomalous difference map
from ... Bijvoet pairs; strongest density ...", "wrote p_2fofc.ccp4, ...";
in their place: "no maps made here - the validation in the model's
setting writes them". (The "anomalous density ... inverted" warning would
also go; it cannot be raised where a forecast is kept.)

Verified on 8sa8, 8xtg, 9ea5 (forecast kept), 8tyy, 7qis, myob_x10sa (no
forecast): p.mtz, p.hkl, p.cif, the three maps, p_maps.mtz and
p_model.cif/pdb md5-identical to the merge commit, repeated and at -N 8;
the discard path (8sa8 with a rotated model) identical files and log.
Time saved is small with the maps already on the GPU: 8sa8 7.6 -> 7.4 s,
8xtg 5.1 -> 5.0 s, 9ea5 within noise.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:34:46 +02:00
leonarski_fandClaude Opus 5.5 f2be4b472e ci: a validation-battery workflow - the ci tier on every push, any subset by hand
Runs on the machine that holds the battery data (runner label jfjoch_battery): builds rugnux with
the CI flags in the rocky8 image (its CUDA 12.9 suits the host driver; python3.12 is installed for
the scorer, the image's own python3 is 3.6), writes a site config naming only the open and in-house
arms, and runs `battery.py run --tier ci` on a push or the given subset (empty: the full battery)
by hand. The job fails when a ci-tier set fails; report.md and the log are kept as the artifact and
under /data/battery/ci/reports, the run directory is deleted.

Only what the job needs is mounted: the two data roots read-only, the PDB/COD caches, a ci scratch
directory and the host's full-run lock file - not /data/battery as a whole, which also holds the
private manifest and the private runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:31:32 +02:00
leonarski_fandClaude Opus 5.5 8c6670a3f4 battery: a ci tier, a closing table per set and a failing exit on it
`battery.py run` now ends with one line per set, each column rugnux / reference: the space group
with a tick or cross by the scorer's own verdict, d_min, ISa, CC1/2 overall and in the highest
shell, R_meas of the lowest shell, the model's R-free and rugnux's wall time. `battery.py table
RUN` prints it for any finished run (rows are rescored, so older runs get the new columns).

The highest-shell CC1/2 is new in results.json (cc_half_high, read off rugnux's own shell table)
with XDS's beside it (cc_half_high_ref, from CORRECT.LP's finest shell; the in-house manifest's
refs regenerated with `refs --write`, which added only that key).

A new "ci" tier: 25 sets that passed with the same space group and steady d_min / ISa in all four
rc174/rc175 full runs (and in the rc175-next verdict runs where they ran), with no warning that
questions the lattice or the group. About 10 minutes of rugnux (616 s measured) on the RTX 5080. A run that
includes ci-tier sets exits 1 when one of them fails (lattice, symmetry, crash, timeout, missing
input); nothing else fails a run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 14:31:32 +02:00
leonarski_fandClaude Opus 5.5 278c488c37 Merge branch 'modelpar' into gpusf
Both sides kept: modelpar's parallel basis scoring, the second validation
started on a forecast beside the first (write gate, schedule parameter),
the multi-GPU placement and delete-before-rewrite; gpusf's GPU structure
factors, maps and null engine, failure-instead-of-restart, and the merge
engine released before the validation (now just before modelpar's
ValidateAgainstModel call, after the forecast lambda is set up).

Placement: each validation's structure-factor engines (d_min and the
null's) are made on the card of the thread that runs it - the main
thread's for the first validation, card 1 % count for the speculative
second, which pins itself there - so two validations on two cards use
both, as the rigid-body pools do.

One memory rule for both, per card, from total memory, up front: a
validation plans at most half of its card - its structure-factor engines
a quarter together (was half for the d_min engine alone), its rigid-body
engines a quarter (RigidBodyGPUPool) - and the second validation runs
beside the first only where twice the first's plan (rigid-body planned
bytes + the d_min engine, twice it where a null is coming, the null's
engine being no larger) fits half of all cards' memory together (was:
twice the rigid-body plan within a quarter). The card's total is read
once when the engine is made; a CUDA error there fails the validation
like any other.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 13:51:41 +02:00
leonarski_fandClaude Opus 5.5 291240a969 rugnux: the measured-centre spot finding beside the first pass keeps off the first pass's card
With two cards or more, the speculation's spot-finding workers took card t % cards, so some of them
shared card 0 with the first pass they run beside. They now take the other cards, worker t on card
1 + t % (cards - 1): on two cards all of them on card 1, as before the previous commit, and on more
cards spread over cards 1..n-1. Spots are a pure function of the frame, so the card cannot change
them; on one card the speculation does not start. Not measured on more than one card.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 13:47:45 +02:00
leonarski_fandClaude Opus 5.5 0f707caab9 ModelValidation: the null's structure factors on the GPU too
The null - nine random placements of the model, and the real model's side
of the comparison - is scored to a coarser limit (3.5 A, or where the free
set allows), and those structure factors still ran on the CPU, one model
at a time on each replicate thread. Where the d_min structure factors are
on the GPU, a second ModelStructureFactorsGPU engine at the null's limit
is now made, on the same card, before any replicate starts; every
replicate and the real side go through it, so all of them are computed
the same way. It is decided like the first engine, from the card's total
memory: the two together must stay within half of it, else the null stays
on the CPU (logged). The replicates' threads may sit on other cards
(modelpar's 8d6e9d3a4); each call works on the engine's own device. Where
the null is scored to d_min it shares the d_min engine, as before.

Measured (validation time, before -> after; CPU path for reference):
  P3_2 1.55 A:  11.2 -> 9.0 s (CPU 17.7)
  P2_1 oblique: 5.1 -> 4.5 s (CPU 7.9)
  C2 1.11 A:   12.9 -> 13.2 s (CPU 23.3) - its null time is in the
               scale fits, which stay on the CPU
p.mtz md5-identical in all four variants per set (before, after, CPU,
-N 8); maps and map MTZ bit-identical after vs -N 8; the null's sigma
moves only in the second decimal (+113.39 -> +113.38, +86.13 -> +86.14).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 13:08:09 +02:00
leonarski_fandClaude Opus 5.5 f33d8e2a68 rugnux: measured-centre spot finding beside the first pass uses every card
The speculation that spot-finds and indexes at the measured beam centre beside the first pass
(only with two GPUs or more) put all its spot-finding workers on card 1. The file centre's spot
finding ends long before the speculation's, so the speculation is what the first pass waits on
(T4 server, kdp: main pass spot finding done at 1.3 s, speculation at 5.6 s). Its workers now
take card t % cards, as the first pass's own workers do; worker t builds and frees its engine on
the same card, since the destructor's OnWorkers maps t the same way.

Spots are a pure function of the frame, so which card finds them cannot change them. The
prototype measured kdp 25.9 -> 23.6 s on the 4-card T4 with md5 unchanged on 5 sets. On a
one-card machine the speculation does not start (get_gpu_count() < 2), so nothing changes there:
p.mtz md5 identical to rc175-next 41fab6156 on myob_x10sa, cytc_x10sa, thau_x10sa_16keV, 8a1a,
11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N, -N 8 and CUDA_VISIBLE_DEVICES=0 (5080
workstation). The multi-card gain and the 2-card risk (speculation workers now share card 0 with
the main pass) are not measured here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:51:38 +02:00
leonarski_fandClaude Opus 5.5 b43844ae06 rugnux: the merging engines sit on their card's NUMA node while they drive it
The image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two
threads that drive the merging tail were not: the quality guard took its card with set_gpu(1),
and the main engine runs on the pipeline's own thread, which goes on to everything else. On a
multi-socket server their pageable copies to the card went through a remote socket.

RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost,
IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous
device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding
covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads
the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use
std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not
squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses
pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask
themselves; the new test checks both that and that a pinned caller keeps its own mask.

No flag, no environment variable. On one NUMA node (this workstation) only the device setting
happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base =
rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa,
cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs),
-N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise,
e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for
multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:51:38 +02:00
leonarski_fandClaude Opus 5.5 e8584423c6 rugnux: free the merge's GPU engine before model validation
The rotation merge's resident device engine (RotationScaleMerge's GPU
half) lived to the end of the scale-merge step, so model validation
reserved its own engines on a card the merge still held: on the F432
1.30 A set 8.8 GB of the 16.6 GB card was taken when the validation
started. Nothing after the validation reads the engine except the opt-in
finalist ledger's P1 merge, which runs on it beside the validation; so,
unless that merge is running, the engine is released just before the
validation call. Device memory in use when the validation starts:
8.8 GB -> 2.2 GB; the validation window's device-wide peak is now 6.5 GB
(structure factors 2.2 GB, ten rigid-body engines 2.5 GB). p.mtz
md5-identical.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:47:04 +02:00
leonarski_fandClaude Opus 5.5 ed2d557bb7 ModelValidation: a CUDA failure fails the validation instead of restarting it on the CPU
A CUDA error on the validation's GPU path (rigid body, structure factors,
maps) used to start the whole validation again on the CPU - a migration
part way through, which makes how long a run takes, and in principle what
its rigid bodies converge to, depend on a device fault. It now ends the
validation with ok = false and a logged reason ("the GPU failed during the
validation (...)"), which the run reports as MODEL_NOT_VALIDATED. A
validation that did not finish decides nothing, so the reflection files
are those of a run without a model. Where there is no GPU at all, or the
up-front rule sends the work to the CPU, nothing changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:47:04 +02:00
leonarski_fandClaude Opus 5.5 d5fcf2f05b rugnux: model validation's structure factors and maps on the GPU
ModelStructureFactorsGPU computes what compute_model_factors() and
map_from_coefficients() compute on the CPU - F_calc from the model's
density (IT92, Refmac-compatible blur, unblurred as prepare_asu_data()
does) and F_mask from the Refmac bulk-solvent mask, both on the
reflections prepare_asu_data(d_min) lists, in its order; and a map from
ASU coefficients on the grid get_size_for_hkl(coef, 0, 3.0) sizes - on a
device. Made once per cell, group, resolution and model, then evaluated
as often as the coordinates change, so refinement or MR can call it in a
loop. The device is an explicit parameter; every call leaves the calling
thread's current device as it found it.

Pieces:
- ModelDensityGPU: the rigid body's deterministic brick gather, moved
  out of RigidBodyGPU.cu into a component of its own (ModelMaskGPU's
  pattern); the rigid body uses it unchanged. MAX_BRICKS_PER_AXIS 8 ->
  16, so fine grids with high-B atoms (lysozyme at 1.2 A, a 0.9 A P1
  cell) are no longer refused; existing zones are gridded identically.
- One copy of the content is gridded and the symmetry composed in
  reciprocal space (SymmetryComposition), operators applied on the fly;
  the mask is ModelMaskGPU (every image of every atom, islands, shrink).
- Maps: gemmi's get_f_phi_on_grid() in ZYX order on the host (the
  coefficients written are the same), in-place cuFFT c2r, transposed back
  to XYZ on the device. One map at a time, in the engine's buffers.

Decided once, up front, per card, from its TOTAL memory: the engine's
bytes (16 N + cuFFT work + reflections, N the larger of the structure-
factor and map grids) must be at most half the card - the rigid body's
engines take at most a quarter beside it. Otherwise, or where the gather
cannot grid the cell, the CPU path runs, logged with needed vs total.
Anything to a resolution other than d_min (the null's 3.5 A fits) stays
on the CPU, so all replicates and the real model's side of the null are
computed the same way. A CUDA failure takes the existing path: the
validation restarts on the CPU.

Measured, model validation total per run (CPU path -> GPU), 16 GB card:
  F432 215 A cubic, 1.30 A, 500^3 grid: 47.7 -> 15.6 s (two validations;
     14.3 -> 2.4 and 33.4 -> 13.2, the rest of the second is writing the
     three 0.5 GB maps); F_calc + F_mask 5.7 s -> 0.05 s
  C2 1.11 A: 23.9 -> 13.6 s; P3_2 1.55 A: 18.2 -> 10.2 s;
  P2_1 1.25 A: 13.4 -> 6.5 s; F4_132 328 A: 13.0 -> 5.5 s;
  P6_5: 8.8 -> 4.2 s; P4_3 0.97 A: 4.6 -> 2.5 s; P1 0.92 A: 4.2 -> 2.2 s;
  small P1: 3.2 -> 1.3 s; P6_1: 8.8 -> 6.2 s; lysozyme: 1.8 -> 1.4 s.
p.mtz md5-identical on all 13 sets. Against the CPU path: FC within
1e-4 of mean |F|, phases of the strong half within 0.003 deg, maps within
1e-4 (2mFo-DFc) and 7e-4 (mFo-DFc) of their rms; every logged R, CC,
FOM, k_sol and anomalous site list identical at the printed precision,
except where a rigid-body commit sat on an exact R-free tie (0.2155 ->
0.2155) and fell the other way (R-work 0.2127 vs 0.2129). The GPU result
is bit-identical run to run and with -N 8 (maps, map MTZ, placed model).
Peak device memory of the engine: 2.5 GB at 500^3 (process total peaked
at 14.4 GB with what the merge still holds).

Tests: ModelStructureFactorsGPU_MatchesCPU (five groups, 3.5 and 1.5 A:
same reflections, F_calc <= 1e-5 of mean |F|, F_mask 2e-7 rms, repeat
bit-identical), ModelStructureFactorsGPU_MapMatchesCPU (<= 5e-6 of rms).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:38:37 +02:00
leonarski_fandClaude Opus 5.5 6025d51a48 ModelMaskGPU: gemmi's shrink step, so fine rigid-body zones stay on the GPU
The GPU bulk-solvent mask had no shrink step (SolventMasker::shrink(),
set_margin_around() with Refmac's r_shrink = 0.8 A), so every rigid-body
zone whose grid has an offset within 0.8 A was sent to the CPU. That is
not only fine grids: in an oblique setting (P2_1, beta ~141 deg) the
3.5 A zone's lattice-plane spacing is 0.69 A, and on such a set the real
fit and all nine null replicates ran their rigid bodies on the CPU.

The shrink is now two passes over the grid: mark the solvent points with
a macromolecule point among gemmi's near offsets, then turn into solvent
every macromolecule point with such an edge point at any stencil offset
(near or far, gemmi's split at the coarsest grid step). That is gemmi's
rule in both of its branches, and the stencil is built on the host in
double exactly as gemmi builds it. ModelMaskGPU::ShrinkIsNoOp() and
RigidBodyGPUEngine::MaskSupports() are gone; the rigid body no longer
refuses such zones.

Verified (ModelMaskGPU_ShrinkMatchesGemmi): the shrink alone on gemmi's
post-island mask is bit-identical, and the whole GPU mask equals gemmi's
put_mask_on_grid() bit for bit (0 differing points) on the five test
groups at 1.5 A and an oblique P2_1 cell at 3.5 A; repeats are identical.

Measured on the oblique-setting set of the open arm (P2_1, 14k atoms,
1.66 A): model validation 17.2 s -> 5.2 s with the GPU structure factors
of the next commit in place (the first validation's ten rigid bodies,
~12 s on the CPU, now take well under a second). p.mtz md5-identical;
the null moves from +73.1 to +86.1 sigma (the replicates are refined on
the GPU, which agrees with the CPU to rounding), verdict unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:38:37 +02:00
leonarski_fandClaude Opus 5.5 89f52fef95 ModelValidation: remove a map file before writing it again, rather than truncating it
The validation in the model's setting writes its three maps and the map
MTZ over the ones the first validation wrote seconds before, and a
two-pass run writes over the first pass's. XFS (like ext4) flushes a file
that was truncated and rewritten when it is closed, so each overwrite
forced ~155 MB out to the disk on the spot. Measured on /data (XFS on a
hard disk): three 155 MB files rewritten 3.16 s, written fresh 0.62 s,
removed and written again 0.66 s. The contents are the same bytes either
way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:32:10 +02:00
leonarski_fandClaude Opus 5.5 8d6e9d3a44 ModelValidation: use every visible GPU - the null's engines and the second validation
The rigid-body pool put all of its engines on the calling thread's current
card, so a validation used one GPU whatever the machine had. Now, by fixed
rules decided up front and never by momentary free memory:

- RigidBodyGPUPool::Create puts engine i on card (d + i) % count, d being
  the calling thread's card; the pool restores that card afterwards
  (an engine's constructor sets its own) and an engine is released on
  its own card.
- The threads that run the null's replicates are pinned with pin_gpu
  (which also binds them to the card's NUMA node where that is enabled),
  replicate thread t to card (d + 1 + t) % count, and Acquire() hands a
  thread an idle engine on its own card where there is one, any other
  otherwise. With one card this is exactly the previous back-of-the-list
  choice.
- The validation started on the forecast runs on card 1 % count, so with
  two cards or more it is on the other card from the first; the up-front
  memory rule becomes: twice the planned engine bytes within a quarter of
  the cards' total memory taken together.

The engines' kernels are deterministic (no floating-point atomics) and an
engine's result does not depend on which engine it is, so on cards of one
model the numbers are those of one card; what changes is only where the
work runs. On this one-card workstation the multi-card path cannot be
exercised: md5 of p.mtz and the validation outputs are identical to the
base, and the card-count arithmetic was checked by reading only.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:11:12 +02:00
leonarski_fandClaude Opus 5.5 b5980c9ba0 rugnux: start the validation in the model's setting on a forecast, beside the first
Where a model that fits is written on other axes than the data, the files
take its setting and the validation is made a second time on the
relabelled data. That second validation depends on the first only through
the setting and the indexing the first settles, and both are known long
before the first has finished: the change of basis right after the frame
scoring, the indexing the probe prefers before the null. So ValidateAgainstModel
now reports them (ModelFrameForecast, through ModelValidationSchedule::
on_forecast), and the second validation is started there, on the
relabelling AdoptModelFrame and relabel_output would make, applied to a
copy of the merge - beside the first one's null, real fit and maps.

It is kept only where the first decides exactly what was forecast (the
model fits, same change of basis, same indexing; a model asserting the
other enantiomorph is not forecast, as the label is decided last, on the
anomalous map). Until then its log is held (Logger::Buffered, replayed
where the serial run logged it) and its files wait on a gate
(ModelValidationSchedule::write_gate) placed before the first map is
written; otherwise it is released with false and returns unwritten, and
the serial validation runs as before.

GPU memory: two validations at once take twice the rigid-body engines.
The parallel start is decided up front from sizes, never from what is
free: allowed where twice what the first pool asked for (bytes per engine
times the engines wanted) fits a quarter of the card's TOTAL memory, the
share one validation may take. Threads: both validations submit to the
one ParallelFor pool from threads outside it (the second runs on a
std::async thread, as the null's replicates do), so no pass runs inline
on a pool worker and the pool's size bounds the workers.

Measured on the loaded 16-core workstation, TIMING model validation:
8sa8 30.6 -> 19.7 s, 8xtg 19.9 -> 13.9 s, 9ea5 22.1 -> 16.4 s (this and
the previous commit together). p.mtz, p.hkl, p.cif, the three maps,
p_maps.mtz and p_model.cif/pdb md5-identical to the base on 8sa8, 8xtg,
9ea5, 7qis and myob_x10sa; the validation's log lines identical as a set
(the null's replicate lines were already in completion order).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:10:46 +02:00
leonarski_fandClaude Opus 5.5 aabac1541a ModelValidation: score the candidate changes of basis at the same time
Each candidate setting of the model was scored one after another on the
coarse shell (a density, an FFT and an isotropic scale fit, 0.2-0.5 s
each), and a run with a model in another setting scores them in both of
its validations. They share nothing but what they read, so they now run
on ParallelFor, each on its own copy of the model, and are read back in
their own order: the log lines, the ranking and the tie-break (first of
equal R wins) are those of the serial loop.

Measured (16-core workstation, loaded): 8sa8 3 candidates 0.64 s ->
0.22 s; 9ea5 3 candidates 1.47 s -> 0.50 s; 8xtg 1.45 s -> 0.40 s.
p.mtz, the maps, the map MTZ and the placed model md5-identical.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 12:08:39 +02:00
leonarski_fandClaude Opus 5.5 91e63b6f52 rugnux: ask the end wedge once, after the first-pass rescues, not inside every pass
79ec61c14 asked the 'end wedge' scheme inside pick_best, so every pass the
axis-sign, beam-centre and long-axis rescues run carried it, and its count went
into the pass those rescues are gated on. A consecutive wedge indexes part of
the validation frames at the WRONG rotation-axis sign as well: on 7kcn (CBF,
P4_122, the file's axis sign wrong for pass 2) the spread and the wedge gave
0/60, the end wedge 33/60, which lifted the pass over the sign rescue's
majority gate - so the flip, where the spread indexes 60/60, was never asked.
Pass 2 then integrated on the end wedge's lattice (33-49/60 frames): ISa 11.8
-> 4.7, R_meas 7.9 -> 15.3 %, wall 27 -> 43 s. The same mechanism let it
defend a file beam centre the measured one beats on 6p8j.

The end wedge is now asked once, after the rescues, at the geometry they
settled on, and only where nothing has indexed a majority; it is judged against
the standing pass as before. It can no longer displace a pass that indexes a
majority, and the passes the rescues run no longer pay for it.

Battery subset vs e99dd4949 (GPU): 7kcn back to 7c9a6c (p.mtz md5-identical
to 7c9a6c, ISa 11.8, R_meas 7.9 %); 5nw5 still pass (P212121, d 7.10, ISa
8.9); 6p8j first pass back to 7c9a6c's (measured centre, spread 60/60).
p.mtz md5-identical to e99dd4949 on 3mc4, 5ky6, 5reo, 7atg, 8v4o, 9qw8, 9rci,
cuhf2, cytidine, myob_x06da_powder_1/2, myob_x06da_split/sparse,
insu_I_x06da_low_isa, kdp/lcystine/yag/thau inhouse sets, nothing_1 (no
lattice) and six private sets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 09:13:49 +02:00
leonarski_fandClaude Opus 5.5 136b9703f7 Revert "rugnux: a rotation frame the sweep's cell refuses is tried on that cell scaled"
This reverts commit a825fb6cb.

The per-frame cell-scale retry restored the space group of its one
target (a P2 set whose cell grows with dose) but not its data: R_meas
went 26 -> 37 % and d_min 1.54 -> 1.62 A against the pre-part-B run. It
patches the early, low-dose cell of pass 2 frame by frame with a
+-2 % range, 0.1 % step and sqrt(n) margin chosen on that set, and runs
inside AnalyzeIndexing on every set for one difficult one. A late cell
belongs to rc176's far-end check of the early decisions instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 09:13:49 +02:00
leonarski_fandClaude Opus 5.5 dc92359f09 Revert "rugnux: the first pass takes the larger of two domains of one lattice"
This reverts commit c5a13d212.

The second-domain test fixed its target (a low-ISa insulin set) but was
evaluated 1132 times on 212 battery sets and adopted a "second domain"
on 50 of them; three sets that passed went to fail (5j23, 9qw2 and one
private set: F432 -> P432). It never asks whether the other domain is a
different orientation, so near-ties at 54:46 decide it. A narrowed
version (regr2-domain) still costs about 120-200 s battery-wide for one
difficult set. It is parked for rc176, where near-tie domains are to be
carried as two hypotheses and decided late.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 09:13:43 +02:00
leonarski_fandClaude Opus 5.5 c6412819ba Revert "rugnux: mask a shadow that turns with the sample, found in blocks of the pre-scan"
This reverts commit f257d51ea.

The block-wise moving-shadow mask fixed its one target (a half-image
lysozyme set whose pin turns with the sample) but fired on 47 sets of
the full battery: on one small-molecule set the mask covered 51 % of
the detector and SHELXL R1 doubled (0.10 -> 0.20), on another the mask
grew from 8 to 21 %. The pre-scan cost rose by +87 s on the HDF5 sets
and +499 s on the CBF sets (the commit claimed +0.1 s / +1-5 s per set).
A narrowed version (regr2-mask) still costs +25-45 s battery-wide, about
+0.3 s per HDF5 set, for one difficult set - more than the owner's
budget allows. It is parked for rc176, where a moved shadow is to be
found by a far-end check of the early decisions instead.

The documentation paragraph the commit added goes with it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 09:13:34 +02:00
leonarski_fandClaude Opus 5.5 2112102943 ModelScaling: name GEMMI's scaling.hpp, whose target FixedSolventFit follows
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 00:29:54 +02:00
leonarski_f 11f71ff8d3 Merge branch 'perf-modelval' into rc175 2026-10-09 00:29:54 +02:00
leonarski_fandClaude Opus 5.5 f956eb25e8 rugnux: model validation does the same work in less time
--model validation (battery-only for users) was 18% of the battery's time. Every
number it produces is unchanged to the bit (p.mtz, maps, placed model and every
model-validation line of the report md5/diff-identical on 11 open sets); only
when and where the work runs changes:

- The bulk-solvent grid fit (FitModelScale, most of the CPU time) fits each
  solvent pair on a copy of gemmi::Scaling's target that takes
  |Fcalc + k_sol exp(-b_sol s^2) Fmask| once per pair instead of at every
  solver evaluation; same expressions, same types (new test checks a grid
  point against gemmi's own Scaling fit with ==).
- Fcalc density and the solvent mask are made on two threads; the model's
  structure factors beside the GPU engine reservation.
- The indexing probe fits the relabellings concurrently.
- The null's replicates run beside the real model's placement (they start
  from a snapshot of the model as read); one GPU engine per replicate plus
  one for the real fit instead of a cap of 4 (engines are interchangeable
  and deterministic).
- The 2mFo-DFc, mFo-DFc and anomalous maps are made and written
  concurrently; the placed model is written beside the reflection files.
- A rigid-body zone whose solvent-mask grid needs gemmi's shrink is sent to
  the CPU when the engines are reserved (ModelMaskGPU::ShrinkIsNoOp), instead
  of failing on the GPU and validating everything again on the CPU - the
  same CPU result, without the wasted first attempt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
2026-10-09 00:01:14 +02:00