The stills distance is found from the data, so the --detector-distance pins go. Ice-ring detection,
the polarisation degree on one Rayonix set, the geometry pre-pass switch-off and the Alvra/one AGIPD
gains are no longer given: each was neutral or better without it. Kept: --wavelength where the file
states none or a different one, the polarisation on the mixed-form set (its beam-stop mask depends on
it) and the gain of one AGIPD set whose frames carry an uncorrected baseline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Refitting the module placement inside the tilt decision did not work on half the
frames: a module that half barely reaches is not placed at all and goes back to where
it was given, which made both arms worse and, on a modular detector, hid a real tilt
behind the placement's loss. The decision's arms and the tilted fit on every frame
now hold the modules where the module placement put them, so what is applied is what
the held-out test judged. Re-placing the modules with the tilt on every frame was
also tried end to end and gave a worse merge than holding them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
AareDB's main is still at 8ece2c2, the commit the setup was copied from, so the structure/ files
match it line for line (bar the black stage, mmCIF input and the anomalous layer). What differed was
around them:
- molstar pinned to 5.11.0, AareDB's exact pin, instead of 5.13.0: the renderer's defaults
(marking, outlines, occlusion, trackball) come with the version, so the same version draws the same.
- Only the first load frames the molecule: a newer result keeps the user's camera and clip slab, as
AareDB keeps them across result switches.
- Load states as in AareDB: "loading model..." centred on the stage until the model is in, then
"computing density maps..." in the panel with the map controls disabled until the maps are ready.
- Map rows labelled "2Fo-Fc" and "Fo-Fc (+/-)", AareDB's names.
- createPlugin.ts carries the credit line for the setup; its comment's directory path is ours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A pedestal-subtracted JUNGFRAU background is mostly empty pixels; the variance per unit mean leaves
out the zeros and read 6.7 keV per photon on a 12 keV run in keV, 2.4 on a 4.6 keV one, so such
files needed --gain. MeasureGain now also histograms the sampled pixel values and takes the gain from
the single-photon peak (background_gain::PhotonPeak): the highest bin past the zero peak's descent,
standing 5 sigma above the valley before it, narrow (below half height a quarter of its value away),
and repeated by a two-photon peak near twice its value. Where there is no such pair the variance per
unit mean decides as before. Float frames are read for the measurement at a sixteenth of their unit,
so a photon of a few keV is resolved rather than rounded to whole keV.
Measured: float32 keV JUNGFRAU files 11.96 (photon energy 12.04 keV) and 4.53 (4.57 keV); the int16
photon copies show no peak past the first bin and keep the 0.55 variance answer, i.e. stay in
photons; the SACLA MPCCD and PAL Rayonix files show humps, not photon peaks, and measure as before.
A gain the file states or --gain gives still wins. Rotation is untouched (the pre-scan's CCD gain
uses only the variance estimator).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The owner's verdict on the screenshots: aaredb (gitea.psi.ch/mx/aaredb, the PSI MX group's own
structure viewer) has it right, so its Mol* setup is taken over, files copied from
frontend/src/components/Structure at 8ece2c2 into frontend/src/components/structure:
- createPlugin.ts: headless DefaultPluginSpec plugin, manual camera reset; background black (owner)
instead of aaredb's 0x0d1a26.
- MolstarCanvas.tsx: fresh canvas per mount, one-tick StrictMode deferral, resize, WebGL context-loss
recovery, the Coot control profile wired.
- cootControls.ts / cootTrackball.ts: Coot mouse language - left drag rotates (Coot 0.9.8 trackball),
right drag zooms, middle drag pans, middle click recentres on a residue, scroll clips the slab,
shift+scroll zooms (camera sync and residue pick left out: no split view here).
- loadModel.ts: cartoon coloured by chain from a palette clear of the map colours, polymer sticks and
waters built hidden, ligands as ball-and-stick; display modes cartoon / sticks / both and a waters
switch; reads mmCIF too, and gives the atom coordinates for carving.
- loadMaps.ts: isosurface wireframe meshes at ABSOLUTE levels from the full cell's mean and RMS,
2Fo-Fc steel blue, Fo-Fc green/red at +/-sigma, not pickable; plus the anomalous map as a magenta
+sigma mesh in the same style.
- densityCrop.ts: the density worker's crop to the model's box + 6 A, the 4.5 A carve of the 2Fo-Fc map
around the atoms (never a difference map) and the CCP4 encoding, applied to the broker's one-cell map.
The panel and defaults follow aaredb's StructureView: map rows with switch and sigma slider (2Fo-Fc 1.0,
0.5-4 by 0.1; Fo-Fc 3.0, 1-8 by 0.5; anomalous 4.0, 1-8 by 0.5), debounced sigma updates, model mode
buttons, a controls legend. This replaces the camera-following 10 A map sphere: aaredb's equivalent is
the carved map with the scroll-driven clip slab.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 5763b619fe4ec9999ac790960a5eb033f511c546)
The owner's choice: the viewer opens Coot-style, the maps within a 10 A sphere around the centre of view;
radius 0 on the slider still draws the whole cell.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 63121f35dd008e606f0fbb336b8ee0c330cde370)
Model validation already made the anomalous difference map's coefficients ((F(+) - F(-)) on the model
phases less 90 degrees, where the merge kept the Bijvoet pairs) to read the anomalous sites, then threw
them away. They now go into the map coefficients as ANOM/PHANOM (phenix.maps' labels), missing on the
reflections without a pair, so _maps.mtz opens as an anomalous map in Coot. Nothing in scaling, merging
or the validation's decisions changes; a merge without the Bijvoet split writes the twelve columns as before.
The broker serves the map as /rugnux/map.ccp4?type=anom (404 without Bijvoet pairs), and
rugnux_live_status.anomalous_map says it is there. The Mol* viewer loads it only then, as a magenta mesh
at +4 sigma with its own level slider and switch.
Tests: ANOM/PHANOM appear with Bijvoet pairs (as many as the reported pairs) and not without; the served
anomalous map of a point peaks at it; the auth test covers type=anom. End to end (myoglobin 2M replay):
anomalous_map true, the served map equals gemmi's FFT of ANOM/PHANOM in the written _maps.mtz to 3.4e-6 rms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit c4d6ad74c706181cbb183cfcaa2481653cd7cad6)
/rugnux/map.ccp4 now serves one unit cell of map, centred on the model, instead of the model's box plus
5 A; along an axis where the model reaches further than the cell (a predicted model with long termini
in a small cell), the model plus 5 A instead, so that all of it has density. The Mol* viewer opens on
the whole cell and the model (map radius 0); a radius still draws the maps only around the centre of
view, Coot-style.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 1c1e1d96359f112d6820d972005e9f7eca4ebd50)
The placed-model END block lost everything but the atoms, so the writer's _model.cif had no sequence
references, authors, software or secondary structure, and gemmi relabelled the subchains. The block now
also carries label_asym_id, label_entity_id and label_seq_id per atom and the input model's metadata as the
mmCIF categories gemmi writes without coordinates (metadata_mmcif), which the writer parses with gemmi and
puts back. The writer's _model.cif is now the same text as the one rugnux writes itself and as
/rugnux/model.cif serves. Older blocks without the new fields are read as before.
Test: a predicted myoglobin model through END -> CBOR -> writer gives byte-identical _model.cif/_model.pdb
to WritePlacedModel and to PlacedModelMmcif, with its _struct_ref, _entity_poly_seq, _audit_author,
_software and _struct_conf.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 29d752e6a107e3b8bc71ad6e9af0d81064966537)
When the run placed a model, the Rugnux live page shows it in a Mol* viewer (molstar 5.13.0, pinned,
loaded lazily in its own chunk): cartoon, ligands and ions as ball-and-stick, the 2mFo-DFc map as a
blue mesh (1.0 sigma) and the mFo-DFc map in green/red (+/-3 sigma), each with a contour-level slider
and a show/hide switch. As in Coot, the maps are drawn within a sphere (10 A by default, adjustable,
0 = the whole box) around the centre of view, which follows the camera. Sigma levels are relative to
the whole-cell statistics the broker writes into the map headers. Reloaded for every new result;
the plugin is disposed with the component.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 08295be10c1d7a76bc7ecdf5df46fd7a3e1febac)
A rugnux live run that placed a model keeps it, with a copy of its map coefficients, beside the live
status (RugnuxStructure), for as long as /rugnux/status keeps the run. Made in memory on the first
request and cached, never written to disk:
- GET /rugnux/model.cif: the placed model as mmCIF, labelled as WritePlacedModel labels _model.cif
(PlacedModelMmcif, the labelling factored out of WritePlacedModel; the file output is unchanged).
- GET /rugnux/map.ccp4?type=2fofc|fofc: the 2mFo-DFc / mFo-DFc map as a CCP4 map, FFT of the
FWT/PHWT or DELFWT/PHDELWT coefficients (get_f_phi_on_grid + MapFromFPhi, sampled as ModelValidation
samples), cut to the box around the model plus 5 A, with the whole cell's mean and RMS in the header.
- 404 without a placed model or maps, 400 for an unknown map type; both routes protected by the dataset
bearer token like /rugnux/status. rugnux_live_status gets placed_model.
C++ model, python client docs, TypeScript client and redoc regenerated.
Tests: a point's coefficients give a map peaking at the atom in a cropped box; the mmCIF; the structure
is cleared at the next start; the auth test covers the new routes. End to end (myoglobin 2M replay with
an AlphaFold model): the maps equal gemmi's FFT of the written _maps.mtz to 2.5e-6 rms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 33124c3288ba112def3f51c0d91f5206886f3495)
The stills geometry bundle (RefineGlobalGeometry) can now refine the detector tilt
together with the beam, the distance, the cell and the modules. With the tilt free the
beam block is the direct beam, from which the PONI is worked out inside the residual: in
PONI terms a tilt moves the whole pattern as a moved beam does (D tan(tilt)/pixel), more
than the beam's 8 px box holds, while the direct beam does not move with it and the tilt
is left with the keystone. The two tilt-like rows of ModuleGauge are gone: within a
module a tilt is a gradient no rigid move makes, and the rows cost held-out accuracy.
The shift, turn and outward gauges stay.
The tilt is a hypothesis (DecideStillsTilt): the strong frames of the last geometry
refinement are split in two halves, each fitted with the tilt held and refined (modules
placed in both where the run placed them), each fit scored on the other half, and the
tilt is kept where the refined fits predict the held-out spots better. No threshold.
It is decided after the stills beam-centre trial and its re-refinement, on frames
indexed at the settled centre: a centre error on frames indexed at it reads as a tilt.
--rot1/--rot2 or --no-refine-tilt hold the tilt on stills.
The stills beam-centre trial now compares direct beams. Its hypotheses (the file's
centre, the background walk, the point-symmetry capture) are taken as the point the
beam lands on and each trial puts the PONI where its distance and the run's tilt land
the direct beam there; the walk, fitted through the geometry's tilt, returns a PONI and
is converted. The pre-scan beam-centre check log compares the walk's direct beam, not
its PONI, with the file's direct beam. Without a tilt nothing changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The absolute distance scan ran at one centre on the 8 strongest of 96 sampled frames, at the run's indexing
tolerance. Three ways that found nothing, and the run then stayed at the file's value or, on an XTC, at the
100 mm placeholder:
- a sparse I-centred lattice whose file states 180 mm for a detector at ~135 mm indexed at no distance from
the background centre (7 px off) - only from the file's centre. Every centre hypothesis is now scanned;
- a 92 A cell at ~199 mm indexed on no centre 3 px off, nor at a distance half a scan step off. The scan's
frames are now indexed at tolerance 0.2 instead of 0.1; the scan only has to find the peak, and every
later step (the peaks on 32 frames, the fine step, the centre grids) counts at the run's own tolerance;
- on a run with a hit rate of a few percent the 8 strongest of 96 frames held no crystal. Where nothing
indexes at any distance at the run's tolerance, the sample is made ten times larger and the search run
again, the scan on 16 frames.
With these an LCLS XTC (no distance stated) with two crystal forms finds 198.93 mm against the 199.0 it was
given (R-free unpinned 0.236, pinned 0.238); a Rayonix CXI stating 839 mm finds 106.23 mm (0.206 vs 0.204
pinned at 105 mm); another XTC finds 193.41 against 193.43 (0.202 vs 0.200); the 180 mm file still finds
136.15 mm. Sets whose stated distance is right keep it, with unchanged results. The fallback costs ~30 s,
only where it fires.
RefineStillsGeometry also reports the slope of the indexed spots' signed excitation error against d*^2 at the
geometry it starts from: positive where the distance (and the cell) is short, negative where long. Reported
only.
The usage message and docs no longer describe --detector-distance as the joint fit's starting value (it is
held since 45abde17a), and say that an XTC with -C has its distance searched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The Cheetah CXI per-frame mask (N x H x W uint16, one gzip chunk per frame) was read with
ReadVector<uint32_t> under hdf5_mutex: the inflate and HDF5's element-wise type conversion serialised
every reading thread and held an AGIPD run to ~40 frames/s. It now goes through ReadStored, as the
data does: only the chunk lookup is under the lock; the read, inflate and unshuffle are not, and the
stored uint16/uint32 values are widened to uint32 exactly as HDF5 converted them.
The direct path is taken where the mask is plain unsigned integers of the frame's size, stored alike
in every file (checked once, in Place); a chunk stored with a filter skipped is read by HDF5 as for
the data, and any other mask keeps the locked HDF5 read. Images and pixel mask are bit-identical to
before (hash of every pixel of 2000 images on two AGIPD runs and one MPCCD run).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The lower bound counts each saturated pixel at the saturation limit, so it needs one. An observation
overloaded only by pixels unreadable on this frame (a per-frame bad-pixel flag, data value 0) was kept
as a box sum with those pixels counted as zero, which bounds nothing; it now stays overloaded and is
dropped, per observation.
With the stuck pixels masked by the pre-scan, the run-level gate (kept overloads must be enriched
coarser than 4 A) has nothing left to catch: without it every still set gives the gated result
(clipped CCD and Rayonix sets identical to the fourth decimal, photon-counting sets unchanged), so it
is removed with Reflection::overload_kept. A neighbour-brightness test per saturated pixel was tried
and not kept: after the mask it only dropped 9-20% of the Rayonix overloads and worsened R_split on
one set (0.245 -> 0.278).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A pixel stuck at the saturation limit was never masked: the defective-pixel scan counted only the
error value as "invalid on most frames", and a saturated pixel merely as lit, which the hot rule
does not judge on stills. A Rayonix stills set has 3,627 pixels at 65535 on every frame, at high
resolution; each flagged every reflection whose disk it fell in as overloaded, and the spot finder
found them on every frame. The saturation limit now counts with the error value (CPU and CUDA), so
such a pixel is masked like one holding the error value.
On that set the indexed fraction rises from 10% to 33% (and 7% to 30% on the second lattice of the
same run), R_split 0.120 -> 0.106 and the unrefined R-free against the deposition -0.001 / -0.024.
Rotation merges checked bit-identical (three in-house sets with overloads).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Twelve public XFEL serial sets (SACLA, PAL-XFEL, LCLS, European XFEL, SwissFEL), each read cold from
disk on the workstation, with time to map, processing rate and R_free of the deposited model against
our merge and against the deposited structure factors under one unrefined protocol: 6 wins, 4 draws,
2 losses (more than 0.01 apart counts).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills distance grid spanned 60-160 % of the file's value, so where it looked depended on the value it
was checking: a Cheetah file stating 839 mm for a detector at about 106 mm left every grid point empty and
the run had to be given the distance. The search is now global first and local after, like the beam
centre's capture and walk:
- a scan over 30-3000 mm in 2.5 % steps (the peak's width) at one centre that does not depend on the file
(the background's, else the capture's), on the 8 frames with most spots;
- its three strongest local maxima, a step either side, at every centre hypothesis on 32 frames; then the
fine +-2 % step on the whole sample, which decides against the file's value with the rule as before.
This replaces the relative grid run at every centre, and costs fewer trials than it did.
The geometry sample is found without the spot finder's resolution limits, which are d-spacings evaluated
at the distance being checked, and spots piled up at one radius (ice and other powder rings) are left out
by their radius in mm. Where the decision moves the distance by more than 5 % the sample is found again
there and the fine step asked again.
A --detector-distance on stills is a hard pin: not searched, and held by the joint fit (the cell alone is
refined). Without one, the file's value is a hypothesis against the search's best, kept unless that puts
materially more spots on the lattice. An XTC that states no distance now runs with -C: the search decides
it, and the first refinement, which would start from a placeholder, is skipped.
The joint distance+cell fit indexed its bundle at the geometry it started from, so its end point carried
the start (0.2 % apart from two starts a percent apart). Where it moves the distance by more than 0.3 %
(and no modules are being placed) the bundle's frames are indexed again at the fitted geometry and fitted
again, up to three times, until the distance moves by less than 0.05 %.
From 839, 50 and 300 mm a Rayonix set ends at 106.24, 106.23 and 106.22 mm; an XTC set run with no distance
ends at 193.41 mm against the 193.43 it is otherwise given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Saturation clips the strongest spots, and those are at low resolution. On a Rayonix stills set the
saturated pixels sat almost entirely past 4 A (0.002% of the reflections coarser than 4 A against
0.59% overall) - flagged or hot pixels, not clipped spots - and keeping those observations as lower
bounds put the saturation limit into high-resolution intensities: R_split 0.12 -> 0.93, CC1/2 0.99
-> 0.33. The run now keeps overloads only where their share coarser than 4 A is at least 1.5x their
overall share; otherwise they are flagged and dropped as before, with a log line saying so.
On clipped CCD stills the share coarser than 4 A is 2.6-5.9x the overall one and nothing changes
(R_split and R_free identical); on the Rayonix set R_split returns to 0.12. A pragmatic gate until
saturation is told from bad pixels per spot. Rotation never keeps overloads and is unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Stills keep overloads as lower bounds on CPU/CUDA but not yet on Metal, an accepted platform difference
to be fixed. On a rotation experiment both engines flag the overloads as before, so the bit-identity
comparison stays meaningful. The engine reads the rotation flag only for that decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The Metal kernel and host code are restored to their panel-s5 content, so neither the background clip
that follows a handed-over radial curve nor the count of saturated pixels in pass A runs on the Metal
path. This is an accepted, temporary platform difference, to be closed once the kernels can be checked
on a Mac: on macOS a still's background clip stays the plain one (the radial correction itself still
uses the handed-over curve, as on the CPU), and an overloaded observation kept as a lower bound counts
its saturated pixels as 0 instead of at the saturation limit.
One host-side change is kept, with no effect on any result: the radial sums are accumulated on the GPU
only where no curve was handed over. StartPassA now sizes those sums for that case alone, so the old
host code would write past them on every still with a curve.
The shared structs need nothing on the Metal side: BraggPassA.h is compiled into the kernel source, the
box-sum kernel zero-initialises BraggBoxSums (n_saturated = 0) and the host zero-initialises
BraggPassAFrame (n_curve = 0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The one conflict, tests/BraggIntegrationEngineCPUTest.cpp, where both branches add cases at the same
place: integ-pal's side is taken and stills-sat's test diff applied on top, which keeps all four cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The joint distance+cell fit freed the distance even where the user had given it. A user-given distance
is a hard pin: the cell alone absorbs any error. The file's distance and DecideStillsGeometry's stay
free as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills bundle held whichever of the two it trusted - a given or stated distance, with the cell
refined to absorb any error, or, after DecideStillsGeometry had rejected the file's distance, the
user's cell anchored and the distance refined from it. The two are degenerate only near the beam: the
radius of a spot goes as D tan(2 theta) and the cell scales sin(theta), so the strong frames'
high-angle spots fix both. A distance given 1.4% short (the user's cell was the depositors' starting
cell, about a percent off the crystal) was absorbed into the cell, mispredicted the spots past 2.5 A
and made the merged intensities there negative.
Both are now free (refine_distance, no cell restraint) whatever their source; a --detector-distance
on stills with a reference cell is the fit's starting value, no longer a pin. The flags that chose
between the two regimes (stills_distance_nominal, stills_distance_from_cell_) are gone.
On a Rayonix stills set given 105 mm the fit converges to 106.1 mm from 105 and from 103 mm, and the
deposited model's R_free against the merge drops 0.354 -> 0.283 (all frames); a stills set whose
distance is right holds it (193.43 -> 193.42 mm, R_free unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
A manifest entry may carry max_images, applied when no tier is given; both
EuXFEL entries get 2,000 (and a smoke tier of the same size), which keeps
their full-set run time out of the development batteries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The batteries now score the deposited model against a merge one way: model_check.py, REFMAC's first
cycle before any shift (bulk solvent and anisotropic scale by default), the depositor's free set,
d_min_used = max(ours, deposited), compared against the deposited data through the same protocol.
- model_check.py: the restrained-refinement protocol is removed (refine, refined_shared,
rename_misfit_ligands, REFINE_CYCLES, the refined parameter, --no-refine). Both data sets are now
scored on their common reflections only, so a low-resolution or anisotropic cut of either applies
to both (measured on three still sets: <= 0.001 change).
- battery.py / paper_plots.py: refmac_refined_* and refmac_refined_ratio are gone; refmac_rfree_ratio
is the headline; paper_plots draws the first-cycle pair only (--rfree removed).
- still_battery.py: dimple is replaced by model_check.check on the same p.mtz. R_free rugnux is
rfree_depflags, the target is rfree_depdata (no longer min(dimple, published)); the published
R-free stays as an information column. The dimple cache, free-flag writer, deposited-MTZ writer
and the reference verb are removed. This also retires dimple's freerflag COMPLETE renumbering,
which scored R-free on depositor work reflections whenever our merge had hkl the deposition
lacks (model_check carries the depositor's flags per hkl and marks ours-only hkl as work, and
with the common-hkl restriction there are none). `report RUN --out DIR` rescores an older run's
merges without rerunning rugnux.
Refinement cost about 6.4 h per full rotation battery; the 0-cycle check takes seconds per set.
Verified: 9V3D still merge reproduces the 0-cycle common-hkl numbers (0.2461 ours, 0.2367 deposited).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The validation battery of 61def00a1 (20261022-0925_29000d_still-integ-pal-r1clip) shows the r1 rule
buys nothing once the ring clip follows the radial curve: lcls222 at r1 4 (4,6,8) 0.1893 against
0.1856 at r1 6; pal86 0.1998 against 0.1997. The radius scans that motivated it were taken with the
flat clip, whose beam-side cut the smaller ring partly avoided. r1 stays 6 on stills.
The empty apertures' corrected column again clips flat. With the clip following the curve there,
lcls242_s read the correction as a clear gain (z = 11) and the run took it, but dimple on 40,000
frames says it costs 0.013 (0.2461 on against 0.2335 off); judged on the flat clip, as in 07a43a13e,
the decision is off there and unchanged on every other set. The integrator still follows the curve
in its clip wherever the correction is on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The stills merge dropped every observation with a saturated pixel. On a still the saturated
observations are the fully recorded ones of the strongest low-resolution reflections, so what
survived were their partial observations, whose partiality correction then set the merged
intensity: on clipped CCD stills the merged low-resolution intensities came out at 0.4-0.6x of
the deposited ones, a ramp no Wilson B absorbs, invisible to CC1/2 and R_split.
The pixels are really clipped (flat-topped plateaus a few per mille wide, a pile-up in the
pixel-value histogram from the per-pixel ADC limit), so the auto ceiling is right; the drop was
the defect. A still now keeps the observation: the box sum with every saturated pixel counted at
the saturation limit, unflagged and marked clipped, counted in BraggIntegrationCounts and
reported as OVERLOADED_REFLECTIONS_KEPT. Rotation data (IsRotationIndexing) are unchanged and
keep the flag for the rocking-event drop.
Measured on a clipped-CCD stills subset (dimple R-free, deposited free set): dropped 0.247,
lower bound 0.230 (same as switching saturation off). A Kabsch fit to the unsaturated wings read
lower than the lower bound (0.237), and inflating the lower bound's sigma lost (0.255).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two stills-only changes to the integration stencil.
1. Where the whole-frame radial curve is in use (the correction on), the ring's high-side clip
follows it: a ring pixel is clipped against the threshold plus the curve's rise from the
reflection's radius to its own (BraggIntegrationEngine::RadialClipOffset, rounded to whole counts
so CPU, CUDA and Metal keep the same pixels). On a background falling steeply away from the beam
the flat mean + 4 sqrt(mean) threshold cut the beam-side half of the ring as outliers, so the
background read low and reflections near the beam stop read 3-150x too strong. The empty-aperture
test's corrected column clips the same way, so the per-run correction decision sees it.
Measured with r1 6 (dimple R_free, battery arguments, all frames):
lcls222 0.196 -> 0.1856, pal86 0.216 -> 0.1997, sacla236 0.2559 -> 0.2559, alvra213_int16 0.1986 -> 0.1986.
2. The signal disk is taken from the lattice: it must not reach into the next node's spot. The
closest two nodes can come on the detector is the shortest reciprocal-lattice vector the centring
allows, s = F_px tan(2 asin(lambda / 2 d_max)), and a spot reaches 2 r80 (measured in the
pre-scan); r1 = the largest of 4, 5, 6 px not above s - 2 r80, never below 4, never above the
disk the run starts with (6). The empty apertures are measured for each candidate disk and the
ring is chosen on the chosen disk's tally. Pixel detectors keep 6 (sacla236 s 22 px, alvra213
20 px); the binned CCDs with a long axis shrink to 4 (lcls222 5.4 px, pal86 6.7 px), which the
radius scans there measured as the optimum; pal90 (s 21 px) and lcls242_s (15.5 px) keep 6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Owner's rule: a set is scored against min(R-free of the deposited data through our dimple protocol,
published R-free). The report shows both values, a "target" column naming which one was lower, and
the difference and win/draw/loss verdict against that target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The published R-free comes from the depositors' own refinement (mostly phenix.refine), while the
battery refines with dimple's 8 REFMAC cycles, so the two are not like-for-like: on LCLS 222 (8TDQ)
the deposited intensities reach 0.1918 under dimple against a published 0.2012.
Every set now carries "R-free deposition (dimple)": the deposition's -sf.cif (intensity_meas, else
the mean of the anomalous pair, else F_meas_au squared) refined with the same refine() as our merge -
same deposited model, the deposition's free set, cut at its resolution. It is computed once per entry
and resolution and kept in the PDB cache (<pdb>-dimple-<d_cut>/reference.json); `run` and `report`
fill in a missing one, `reference` computes or lists them up front. The report gains the difference
(ours minus the deposition's) and a quality verdict (win below -0.01, draw within 0.01, loss
otherwise); the published R-free stays as an informational column.
Reproduces the hand-computed references: 8TDQ 0.1918, 9V3D 0.2220, 31JT 0.1945.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
One Ceres fit keeps only ~5-6 of 32 cores busy whatever it is given, and on a loaded machine (load
28-36, JF16M int16, 4000 strong frames) the five side-by-side fits measured 10.9 s with every thread
each, 13.8 s with half, 14.6 s with a fifth, against 18.9 s one after the other (bb7c9c). The
thread split is dropped: each fit gets gr_settings' threads, as before the fits ran side by side.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The background-ring log line now gives how many standard errors the correction lowers the chosen
ring's mean square by, over the apertures measured both ways. Diagnostic only; the decision is
unchanged. Full pal90: z = -2.7 (correction off), where dimple prefers it on (0.2124 vs 0.2183).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
Without a resolution limit the outer d*^2 shells of the width measurement lie beyond the diffraction
limit, where the summed intensity is noise or even negative; one such shell made the whole measurement
fail, and 5 of 11 still-battery sets (sacla237, pal86, pal90, lcls242_s, lcls242_t) fell back to the
per-crystal window fit. A shell now enters the fit only where its summed intensity is measured to 10 %
(sum I / sigma >= 10), and at least three shells are needed.
Weighting the fit by the shells' signal instead was tried and rejected: it lets the strong low-resolution
shells set the width alone, and on the MicroMAX pink set the dimple R-free went 0.2442 -> 0.2748.
The matrix subsets, which are cut at their resolution, are unchanged within 0.0010 dimple R-free
(SwissFEL 0.2453, -C/-S 0.2513, mono 0.3188, pink 0.2442, SACLA 243 0.2455, SACLA 237 0.2327,
LCLS 222 0.2335, PAL 86 0.3628).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
The first stills battery of e8145d8 lost on one Rayonix set (lcls242_s, dimple R_free 0.2302 ->
0.2323): there the whole-frame radial correction hurts. On 40,000 of its frames: battery default
0.2462, ring 12 with the correction 0.2444, ring 12 without it 0.2336, ring 14 with it 0.2467. Its
beam is strongly polarised and the azimuthally averaged ring background does not describe the local
background.
So the empty apertures now also carry the corrected difference: the frame's ring background
(RingBackground, factored out of the engine) averaged over the ring pixels minus over the disk pixels
- what the integrator's kernel takes off a reflection - is subtracted from each disk-minus-ring
difference. The ring is still chosen on the plain differences, so its reach does not rest on the
radial model; where the correction was left to auto it is then kept only if it makes that ring predict
the disk better. Decisions on 15-60 pre-scan frames:
pal86 ring 10, correction on (240.6 -> 221.1) lcls222 ring 10, on (6.34e4 -> 3.41e4)
lcls242_s ring 12, off (277.2 -> 285.8) alvra213_int16 ring 14, on (0.01424 -> 0.01422)
Choosing ring and correction jointly instead picked ring 14 with the correction on lcls222, where dimple
reads 0.204 against 0.196 at ring 10; a median instead of the mean square picked ring 14 with the
correction on lcls242_s, the worst arm there. The sequential rule is the one that agrees with dimple on
every set measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
GetConsensusUnitCell averaged every indexed frame within 5% of the -C cell. The given-cell
population filter runs after it, so the MTZ, report and model-validation cell was a mixture of
both forms. Frames whose retained outcome is empty now do not count, and the consensus cell is
read again once the population filter has cleared the other form's frames.
Two-form P21 serial data (LCLS):
- T run: 93.85/93.66/49.98/92.00 -> 95.21/94.50/50.42/91.09 (kept population ~95.5/94.1/50.3/91.2)
- S run: 92.38/93.29/49.59/93.26 -> 91.90/93.06/49.45/93.65 (deposited 92.04/93.02/49.01)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
