Validated on the 12-set --model comparison against the CPU B+C run
(20260928-1206_e0e3e9_rigidbc-redraw): p.mtz, p.hkl and p_unmerged.mtz
identical on every set, MODEL_FIT, the indexing decisions and the redraw
counts unchanged, no rigid-body commit flipped, MODEL_FIT sigma within -5%
to +3%; two GPU runs identical to the byte on all five output files; the CPU
fallback (JFJOCH_RIGID_BODY_CPU, and CUDA_VISIBLE_DEVICES=) identical to the
CPU build.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
BraggPrediction::Calc and BraggPredictionRot::Calc stopped filling at
max_reflections (kPredictionCapacity = 20000), silently, inside the h/k/l
walk - so a frame that predicted more kept whichever reflections the walk
reached first, while the GPU predictors grow their buffer (GrowCapacity) and
keep them all. A large cell therefore merged a different reflection set on a
CPU-only build (the macOS/Windows viewer path, or CUDA_VISIBLE_DEVICES=) than
on a GPU one.
Both CPU predictors now double the buffer when the walk fills it, so they
return the same set as the GPU; output_limit and TruncateToOutput are
unchanged and remain the one cap on what leaves Calc, on both paths.
Tests: BraggPrediction_GrowsPastStartingCapacity ([portable], CPU only) - a
large cell predicting 28k (still) / 54k (rotation) reflections on one frame
gives exactly what a buffer large enough from the outset gives, and an
output_limit keeps the best-recorded half. BraggPrediction_CPU_GPU_
consistency_large_cell - the same frame on CPU and GPU: 28384/28384 and
53671/53671 matched by hkl, positions within 0.1 px. Both fail on the old
code (count == 20000).
Validation (CPU build): four in-house sets md5-identical to the base. A
dense rotation set from the private arm predicted 72.0M (20000 x 3600
frames) before and 235.9M now, against the GPU's 235.9M; it ingests
218.65M partials, as the GPU does, and now reaches the GPU's cell, space
group and merging statistics, where the capped CPU run settled on a
different lattice. Cost on the CPU: 72.5 GB peak RSS (17.5 GB before),
44.5 min wall on a shared machine (7.2 min before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The r2 regions set aside from a reflection's r2..r3 background ring were
those of every prediction in the +-4 sigma rocking window. On a finely
sliced dense pattern most of them are the tails of reflections recorded on
the frames either side, which fill every ring while the frame shows nothing
there. 9ac2ca677 kept the reflections those rings starved by taking the ring
whole, neighbour pixels included, and relying on the high-side clip.
A prediction now masks the ring only where it puts at least 5% of its flux
on the frame (partiality >= 0.05). A ring still starved by those neighbours
has real flux in it, and the reflection is dropped, as before 9ac2ca677:
taking it whole let the neighbours' wings into the background. The CPU mask
holds two levels (tail, flux); the GPU writes them in two launches so the
flux mark wins, with no atomics.
The count the widened-radius guard reads keeps measuring the pattern's
density over every prediction, tails included, so the guard decides on
the quantity its 1.13% bound was read off. Read on the gated mask it let a
cubic set keep r1=6, and R_meas went 36.9 -> 43.2%. Masked but
unreadable ring pixels are no longer counted as the neighbours' doing.
Fixed radius, same code base, --model, the 0.05 deg / 7200-frame set:
pre-9ac2ca677 9ac2ca677 this
rings starved 88.9% 88.9% 21.5%
partials ingested 7.4M 67M 53M
completeness 17.9% 79.7% 85.1%
ISa 12.8 14.3 14.7
R_meas 6.8% 8.4% 7.6%
R_free 0.139 0.176 0.171
radial misfit 0.11 0.185 0.066
R vs model, common hkl to 0.69 A (61k):
0.133 0.130 0.129
(the R_free rise over pre-9ac2ca677 is composition: 63k -> 319k
reflections to 0.69 A, the added ones weaker.) Gate at 0.2 instead:
0.1% starved but radial misfit 0.35, R_free 0.180.
A crystal whose header-geometry pass indexes a 7x supercell:
9ac2ca677's whole rings raised that pass's I/sigma >= 2 count past the
refined pass's by more than 10%. RefinedPassIsWorse then sent the run
back to the supercell. Now: the true cell and space group as before
9ac2ca677, same ISa, 104M partials in the header pass against 219M.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The equivalent of PutMaskOnGrid() / gemmi SolventMasker(Refmac).put_mask_on_grid() for the GPU
rigid body. Every symmetry image of every atom is masked directly (one block per image, gemmi's
box and !(d2 > r2) rule), which equals gemmi's mask-then-symmetrize-with-min because the operators
are isometries. Islands are removed by a 26-connected periodic union-find (atomicCAS, larger root
under smaller, so each component is rooted at its lowest index), a read-only root pass, a size
count capped at the island limit and a removal pass - no float atomics, bit-identical repeats.
The shrink is not implemented: SetGrid() throws when gemmi's 0.8 A stencil would be non-empty,
which it is not on any rigid-body grid (spacing d/3 >= 1.17 A).
Atoms come in as double fractional coordinates (ModelMaskAtom): with float input, 9fhc at 3.5 A
differed from gemmi at one 24-fold orbit of radius points (|d - r| ~ 2e-6 A); with double input
the mask equals gemmi's on all 8 prototype sets at 3.5 and 6 A, including 6oel, 8t7r, 9hnc, 9fhc.
Time per mask (RTX 5080, Compute incl. islands): 6oel 288^3 4.2 ms, 8t7r 18.4M 2.8 ms,
9fhc 8.0M 1.5 ms, 9hnc 1.9M 0.45 ms.
Union-find credited to Playne & Hawick (2018) at the kernel and in ACKNOWLEDGEMENT.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The dataset-info and per-image plots used matplotlib's tab10 (blue, orange,
...) or the Qt chart theme's defaults, unrelated to the salmon/navy and
indigo/light-blue themes. They now come from ViewerTheme:
- PlotSeriesColor(i): light/dark pairs of one hue each. First a steel navy
#3A6EA5 on light (the heading navy #1F3A5F drawn 3 px wide reads as black)
and the headings' light blue #9DBBE3 on dark; then the caution amber
#B8860B / #E9C46A; then teal, plum, green, rose, slate, brown softened to
the same weight. Contrast on the plot area: 3.2-5.7:1 light, 8-11:1 dark.
The navy/amber pair (Spots + background) is also colour-blind safe.
Coral and red are kept out: hover lines, the marker and alerts use them.
- The old pair measured 2.5:1 (orange on white) and 3.8:1 (blue on the dark
#12142B plot); navy itself would be 1.6:1 there.
- Current-image marker: coral (the sliders' fill) with a ring of plot
background, instead of black/near-white.
- StyleChartAxes: grid lines take the slider groove colour (salmon #F3D9D4 /
indigo #363A66) and the dark theme's axis lines are muted; Qt's light grey
grid outshone the data on dark.
- Run colours are picked again on a theme change (they used to keep the
previous theme's).
Verified headlessly in both themes, including a live theme switch: rendered
line pixels are exactly #3A6EA5/#B8860B and #9DBBE3/#E9C46A.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pre-draw guard keeps a null replicate from STARTING within RigidBodyReachDeg of an orientation
equivalent to the model's (space-group rotations x {identity + twin laws}), but the analytic-Jacobian
rigid body walks further than that reach: on 9yzk one replicate drawn outside the 10.6 deg reach was
walked 23.9 deg onto an equivalent orientation (R-free 0.5578 -> 0.3917), the null SD went
0.016 -> 0.066 and MODEL_FIT flipped ACCEPTED -> REJECTED.
Now where the replicate ENDS is checked too: RefineRigidBody reports the rotation it applied, the
final orientation is that times the drawn one, and a replicate that ended within reach is drawn again
(through the same pre-draw filter) and placed again, capped at NULL_MAX_REDRAWS. The redraws come from
a generator per replicate (NULL_SEED + 1 + i), so they do not depend on the order the concurrent
replicates finish in; the count is logged once per run. The real fit's procedure is unchanged.
Validation, 12 open-arm sets against 20260928-0459_24ae26_r6-pooled: post-placement redraws on 9yzk (1)
and 5epe (1), none elsewhere. 9yzk back to ACCEPTED, +14.15 sigma, null 0.5880 +- 0.0138 (base +12.21,
0.5864 +- 0.0158), p.mtz byte-identical to base. 5epe: MODEL_FIT +41.62 -> +46.39, indexing margin
+40.94 -> +31.97 sigma, same decision. Every other set identical to the previous rigid-bc commit.
-N 1 and the default thread count give identical outputs and logs on 9yzk and 6toc.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
- The background (black-point) slider is hidden by default. It is shown from
View > Background slider, or as soon as the background becomes non-zero
(B + wheel), so an active background is never invisible. The choice is kept
in QSettings ("backgroundSlider") and cleared by "Reset all settings".
- Auto contrast, continuous or the one-shot A key, sets the background back
to zero.
- A dataset-info plot without a run colour kept the default QPen: black, and
near-invisible on the dark theme. This is every plot of a live HTTP dataset,
whose run list is empty. It now takes #1F77B4 on the light theme and
#FF7F0E on the dark one (the spots / background colours of the split plot).
Checked headlessly: the rendered line pixels are exactly those colours on
the #FFFFFF and #12142B plot backgrounds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
B+C of the rigid-body speed-up plan (not bit-exact, by design):
- Fcalc is composed from the transform of ONE copy of the model, gridded without
symmetrization: F(h) = n_cen * sum_S exp(2 pi i h.t) F1(hR), through a per-zone gather table
(half-l index, Friedel flag, phase, Cartesian hR). Only the indices prepare_asu_data() would
give a value are composed (ASU, strict d_min, Nyquist box, model-group absences, not 000),
so the unmatched-observation semantics are unchanged. The mask stays symmetrized and is read
at h directly.
- Translation columns of the Jacobian are exact (the phase factor's derivative), computed with
every evaluation; rotation columns are forward differences needing only one copy gridded and
transformed (3 per Jacobian instead of 6 full evaluations).
- The Jacobian holds the scale and the bulk-solvent mask at the evaluation's; the residuals keep
both exact. The per-evaluation scale re-fit is folded in by Kaufman's variable projection.
- Composition and projection reductions are per-reflection / serial, so the result does not
depend on the thread count.
Tests: RigidBodyP1FcalcMatchesSymmetrized (hkl set identical, <= 4.3e-6 of mean |F|),
RigidBodyTranslationDerivativeIsExact (5-point difference, <= 6.3e-4, the grid sampling),
RigidBodyRotationDerivativeMatchesDifferences (q != 0, anisotropic atoms, cos >= 0.9968),
RigidBodyJacobianMatchesNumericScaleRefit (cos >= 0.998 at 0.07 A; >= 0.983 at 0.35 A, the
residual-weighted term Kaufman drops). Existing rigid-body tests pass unchanged.
Validation on 12 open-arm sets against 20260928-0459_24ae26_r6-pooled: 9hnc's real fit, which
moved 0.000 deg / 0.000 A before, now moves 1.46 deg / 0.65 A and R-free 0.5159 -> 0.4651
(MODEL_FIT_SIGMA 15.4 -> 43.3). STOP finding on 9yzk: a null replicate drawn outside the
redraw guard's 10.6 deg reach was walked 23.9 deg onto an equivalent orientation (R-free
0.5578 -> 0.3917), the null SD went 0.016 -> 0.066 and MODEL_FIT flipped ACCEPTED (12.2 sigma)
-> REJECTED (2.6 sigma). The stronger optimiser's capture radius outgrows the null's redraw
guard; not ready to merge until that is addressed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The null is nine fixed-seed random orientations of the model, each placed by the same rigid body as
the real fit and fitted to the data AS MERGED. Where the indexing probe reindexed the data to the
model's, the model's solution against the merged data is its twin-related orientation - and more
generally the crystal looks the same from every orientation the lattice's rotations (the space
group's, composed with the twin laws) take the model to. A draw within the rigid body's reach of one
of those is walked onto it and is not a sample of the null.
On 5epe (F 2 3, twin law y,x,-z, 24 equivalent orientations) replicate 8 of 9 was drawn 7.9 deg from
one of them; the rigid body moved it 7.9 deg, held-out R-free 0.53 -> 0.20, R-work 0.197 against
0.56-0.58 for the other eight. The null's sd became 0.125 and the crystal's own model read +0.01
sigma, DOES NOT FIT, so the indexing it had decided at +25.8 sigma was refused and the files kept the
merged indexing (R-free 0.52). This happens on every binary whose indexer picks the other hand.
The fix is at the draw, not the score: a draw whose angle to the nearest equivalent orientation is
under the rigid body's reach is drawn again from the same generator, so everywhere no draw falls
there the nine rotations - and every output - are exactly the ones before. The reach is the rotation
about the centroid that moves the model's rms-radius atom by the ladder's first zone (6 A, or d_min
where coarser): 18.7 deg on 5epe, whose other draws sat at 21.7 deg and beyond and did not converge.
Twin laws come from the lattice (ReindexAmbiguityOperators on the data cell and group) regardless of
whether the probe ran. Redraws are capped at 1000 so that a model small enough for the reach to cover
the whole of SO(3) takes its draws as they come instead of looping.
Scoring the real model against the best of both hands instead was not taken: the real fit is
already scored in the hand that wins, and the contamination is in the null, not in the real fit.
Measured, battery commands with --model, rc173 27e154e5d vs this:
5epe 1 redraw; DOES NOT FIT +0.01 -> FITS +55.86 sigma (null 0.5730 +- 0.0072); indexing
decided at +40.4 sigma and reindexed y,x,-z; R-free 0.5246 -> 0.1749 (deposited 0.1671)
9fhc 1 redraw (reach 8.8 deg; the replaced draw had not converged); files byte-identical,
MODEL_FIT_SIGMA +201.27 -> +199.39, verdict unchanged
9hnc, 8sa8, 8xtg, 9bn8: no redraw; p.hkl, p.mtz, p.cif (bar the audit date), p_maps.mtz,
p_model.cif and the model-validation report byte-identical
[ModelValidation] tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The rigid-body target re-grids the model at every evaluation (gemmi's
put_model_density_on_grid with the refmac blur, then the Refmac solvent
mask), and that gridding was ~80% of an evaluation on low-symmetry cells.
The central evaluation ran it on one thread and the six Jacobian columns
on one pool worker each, so most of a 32-thread machine sat idle through
the real fit.
New rugnux/ModelGrid.{h,cpp} reimplements the two gemmi routines so that
they are bit-identical to gemmi on any number of threads:
- Atoms are put on the grid per w plane: each plane is one task and walks
every atom whose box reaches it, in model order, visiting only its own
points (a copy of gemmi's do_use_points_in_box restricted to one plane).
Every grid point therefore receives the same additions in the same
order as in gemmi's serial loop. Per-atom coefficients and radius are
computed once up front, and each atom's density function is copied to a
local so the compiler keeps it in registers.
- Symmetrization runs over orbits: gemmi reduces each orbit into its
lowest index, so the leaders are the points with no lower mate. They
depend only on the grid size and group, so RefineRigidBody finds them
once per zone; each orbit is then reduced by one thread with gemmi's
operands in gemmi's order. Orbits share no points.
- The solvent mask uses the same two passes (setting points to 0 is order
independent anyway); gemmi's own island removal and shrink follow.
vendored gemmi is untouched.
The six Jacobian columns now run on std::async threads, not on the pool,
so each column's gridding can spread over the pool (a pass reached from a
pool worker runs inline). With one thread they run deferred, serially.
Evidence: the grids are memcmp-identical to gemmi's at nt 1/5/32 on 41
deposited models from the battery's PDB cache plus the 8 battery sets
(23 of them with anisotropic atoms; P1 up to F4132, R3/R32, I and C
centring) at 6/4.5/3.5 A; new test ModelValidation_ParallelGriddingMatchesGemmi
covers P1, C2, P212121, I23 and F4132 with iso and aniso atoms. A
standalone RefineRigidBody benchmark (synthetic |F| from the model,
displaced model) gives identical evaluations/angle/shift/coordinates to
the unchanged code: 5lzl 24-33 s -> 12 s real fit on a loaded machine;
null (9 concurrent replicates) unchanged within noise.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
jfjoch_broker: /start takes an optional `tokens` list (any number, all equivalent; writeOnly in the
API). While the current dataset has any, the endpoints that expose it - /statistics/data_collection,
/result/scan, /image_buffer/{start.cbor,image.cbor,image.jpeg,image.tiff}, /preview/plot{,.bin} -
answer 401 unless the request carries `Authorization: Bearer <token>`; /statistics keeps serving
the instrument view and only omits its `measurement` block. Enforcement is one pre-routing hook
over a named path set (the same set carries `bearerAuth` in jfjoch_api.yaml); tokens are compared
in constant time and never read back or logged. The tokens are replaced only by an accepted start,
and atomically with clearing the previous run's status, plots and image buffer, in this order:
clear, swap, import the new settings - so no moment serves the old run under the new tokens or the
new run's name under the old ones. Settings are validated on a copy first so a refused start
changes nothing.
jfjoch_viewer: a Token field (password echo) next to the http/https scheme in Open HTTP Connection,
JUNGFRAUJOCH_HTTP_TOKEN, and D-Bus LoadFile(..., token) / SetHttpToken; the dialog overrides the
others, nothing is persisted. A 401 clears the display, stops following and puts a line on the
status bar - no dialog, since a dataset changing hands is the normal cause.
Web frontend: key button in the top bar (token in sessionStorage, applied to the bearerAuth
operations by the generated client, and to the raw preview fetch), a tokens field in the start
form, and a "token required" hint on the plots. Python client: Configuration(access_token=...)
after regeneration.
Viewer: reference dataset accepts a structure-factor mmCIF (already read by content; the dialog
now offers it) and a model file can be chosen next to it (ProcessConfig::model_path, as
rugnux --model); the job also carries the reference's free-R flags, cell and setting, and the
copied command line states -z / --reference-column / --model. A grid scan keeps its own preferred
dataset-info plot and "Spots + background" means the spot count there. Dark theme: the navy hero
buttons, checked segments, warning texts and chart guide lines follow the theme instead of their
light-theme colours.
Docs: SECURITY.md section 3 is now the implemented scheme; a guided tour with four screenshots
in JFJOCH_VIEWER.md; broker, OpenAPI, Python client and frontend pages mention the token.
Tests: BearerTokens unit test and an HTTP round trip over the real broker HTTP layer (jfjoch_test
now compiles JFJochBrokerHttp.cpp and links httplib).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
macOS turns Shift + a vertical mouse wheel into horizontal scrolling, so
the notch arrives in angleDelta().x() and y() is 0; the contrast step
read only y and did nothing. The wheel handler now takes whichever axis
carries the notch, for zoom, foreground and background alike.
Help > Mouse Shortcuts names Cmd for Ctrl on macOS (Qt maps Command to
Ctrl), Ctrl-click for right click and Fn + arrows for Home/End/Page
Up/Down, and lists B held + wheel, which was missing. The magnifier's
tooltip says how to drive it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RELEASE_CONTENTS gains the macOS .dmg and rugnux .tgz in the artefact
table, the CPU floor (any Apple Silicon Mac), the OS floor (macOS 13), the
CPU-only note in the CUDA/GPU tables, and a macOS section: Apple Silicon
only (Rosetta does not run arm64 code on Intel), drag-to-Applications,
notices inside the bundle, and how to open the not-yet-notarized release
(Open Anyway on macOS 15+, Control-click Open on 13/14, or xattr).
JFJOCH_VIEWER states the platforms and requirements, that the Mac build is
CPU-only, that D-Bus is Linux-only, and adds Building from source on macOS.
RUGNUX_INSTALL adds the macOS archive and the quarantine note - checked: a
browser-downloaded .tgz hands its quarantine flag to everything tar
extracts, Gatekeeper rejects rugnux, and xattr -dr clears it. DEPLOYMENT
points to the pre-built Windows/macOS viewers and the macOS rugnux archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision: a warning is a prompt to check and must catch the real cases (9min) at the
cost of some spurious ones. The physically motivated corrections stay (<|L|> outside its
physical range is not twinning; a single sweep's indexing choice; NO_LATTICE on rotation;
the no-crystal report); thresholds raised only to cut noise come back down:
- SUPERCELL_POSSIBLE warns wherever the class measures and rocks (as before rc173's audit
fix), worded as "check the cell", naming weak ordered intensity of a correct cell and spots
of further lattice domains as the other readings. 9min (rock 4.2%) warns again.
- LATTICE_TRANSLATION warns on every admitted vector (>=75% of the origin); below 90% the
wording names a very strong pseudo-translation as the other reading.
- PSEUDO_TRANSLATION warns on every detection; below a 20% peak it is worded as weak, check.
- SWEEP_GAPS warns where the degraded ranges cover at least 1% of the sweep (the 4 sets of 77
below that had 1-2 frames, 0.4-0.6% of the sweep).
- Powder rings are split between hexagonal-ice positions and the rest (MeasurePowderRings,
report-only fields); ICE_RINGS and POWDER_RINGS warn separately from 5% of the spots, and
ICE_RINGS also where the merge's ice gate found ice.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Report-layer fixes from the warnings audit of the rc173 full battery. Processing is
unchanged (merged MTZ/HKL bit-identical on the sets checked).
- SUPERCELL_POSSIBLE stays a key and a summary line; a warning only where the rocking part
reaches 20% (sub-cells that refine normally rocked at 2.1-16.2%, the accepted doubled cells
at 4.2% and 29.1%). The summary says where further lattice domains may put spots on the
half-integer nodes.
- LATTICE_TRANSLATION only where the Patterson at the vector reaches 90% of the origin
(UNDECLARED_LATTICE_TRANSLATION_PCT, new); the two admitted vectors on the battery read 79%
and 85% on crystals refining in the deposited cell, and are now reported as a very strong
pseudo-translation. PSEUDO_TRANSLATION warns from a 20% peak (xtriage convention; detected
peaks were 8.1% on one set, 24.8-63.2% on the rest).
- <|L|> outside 0.375-0.55 (on the adopted or the pre-search merge) is TWINNING_VERDICT=
NOT_READABLE with no warning, instead of a twin or SYMMETRY_SUSPECT reading.
- NO_LATTICE no longer fires on a rotation run that indexed the sweep with no frame indexed
on its own; INDEXING_RATE is --developer on rotation.
- A rotation run that finds no lattice writes a report (VERDICT= FAILED, NO_LATTICE) before
exiting 1, instead of an input-parameter error and no report (NoLatticeFound).
- Powder and Ice summary lines, ICE_* keys, and a POWDER_RINGS flag from a 0.5 spot fraction.
- SWEEP_GAPS warns only where the degraded ranges cover 10% of the sweep (77 -> 48 sets).
- A single rotation sweep no longer carries an INDEXING_AMBIGUITY warning (one orientation
matrix, consistent hand); INDEXING_AMBIGUITY_OPERATORS and an Indexing choice line instead.
- A DOMAIN more than 10 deg away is a second crystal; the MULTIPLE_LATTICES sum is documented.
- Mosaicity line and docs name XDS's suggested REFLECTING_RANGE_E.S.D. as the comparable
number (0.90x median over 34 in-house sets; per-image SIGMAR runs ~1.35x higher).
- HARMONIC_CONTAMINATION, SCALING_NOT_CONVERGED, POWDER_RINGS in the documented vocabulary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
After the existing mask is complete, a second step takes the pixels it left out that are
significantly dimmer than their ring, joins them through a 6 px bridge and across module gaps of
any width, and adds a piece whole when it holds >= 2000 dim pixels of which >= 200 are deep. The
existing mask is never touched, so a sweep with no such piece keeps its mask bit for bit.
Quick subset tests (battery --only, no model check) against the rc173 a0518abe6 full battery:
152 of 233 masks identical; 10 of 10 controls (including the sets where earlier shadow changes
regressed through marginal decisions) give byte-identical merges. On the 21 sets that gain a
piece the space group never changes, merge outlier rejections fall on 19, R_meas falls and ISa
rises on 17, and the shell-scaled model R improves on 16 of 18; a transmitting arm with an
over-subtracted background strip is recovered (R_meas 13.3 -> 12.0 %, ISa 13.4 -> 15.5).
Known costs: on one set with background bumps at ring radii the low-resolution agreement with
the model falls (CC 0.89 -> 0.85) while its internal statistics improve, and two sets cut
slightly coarser (1.50 -> 1.55 A, 1.72 -> 1.80 A) with a better model R.
Squashed from branch hq-beamstop (cfa080a59..60b6c6822), whose history records the variants
tried and dropped: a straight port of the earlier arm finder moved every mask, a two-shape
construction filled almost a whole sweep through hole filling, and a deep-fraction floor
rejected a real arm that is dim along its whole length.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The local radial background at ring radii (--background-radial) already
existed and is gated per image on the smooth-ice channel of the ice score,
but it was off unless asked for. Systematically negative merged
intensities at ring radii (I/sigma < -4 on 0.4-1.6% of reflections of the
affected open sets) come from the flat annulus mean over-subtracting a
sharp ring. rugnux now runs with =auto unless the flag says otherwise; the
library default (and so the broker) stays off.
Measured on the open arm against rc173 (commit 727bd0d6d as reference):
auto: 8agq (smooth ice, fires) CC_model .9327->.9476, R shell-scaled
.1987->.1795, I/sig<-4 1.55%->1.15%; 8rud R shell-scaled
.2507->.2456, ISa 10.0->13.3; 5reo, 7kcn, 9fhc bit-identical
(gate does not fire); wall time unchanged.
forced on (not adopted): same 8agq gain and 8rud .2414, 9fhc/8qq7
small gains, but 7kcn (no ice) CC_model .9491->.9411, ISa
11.0->7.5 - which is why the gate stays.
ISa falls on 8agq (15.6->13.3) while agreement with the model rises: the
merge-statistics vs external-model trade the settings comment describes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
A new correction surface, fitted after the time x detector surface and
before the goniometer-frame 8x8 grid: log A is a sum of real spherical
harmonics (l = 1..6, 48 terms) of the diffracted-beam direction de-rotated
into the crystal frame. The incident-beam path depends on phi alone and is
in the per-frame scale already.
It runs through ApplyCellSurface unchanged in everything but the update:
the cells are 32 x 64 equal-solid-angle direction bins, and each round the
per-cell sums (ref2, cross, the same damping) become one ridge-regularised
Gauss-Newton step on the coefficients (prior width 0.1/l per degree-l
coefficient) instead of independent per-cell steps. The half-set Fisher-z
gate adopts or refuses it exactly as it does the grids; where it is
refused, the grid after it sees what it saw before.
Why: the folded 8x8 grid (hemispheres share a cell) is the weak basis for
long-wavelength absorption. Offline, held out by unique reflection, this
basis lowered held-out scatter 4-8% on 6 of 11 long-wavelength sets where
no cell grid did, raised model-phased anomalous peaks 0.02-0.2 sigma, and
was neutral on hard-X-ray controls.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Compared with ctruncate (CCP4 9, version 1.17.29) on rugnux's own merged
intensities of the open battery arm, three differences:
- ctruncate gives no amplitude to an intensity below -3.7 sigma (exactly
that bound on every set checked; up to 2498 reflections on one set).
Rugnux turned them into small, confident amplitudes (F/Fc ~0.3 on the
sets whose background is over-subtracted on powder/ice rings). They now
get F = NaN (missing in MTZ, '?' in mmCIF); IMEAN is kept. They are
also left out of the shell mean that sets the Wilson prior.
- The switch to sqrt(I) at I/sigma = 4 left 4-6 sigma amplitudes 2-3%
above ctruncate's on every set (1.019-1.028). The posterior now applies
up to 20 sigma (emulated: 0.995-1.000).
- A shell whose mean intensity is not positive gave a prior at the 1e-10
clamp and amplitudes of ~0 (one set's outer shell); it now takes the
nearest lower-resolution shell's mean.
The remaining gap (weak amplitudes 2-5% below ctruncate's in the outer
shells) is ctruncate's anisotropy-corrected prior; not attempted. An
offline R-free ablation put the whole ctruncate conversion at -0.0006
median (15/18 sets better) and dropping its rejected negatives at -0.0009
mean (-0.009 on the worst set).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
MULTIPLE_LATTICES and EXTRA_LATTICE_INTENSITY_PCT now count DOMAIN, TWIN_DOMAIN and SEGMENTED
lattices only. A FOREIGN lattice (an unrelated cell) is still listed but no longer warned about: it
is as often a wrong main lattice or tNCS as a second crystal (one of the four such sets in the
battery census is a tNCS crystal). On the census this names 25 of 200 rotation sets instead of 29.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
-z (now also --reference; --reference-mtz still works) takes an SF-mmCIF
(e.g. a deposited -sf.cif, gzipped or not) as well as an MTZ. The format is
recognised by content: a file starting with "MTZ " is an MTZ, anything else
is parsed as CIF. The first merged reflection block with the requested
column (or, by default, one the auto choice accepts) is converted to a
gemmi::Mtz in memory with GEMMI's CifToMtz and then read by the unchanged
MTZ loader, so the in-memory reference is exactly what the MTZ path yields.
Unmerged (_diffrn_refln) and anomalous-only blocks are passed over; the log
names the block used.
The R-free set comes from _refln.status (f -> FreeR_flag 0, o -> 1: the
CCP4 convention the loader already reads) and is preferred to
_refln.pdbx_r_free_flag, whose convention varies by program; a status
column with no 'f' is ignored and pdbx_r_free_flag is used instead.
Checked on two open-arm sets in the deposited setting (one with F_meas_au
only, one with intensity_meas): reference loaded with the deposited cell and
group, the inherited free set agrees with the deposited status 'f' on every
common reflection, and the merge correlates at CC 0.99 with the deposition.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
RUGNUX_ADVANCED (new section "The setting the files are written in", -z and
--model paragraphs), RUGNUX_REPORT (SETTING_* keys, REFERENCE_MISMATCH, model
change-of-basis keys), CPU_DATA_ANALYSIS_DECISIONS / _INTEGRATION, CHANGELOG
entries under rc.173.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The magnifier moves from a helper window into its own dock below the
inspector. It has three fixed zoom levels (x64 and x32 with the pixel
values written on the pixels, x10 without) instead of wheel zoom, and
follows the cursor only while Shift is held over the diffraction image.
Meanwhile the diffraction view draws a frame around the area it covers,
just outside that area so it stays visible at low zoom without hiding the
pixels; releasing Shift (seen by the application-wide key filter, so
wherever the focus is), moving without it or leaving the image hides it.
A "Pop out" button moves the magnifier into a separate window and back.
Qt's own dock floating stays disabled: a floated dock is placed
off-screen on WSLg. Whether it was popped out, and the window geometry,
persist across sessions. The inspector's image statistics become a
collapsible section so a small screen can give the space to the
magnifier. kLayoutVersion is bumped for the new dock.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Advisory only: the class is measured (<I/sigma> >= 0.5) and its rocking part is at least 2% of the
lattice's, three standard errors clear. The warning asks the user to process both settings - the
run's cell and the reported doubled cell with -C - and compare them in refinement. Of 105 battery
rotation sets it names 8: both crystals whose accepted cell is the doubled one, one pseudo-translation
whose doubled description is an accepted alternative, and five correct sub-cells with weak ordered
half-integer intensity.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The report-only supercell probe now reaches the results report: SUPERCELL_CLASS (parity of the
primitive indices), its occupancy and Bragg-like (rocking) part against the lattice's own reflections,
its <I/sigma>, and SUPERCELL_DOUBLED_CELL, the Niggli-reduced cell to give with -C. No decision is
taken on it: correct cells whose half-integer class is diffuse, and cells whose depositor kept the
sub-cell, cannot be told from a real doubling by the data alone. On a crystal with a real doubled
axis the reported cell matched the deposited one, and processing on it with -C brought R_free
against the deposited model from 0.60 to 0.29.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The between-pass geometry walk kept a round only when the realised held-out residual fell by more
than its noise, and started only on a fit move of a trust-region step or more. The residual is
dominated by low-resolution reflections, where a distance and the compensating cell scale move every
spot alike, and its centroids are taken inside a disc centred on the prediction, so it barely sees a
distance error that costs the high-resolution shells. On 8pqd the canonical pass ran at 96.456 mm; the
fit asked for 95.878 mm (0.6 %, less than a step, so no walk). Forced, that round read the residual
only 0.74 sigma lower but put 70.4 % of the validation spots on the lattice against 58.1 %.
Now a round is kept when either the residual falls beyond its noise (HeldOutResidualFell) or the
validation evidence prefers it (ValidationEvidencePrefers, z = 3.29). A move of less than a step is
first tried as two index-only probes on the validation frames (fit's geometry vs the one in hand) and
pays for a re-integrated round only where the fit's geometry scores higher; walks started by a large
move run as before.
8pqd: 96.456 -> 95.878 mm, d_min 1.374 -> 1.306 A, R_free .218 -> .206, REFMAC R_free .209 -> .202,
Wilson B 34.4 -> 30.7 (pool had 1.325 A / .209 / .203). 6vww 7n0i 7ris 8egn 8xtf 9qw8 9w3y 6qaj
6cdl 9i0a myob_x06da_split lyso_x06da_half_image unchanged (probes say the geometry in hand stands);
probe cost 1-10 s per run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
R_MEAS is what users set beside XDS/AIMLESS, so its meaning does not change; the merge-weighted value
from hq-cure-merge is reported beside it (R_MEAS_WEIGHTED, REFRES_R_MEAS_WEIGHTED). The per-hand
table keeps the ordinary statistic.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The delta-CC1/2 disposition keeps a weak stretch in the merge (downgraded)
wherever removing it would not raise CC1/2, and the merge then carries each
of its observations at the small 1/sigma^2 its scaled-up counting error gives
it. R_meas counted those observations at full weight, so the frames that add
next to nothing to the intensities set the number. hq-pool battery: 8xtf kept
115 weak frames (scale 0.1-0.2 of the run's) with the same CC1/2, <I/sigma>
and CC_model per shell as rc173 with 84 frames rejected, and R_meas was
1.3-1.5x higher in every shell; 9w3y (33 deg rejected -> 0, every model
metric better) and lyso_x10sa_strong read the
same way. The disposition itself is not segmentation-dependent: conviction
is on the batch grid, not the ledger ranges, and those sets' dispositions
changed because the corrected data changed.
R_meas now weights each observation by its merge weight v = 1/sigma^2 under
the error model (corrected_sigma on the host, ModelSigma on the GPU, with the
error model of the last MergeAccum), normalised per reflection to Kish's
effective count, so equal sigmas give the ordinary formula
(WeightedRmeasTerms). Where the proportional term dominates - strong
reflections - frames are weighted alike, as in the merge: 8xtf's lowest shell
reads 15.0%, the rc173 run with 84 frames rejected 15.2%. The per-hand table
is weighted the same way. R_MEAS_UNWEIGHTED / REFRES_R_MEAS_UNWEIGHTED keep
the XDS/AIMLESS convention and are what to set beside XDS; the battery scorer
records them. MULTIPLICITY stays a count.
Effect (weighted / unweighted): lyso_x06da_ref 0.0454 / 0.0479, 8xtf
0.225 / 0.907, insu_I_x06da_ref REFRES 0.072 / 0.232. GPU and CPU paths agree
to the last printed digit on lyso_x06da_ref.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
ComputeSmoothGWindow leaves a frame below MIN_CREDIBLE_SCALE_RATIO of the
median out of every window, and DropCollapsedScales drops it after the loop,
but RunScalingLoop still counted its step. Such a frame has lost its vote in
its own references and keeps moving, so the loop was stopped as "unsettled".
On 8xtf (hq-pool battery) the first-pass partials loop stopped after 16 rounds
at rms dlogG 1.8e-01 (base 9.5e-04); with the frame left out of the step it
settles in 23 rounds (8.0e-04), and every other loop of that run that had
stopped unsettled (pass 2 at 1.6e-03) now settles too. Every loop of that
sweep that stopped unsettled was one that went on to drop a frame. The final
8xtf merge is bit-identical; the false SCALING_NOT_CONVERGED warning goes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The pre-scan's spot-shape estimate (spot_width::EstimateBandwidth) becomes the run's bandwidth where it
is significant (z > 3) and neither the file nor --bandwidth states one; an explicit value, including
--bandwidth 0, still wins. Everything that already reads GetBandwidthFWHM() - prediction, partiality,
the profile's radial width, the stencil growth, scaling - takes it from there. Below z 3 the bandwidth
stays unset, so monochromatic runs are bit-identical (12 mono sets checked on p.hkl md5).
The width tiers stop as soon as r80 settles, often at 15 frames and a few hundred spots, which is
too few for the slope. The pre-scan now keeps measuring spot shapes for the bandwidth alone until the
pool holds 1000 spots or the sample ends. Those extra frames feed neither r80 nor the powder
measurement, so r1..r3, powder and the spot-resolution quantiles are unchanged. Over the 68-set
battery plus the known mono controls (47 sets re-probed) no monochromatic set crosses z 3 (highest
lyso_x06da_5keV 2.0, insu 5-6 keV 1.7); the one new positive is 9z44 (ALS 8.2.1, multilayer
beamline, 0.38% at z 3.1 from 382 spots instead of z 1.0 from 94).
Two hidden b > 0 mode switches go, so the bandwidth acts continuously:
- the background clip default dropped from 4 to 3 sigma when a bandwidth was set. With the measured
bandwidth, clip 3 vs 4 on the MicroMAX pink sets: REFRES R_meas 0.0507/0.0507 (lyso),
0.0842/0.0844 (thau), REFRES ISa 28.83/28.77, 14.12/14.12 - no bandwidth-specific effect.
- the "--integration-stencil has no effect without --bandwidth" message, which is decided before the
pre-scan and would now be wrong.
The one set where clip 3 helps (9z44, 7 A: R_meas 0.252 -> 0.201) gains the same at zero bandwidth, so
that is a property of the clip, not of the beam.
Effect of the measured bandwidth (stencil 0, clip 4), against hq-integ 9b6736dbd at the same
resolution range (REFRES; open sets rerun with --report-resolution at the base d_min): R_meas up on
all six consumers - MicroMAX pink lyso 0.0499 -> 0.0507, thau 0.0824 -> 0.0844, 9q41 0.250 -> 0.265,
8u0i 0.0796 -> 0.0816, 9sl0 0.0758 -> 0.0773, 9z44 0.252 -> 0.255; CC1/2 unchanged to 1e-4; CC_MODEL
overall +0.001/+0.0004/+0.0005/-0.0025 (9q41/8u0i/9sl0/9z44); no space-group change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The pass-1 post-refinement fits the rotation scale k only on the frames the stored angles still
track, and a rate error is exactly what stops them tracking the rest: on a sweep whose stage turned
~3 % slow the fit read 0.979 over 94 deg, failed its leave-a-fifth-out test and was thrown away,
leaving half the frames unscaled.
The pass no longer decides. Between the passes, at the pass-2 detector geometry, the lattice is
indexed (index-only probe) under the stored angles and under the fitted k and scored on the
validation frames of the whole sweep: share of the spots on the lattice beyond the wrong-spindle
null. k is adopted only where it scores higher by more than the binomial noise of the two
(ValidationEvidencePrefers - the test the beam-centre arms already used, now one function); the
run then integrates and post-refines at k (post-refine-only probe), fits again on top of it and
repeats until the next k no longer scores better (WalkRotationScale). The stored angles are the
first hypothesis. Measured: 1.000 25.1 %, 0.97874 58.9 %, 0.97041 90.0 %, 0.97006 90.7 % (not
significant) -> 0.97041 adopted.
Probes restore the experiment, the pass-2 geometry, pass-1 mosaicity and the beam-centre-search
flag; a probe opens no beam-centre search. Forced pass-1 results get their axis scaled; the header
revert drops the scale. GONIOMETER_ROTATION_SCALE reports the adopted k (SUSPECT = adopted). The
leave-a-fifth-out figure stays in the log only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The pair rule drops the improbable (higher) member of a discordant pair. If the
lower member lost its core to saturation or the mask, its profile estimate of
what remained may be low, and a genuinely strong reflection would be replaced
by the clipped value - the failure Aimless guards against.
Integration now marks a reflection whose signal disk was not fully readable
(Reflection::clipped, from the engines' existing full-disk test); the flag is
carried through the partials to the fulls (OR over an event's partials, CPU and
GPU combine alike), and the Wilson test never counts a clipped observation as
a probable witness against a larger one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The help text claimed 0.99 is right for every synchrotron in the corpus.
Bending-magnet, superbend and wiggler sources are plausibly 0.8-0.95, no file
format read states the value reliably, and it cannot be fitted from a single
dataset (it correlates with spindle-symmetric absorption). Say what the number
is (degree p; XDS FRACTION_OF_POLARIZATION = (1+p)/2) and that a known
beamline value can be passed. Documentation only; the default stays 0.99.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
