RUGNUX_ADVANCED: --bandwidth defaults to the pre-scan's measured value where
the file states none; --background-clip is 4 whatever the bandwidth (the
broadband 3-sigma default is gone); --integration-stencil no longer needs
--bandwidth; --polarization text follows the usage (XDS fraction, undulator
default); --beam-center-check describes the both-centres merge arbitration;
the --model placement runs on the GPU when present, CPU otherwise.
RUGNUX_INSTALL: the rigid-body placement is on the GPU, with automatic CPU
fallback; CUDA_VISIBLE_DEVICES= runs a CUDA build on the CPU, as the GPU
scaling fail-fast message says.
JFJOCH_VIEWER: the magnifier follows the cursor only while Shift is held;
the inspector folds away on a narrow window; B + wheel shows the background
slider.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Two decisions in the rotation two-pass flipped whenever the integration
changed, because each compared a raw number with no account of its noise.
The quality guard (RefinedPassIsWorse) counted reflections at I/sigma >= 2 in
each pass's search merge. Across two lattices that count is not like for like:
a header pass on a 7x supercell merges seven times the reflections, six in
seven empty, and their noise reached the strong count - it out-counted the
refined pass on the crystal's own lattice and the run went back to the
supercell, while that merge held more reflections at I/sigma <= -2 than at
>= 2. The guard now reads the signal as the strong count less the count at
<= -2 (the merge's own noise tail), and its dilution clause scales the
reflection count of the smaller lattice by the integer index of one lattice in
the other, so the same lattice (index 1) compares exactly as before. Measured
on 37 sets: no guard decision changes except the supercell case, which now
keeps the refined pass and the crystal's own lattice both in the configuration
that flipped and at HEAD; the axial-row arm still fires on the crystal it was
written for.
The post-refinement commit gate required each residual family to fall by any
amount. The positional family is what sees the detector geometry a commit hands
the next pass, and its sign was a coin toss where it did not move: on 9yzk it
read +7.1 % in one build and -0.14 % (0.1 of its paired noise) in the next, and
the second committed a geometry the positions had no opinion about (R-free
0.382 -> 0.412, radial misfit 0.30 -> 0.39). The gate now asks the positional
residual to fall by more than its paired standard error (both geometries read on
the same held-out reflections) and the excitation residual not to rise by more
than its own. On the 37-set validation this moves five fit decisions: 9yzk
(R-free 0.412 -> 0.394, radial misfit 0.39 -> 0.29, ISa 9.7 -> 7.8), two
canonical-pass probes whose output is unchanged, and one pre-pass commit on 9pbb
whose excitation moved +0.1 of its noise (ISa 14.77 -> 14.54, R-free 0.2353 ->
0.2357, inside the noise). The in-house myob, cytc, lyso and sparse sets are
md5-identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Calibration (--mode calibration): RebinAndRefit gave each worker one
AzimuthalIntegrationProfile that AzIntEngineCPU::Run clears on every
image, so the re-binned profile held only the last image each worker
read, added in completion order - on a 1800-frame LaB6 run the re-binned
fit changed with -N (197 / 181 / 205 ring points at rms ~7 px). Each
fixed block of images is now summed in image order into its own slot
(ParallelBlocks) and the slots added in block order, so the profile is
the sum over every image and the same at any thread count (rms 0.93 px,
identical at -N 1, 7 and 32). On the LaB6 runs with a sane header the
first pass still wins and the .poni is unchanged; with a header 100 px
off the re-binned fit is now adopted (218 vs 140 ring points). New test
Rugnux_CalibrationRebinsEveryImage spreads the rings over six frames
and fails on the old code (241 vs 141 points at -N 1 vs -N 4).
Host memory: out-of-memory now ends with "Processing failed: out of
host memory (...) - this data set needs more RAM than is available" and
exit 1. Tested with ulimit -v and an LD_PRELOAD allocator that refuses
allocations from a chosen phase (CPU and GPU builds, image loop through
merging and writing, calibration mode). Paths that crashed instead:
- PostIndexingRefinement ran candidate blocks on bare std::threads, so
a bad_alloc there called std::terminate (seen: "terminate called
recursively", SIGABRT); now std::async futures.
- FFTW aborts on its own failed allocation (CK(p) in kernel/alloc.c,
also inside buffered transforms: BeamCenterFFTCPU, FFTIndexerCPU);
rugnux now defines fftwf_assertion_failed to report it and exit 1.
- libjpeg's default error_exit calls exit() from the diagnostic-JPEG
thread ("Insufficient memory (case 12)"); WriteJPEGToMem now
longjmps back and throws (bad_alloc for JERR_OUT_OF_MEMORY).
- WorkerPool construction that fails to start a thread destroyed
joinable threads (terminate); it now joins them and rethrows.
Also: length_error counts as a fatal resource error, pinned host
allocation failure is MemAllocFailed, and the GPU scaling fail-fast
message says the CPU path needs a lot of host memory.
Docs: very large cells and what running out of host memory looks like
(including the OOM killer) in RUGNUX_INSTALL.md, pointer in RUGNUX.md.
Validation: myob, cytc, lyso, sparse md5-identical to 82e6583e0 on
both GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Validated on the 12-set --model comparison against the CPU B+C run
(20260928-1206_e0e3e9_rigidbc-redraw): p.mtz, p.hkl and p_unmerged.mtz
identical on every set, MODEL_FIT, the indexing decisions and the redraw
counts unchanged, no rigid-body commit flipped, MODEL_FIT sigma within -5%
to +3%; two GPU runs identical to the byte on all five output files; the CPU
fallback (JFJOCH_RIGID_BODY_CPU, and CUDA_VISIBLE_DEVICES=) identical to the
CPU build.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
BraggPrediction::Calc and BraggPredictionRot::Calc stopped filling at
max_reflections (kPredictionCapacity = 20000), silently, inside the h/k/l
walk - so a frame that predicted more kept whichever reflections the walk
reached first, while the GPU predictors grow their buffer (GrowCapacity) and
keep them all. A large cell therefore merged a different reflection set on a
CPU-only build (the macOS/Windows viewer path, or CUDA_VISIBLE_DEVICES=) than
on a GPU one.
Both CPU predictors now double the buffer when the walk fills it, so they
return the same set as the GPU; output_limit and TruncateToOutput are
unchanged and remain the one cap on what leaves Calc, on both paths.
Tests: BraggPrediction_GrowsPastStartingCapacity ([portable], CPU only) - a
large cell predicting 28k (still) / 54k (rotation) reflections on one frame
gives exactly what a buffer large enough from the outset gives, and an
output_limit keeps the best-recorded half. BraggPrediction_CPU_GPU_
consistency_large_cell - the same frame on CPU and GPU: 28384/28384 and
53671/53671 matched by hkl, positions within 0.1 px. Both fail on the old
code (count == 20000).
Validation (CPU build): four in-house sets md5-identical to the base. A
dense rotation set from the private arm predicted 72.0M (20000 x 3600
frames) before and 235.9M now, against the GPU's 235.9M; it ingests
218.65M partials, as the GPU does, and now reaches the GPU's cell, space
group and merging statistics, where the capped CPU run settled on a
different lattice. Cost on the CPU: 72.5 GB peak RSS (17.5 GB before),
44.5 min wall on a shared machine (7.2 min before).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The r2 regions set aside from a reflection's r2..r3 background ring were
those of every prediction in the +-4 sigma rocking window. On a finely
sliced dense pattern most of them are the tails of reflections recorded on
the frames either side, which fill every ring while the frame shows nothing
there. 9ac2ca677 kept the reflections those rings starved by taking the ring
whole, neighbour pixels included, and relying on the high-side clip.
A prediction now masks the ring only where it puts at least 5% of its flux
on the frame (partiality >= 0.05). A ring still starved by those neighbours
has real flux in it, and the reflection is dropped, as before 9ac2ca677:
taking it whole let the neighbours' wings into the background. The CPU mask
holds two levels (tail, flux); the GPU writes them in two launches so the
flux mark wins, with no atomics.
The count the widened-radius guard reads keeps measuring the pattern's
density over every prediction, tails included, so the guard decides on
the quantity its 1.13% bound was read off. Read on the gated mask it let a
cubic set keep r1=6, and R_meas went 36.9 -> 43.2%. Masked but
unreadable ring pixels are no longer counted as the neighbours' doing.
Fixed radius, same code base, --model, the 0.05 deg / 7200-frame set:
pre-9ac2ca677 9ac2ca677 this
rings starved 88.9% 88.9% 21.5%
partials ingested 7.4M 67M 53M
completeness 17.9% 79.7% 85.1%
ISa 12.8 14.3 14.7
R_meas 6.8% 8.4% 7.6%
R_free 0.139 0.176 0.171
radial misfit 0.11 0.185 0.066
R vs model, common hkl to 0.69 A (61k):
0.133 0.130 0.129
(the R_free rise over pre-9ac2ca677 is composition: 63k -> 319k
reflections to 0.69 A, the added ones weaker.) Gate at 0.2 instead:
0.1% starved but radial misfit 0.35, R_free 0.180.
A crystal whose header-geometry pass indexes a 7x supercell:
9ac2ca677's whole rings raised that pass's I/sigma >= 2 count past the
refined pass's by more than 10%. RefinedPassIsWorse then sent the run
back to the supercell. Now: the true cell and space group as before
9ac2ca677, same ISa, 104M partials in the header pass against 219M.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The equivalent of PutMaskOnGrid() / gemmi SolventMasker(Refmac).put_mask_on_grid() for the GPU
rigid body. Every symmetry image of every atom is masked directly (one block per image, gemmi's
box and !(d2 > r2) rule), which equals gemmi's mask-then-symmetrize-with-min because the operators
are isometries. Islands are removed by a 26-connected periodic union-find (atomicCAS, larger root
under smaller, so each component is rooted at its lowest index), a read-only root pass, a size
count capped at the island limit and a removal pass - no float atomics, bit-identical repeats.
The shrink is not implemented: SetGrid() throws when gemmi's 0.8 A stencil would be non-empty,
which it is not on any rigid-body grid (spacing d/3 >= 1.17 A).
Atoms come in as double fractional coordinates (ModelMaskAtom): with float input, 9fhc at 3.5 A
differed from gemmi at one 24-fold orbit of radius points (|d - r| ~ 2e-6 A); with double input
the mask equals gemmi's on all 8 prototype sets at 3.5 and 6 A, including 6oel, 8t7r, 9hnc, 9fhc.
Time per mask (RTX 5080, Compute incl. islands): 6oel 288^3 4.2 ms, 8t7r 18.4M 2.8 ms,
9fhc 8.0M 1.5 ms, 9hnc 1.9M 0.45 ms.
Union-find credited to Playne & Hawick (2018) at the kernel and in ACKNOWLEDGEMENT.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The dataset-info and per-image plots used matplotlib's tab10 (blue, orange,
...) or the Qt chart theme's defaults, unrelated to the salmon/navy and
indigo/light-blue themes. They now come from ViewerTheme:
- PlotSeriesColor(i): light/dark pairs of one hue each. First a steel navy
#3A6EA5 on light (the heading navy #1F3A5F drawn 3 px wide reads as black)
and the headings' light blue #9DBBE3 on dark; then the caution amber
#B8860B / #E9C46A; then teal, plum, green, rose, slate, brown softened to
the same weight. Contrast on the plot area: 3.2-5.7:1 light, 8-11:1 dark.
The navy/amber pair (Spots + background) is also colour-blind safe.
Coral and red are kept out: hover lines, the marker and alerts use them.
- The old pair measured 2.5:1 (orange on white) and 3.8:1 (blue on the dark
#12142B plot); navy itself would be 1.6:1 there.
- Current-image marker: coral (the sliders' fill) with a ring of plot
background, instead of black/near-white.
- StyleChartAxes: grid lines take the slider groove colour (salmon #F3D9D4 /
indigo #363A66) and the dark theme's axis lines are muted; Qt's light grey
grid outshone the data on dark.
- Run colours are picked again on a theme change (they used to keep the
previous theme's).
Verified headlessly in both themes, including a live theme switch: rendered
line pixels are exactly #3A6EA5/#B8860B and #9DBBE3/#E9C46A.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pre-draw guard keeps a null replicate from STARTING within RigidBodyReachDeg of an orientation
equivalent to the model's (space-group rotations x {identity + twin laws}), but the analytic-Jacobian
rigid body walks further than that reach: on 9yzk one replicate drawn outside the 10.6 deg reach was
walked 23.9 deg onto an equivalent orientation (R-free 0.5578 -> 0.3917), the null SD went
0.016 -> 0.066 and MODEL_FIT flipped ACCEPTED -> REJECTED.
Now where the replicate ENDS is checked too: RefineRigidBody reports the rotation it applied, the
final orientation is that times the drawn one, and a replicate that ended within reach is drawn again
(through the same pre-draw filter) and placed again, capped at NULL_MAX_REDRAWS. The redraws come from
a generator per replicate (NULL_SEED + 1 + i), so they do not depend on the order the concurrent
replicates finish in; the count is logged once per run. The real fit's procedure is unchanged.
Validation, 12 open-arm sets against 20260928-0459_24ae26_r6-pooled: post-placement redraws on 9yzk (1)
and 5epe (1), none elsewhere. 9yzk back to ACCEPTED, +14.15 sigma, null 0.5880 +- 0.0138 (base +12.21,
0.5864 +- 0.0158), p.mtz byte-identical to base. 5epe: MODEL_FIT +41.62 -> +46.39, indexing margin
+40.94 -> +31.97 sigma, same decision. Every other set identical to the previous rigid-bc commit.
-N 1 and the default thread count give identical outputs and logs on 9yzk and 6toc.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
- The background (black-point) slider is hidden by default. It is shown from
View > Background slider, or as soon as the background becomes non-zero
(B + wheel), so an active background is never invisible. The choice is kept
in QSettings ("backgroundSlider") and cleared by "Reset all settings".
- Auto contrast, continuous or the one-shot A key, sets the background back
to zero.
- A dataset-info plot without a run colour kept the default QPen: black, and
near-invisible on the dark theme. This is every plot of a live HTTP dataset,
whose run list is empty. It now takes #1F77B4 on the light theme and
#FF7F0E on the dark one (the spots / background colours of the split plot).
Checked headlessly: the rendered line pixels are exactly those colours on
the #FFFFFF and #12142B plot backgrounds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
B+C of the rigid-body speed-up plan (not bit-exact, by design):
- Fcalc is composed from the transform of ONE copy of the model, gridded without
symmetrization: F(h) = n_cen * sum_S exp(2 pi i h.t) F1(hR), through a per-zone gather table
(half-l index, Friedel flag, phase, Cartesian hR). Only the indices prepare_asu_data() would
give a value are composed (ASU, strict d_min, Nyquist box, model-group absences, not 000),
so the unmatched-observation semantics are unchanged. The mask stays symmetrized and is read
at h directly.
- Translation columns of the Jacobian are exact (the phase factor's derivative), computed with
every evaluation; rotation columns are forward differences needing only one copy gridded and
transformed (3 per Jacobian instead of 6 full evaluations).
- The Jacobian holds the scale and the bulk-solvent mask at the evaluation's; the residuals keep
both exact. The per-evaluation scale re-fit is folded in by Kaufman's variable projection.
- Composition and projection reductions are per-reflection / serial, so the result does not
depend on the thread count.
Tests: RigidBodyP1FcalcMatchesSymmetrized (hkl set identical, <= 4.3e-6 of mean |F|),
RigidBodyTranslationDerivativeIsExact (5-point difference, <= 6.3e-4, the grid sampling),
RigidBodyRotationDerivativeMatchesDifferences (q != 0, anisotropic atoms, cos >= 0.9968),
RigidBodyJacobianMatchesNumericScaleRefit (cos >= 0.998 at 0.07 A; >= 0.983 at 0.35 A, the
residual-weighted term Kaufman drops). Existing rigid-body tests pass unchanged.
Validation on 12 open-arm sets against 20260928-0459_24ae26_r6-pooled: 9hnc's real fit, which
moved 0.000 deg / 0.000 A before, now moves 1.46 deg / 0.65 A and R-free 0.5159 -> 0.4651
(MODEL_FIT_SIGMA 15.4 -> 43.3). STOP finding on 9yzk: a null replicate drawn outside the
redraw guard's 10.6 deg reach was walked 23.9 deg onto an equivalent orientation (R-free
0.5578 -> 0.3917), the null SD went 0.016 -> 0.066 and MODEL_FIT flipped ACCEPTED (12.2 sigma)
-> REJECTED (2.6 sigma). The stronger optimiser's capture radius outgrows the null's redraw
guard; not ready to merge until that is addressed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The null is nine fixed-seed random orientations of the model, each placed by the same rigid body as
the real fit and fitted to the data AS MERGED. Where the indexing probe reindexed the data to the
model's, the model's solution against the merged data is its twin-related orientation - and more
generally the crystal looks the same from every orientation the lattice's rotations (the space
group's, composed with the twin laws) take the model to. A draw within the rigid body's reach of one
of those is walked onto it and is not a sample of the null.
On 5epe (F 2 3, twin law y,x,-z, 24 equivalent orientations) replicate 8 of 9 was drawn 7.9 deg from
one of them; the rigid body moved it 7.9 deg, held-out R-free 0.53 -> 0.20, R-work 0.197 against
0.56-0.58 for the other eight. The null's sd became 0.125 and the crystal's own model read +0.01
sigma, DOES NOT FIT, so the indexing it had decided at +25.8 sigma was refused and the files kept the
merged indexing (R-free 0.52). This happens on every binary whose indexer picks the other hand.
The fix is at the draw, not the score: a draw whose angle to the nearest equivalent orientation is
under the rigid body's reach is drawn again from the same generator, so everywhere no draw falls
there the nine rotations - and every output - are exactly the ones before. The reach is the rotation
about the centroid that moves the model's rms-radius atom by the ladder's first zone (6 A, or d_min
where coarser): 18.7 deg on 5epe, whose other draws sat at 21.7 deg and beyond and did not converge.
Twin laws come from the lattice (ReindexAmbiguityOperators on the data cell and group) regardless of
whether the probe ran. Redraws are capped at 1000 so that a model small enough for the reach to cover
the whole of SO(3) takes its draws as they come instead of looping.
Scoring the real model against the best of both hands instead was not taken: the real fit is
already scored in the hand that wins, and the contamination is in the null, not in the real fit.
Measured, battery commands with --model, rc173 27e154e5d vs this:
5epe 1 redraw; DOES NOT FIT +0.01 -> FITS +55.86 sigma (null 0.5730 +- 0.0072); indexing
decided at +40.4 sigma and reindexed y,x,-z; R-free 0.5246 -> 0.1749 (deposited 0.1671)
9fhc 1 redraw (reach 8.8 deg; the replaced draw had not converged); files byte-identical,
MODEL_FIT_SIGMA +201.27 -> +199.39, verdict unchanged
9hnc, 8sa8, 8xtg, 9bn8: no redraw; p.hkl, p.mtz, p.cif (bar the audit date), p_maps.mtz,
p_model.cif and the model-validation report byte-identical
[ModelValidation] tests pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The rigid-body target re-grids the model at every evaluation (gemmi's
put_model_density_on_grid with the refmac blur, then the Refmac solvent
mask), and that gridding was ~80% of an evaluation on low-symmetry cells.
The central evaluation ran it on one thread and the six Jacobian columns
on one pool worker each, so most of a 32-thread machine sat idle through
the real fit.
New rugnux/ModelGrid.{h,cpp} reimplements the two gemmi routines so that
they are bit-identical to gemmi on any number of threads:
- Atoms are put on the grid per w plane: each plane is one task and walks
every atom whose box reaches it, in model order, visiting only its own
points (a copy of gemmi's do_use_points_in_box restricted to one plane).
Every grid point therefore receives the same additions in the same
order as in gemmi's serial loop. Per-atom coefficients and radius are
computed once up front, and each atom's density function is copied to a
local so the compiler keeps it in registers.
- Symmetrization runs over orbits: gemmi reduces each orbit into its
lowest index, so the leaders are the points with no lower mate. They
depend only on the grid size and group, so RefineRigidBody finds them
once per zone; each orbit is then reduced by one thread with gemmi's
operands in gemmi's order. Orbits share no points.
- The solvent mask uses the same two passes (setting points to 0 is order
independent anyway); gemmi's own island removal and shrink follow.
vendored gemmi is untouched.
The six Jacobian columns now run on std::async threads, not on the pool,
so each column's gridding can spread over the pool (a pass reached from a
pool worker runs inline). With one thread they run deferred, serially.
Evidence: the grids are memcmp-identical to gemmi's at nt 1/5/32 on 41
deposited models from the battery's PDB cache plus the 8 battery sets
(23 of them with anisotropic atoms; P1 up to F4132, R3/R32, I and C
centring) at 6/4.5/3.5 A; new test ModelValidation_ParallelGriddingMatchesGemmi
covers P1, C2, P212121, I23 and F4132 with iso and aniso atoms. A
standalone RefineRigidBody benchmark (synthetic |F| from the model,
displaced model) gives identical evaluations/angle/shift/coordinates to
the unchanged code: 5lzl 24-33 s -> 12 s real fit on a loaded machine;
null (9 concurrent replicates) unchanged within noise.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
jfjoch_broker: /start takes an optional `tokens` list (any number, all equivalent; writeOnly in the
API). While the current dataset has any, the endpoints that expose it - /statistics/data_collection,
/result/scan, /image_buffer/{start.cbor,image.cbor,image.jpeg,image.tiff}, /preview/plot{,.bin} -
answer 401 unless the request carries `Authorization: Bearer <token>`; /statistics keeps serving
the instrument view and only omits its `measurement` block. Enforcement is one pre-routing hook
over a named path set (the same set carries `bearerAuth` in jfjoch_api.yaml); tokens are compared
in constant time and never read back or logged. The tokens are replaced only by an accepted start,
and atomically with clearing the previous run's status, plots and image buffer, in this order:
clear, swap, import the new settings - so no moment serves the old run under the new tokens or the
new run's name under the old ones. Settings are validated on a copy first so a refused start
changes nothing.
jfjoch_viewer: a Token field (password echo) next to the http/https scheme in Open HTTP Connection,
JUNGFRAUJOCH_HTTP_TOKEN, and D-Bus LoadFile(..., token) / SetHttpToken; the dialog overrides the
others, nothing is persisted. A 401 clears the display, stops following and puts a line on the
status bar - no dialog, since a dataset changing hands is the normal cause.
Web frontend: key button in the top bar (token in sessionStorage, applied to the bearerAuth
operations by the generated client, and to the raw preview fetch), a tokens field in the start
form, and a "token required" hint on the plots. Python client: Configuration(access_token=...)
after regeneration.
Viewer: reference dataset accepts a structure-factor mmCIF (already read by content; the dialog
now offers it) and a model file can be chosen next to it (ProcessConfig::model_path, as
rugnux --model); the job also carries the reference's free-R flags, cell and setting, and the
copied command line states -z / --reference-column / --model. A grid scan keeps its own preferred
dataset-info plot and "Spots + background" means the spot count there. Dark theme: the navy hero
buttons, checked segments, warning texts and chart guide lines follow the theme instead of their
light-theme colours.
Docs: SECURITY.md section 3 is now the implemented scheme; a guided tour with four screenshots
in JFJOCH_VIEWER.md; broker, OpenAPI, Python client and frontend pages mention the token.
Tests: BearerTokens unit test and an HTTP round trip over the real broker HTTP layer (jfjoch_test
now compiles JFJochBrokerHttp.cpp and links httplib).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
macOS turns Shift + a vertical mouse wheel into horizontal scrolling, so
the notch arrives in angleDelta().x() and y() is 0; the contrast step
read only y and did nothing. The wheel handler now takes whichever axis
carries the notch, for zoom, foreground and background alike.
Help > Mouse Shortcuts names Cmd for Ctrl on macOS (Qt maps Command to
Ctrl), Ctrl-click for right click and Fn + arrows for Home/End/Page
Up/Down, and lists B held + wheel, which was missing. The magnifier's
tooltip says how to drive it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RELEASE_CONTENTS gains the macOS .dmg and rugnux .tgz in the artefact
table, the CPU floor (any Apple Silicon Mac), the OS floor (macOS 13), the
CPU-only note in the CUDA/GPU tables, and a macOS section: Apple Silicon
only (Rosetta does not run arm64 code on Intel), drag-to-Applications,
notices inside the bundle, and how to open the not-yet-notarized release
(Open Anyway on macOS 15+, Control-click Open on 13/14, or xattr).
JFJOCH_VIEWER states the platforms and requirements, that the Mac build is
CPU-only, that D-Bus is Linux-only, and adds Building from source on macOS.
RUGNUX_INSTALL adds the macOS archive and the quarantine note - checked: a
browser-downloaded .tgz hands its quarantine flag to everything tar
extracts, Gatekeeper rejects rugnux, and xattr -dr clears it. DEPLOYMENT
points to the pre-built Windows/macOS viewers and the macOS rugnux archive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision: a warning is a prompt to check and must catch the real cases (9min) at the
cost of some spurious ones. The physically motivated corrections stay (<|L|> outside its
physical range is not twinning; a single sweep's indexing choice; NO_LATTICE on rotation;
the no-crystal report); thresholds raised only to cut noise come back down:
- SUPERCELL_POSSIBLE warns wherever the class measures and rocks (as before rc173's audit
fix), worded as "check the cell", naming weak ordered intensity of a correct cell and spots
of further lattice domains as the other readings. 9min (rock 4.2%) warns again.
- LATTICE_TRANSLATION warns on every admitted vector (>=75% of the origin); below 90% the
wording names a very strong pseudo-translation as the other reading.
- PSEUDO_TRANSLATION warns on every detection; below a 20% peak it is worded as weak, check.
- SWEEP_GAPS warns where the degraded ranges cover at least 1% of the sweep (the 4 sets of 77
below that had 1-2 frames, 0.4-0.6% of the sweep).
- Powder rings are split between hexagonal-ice positions and the rest (MeasurePowderRings,
report-only fields); ICE_RINGS and POWDER_RINGS warn separately from 5% of the spots, and
ICE_RINGS also where the merge's ice gate found ice.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Report-layer fixes from the warnings audit of the rc173 full battery. Processing is
unchanged (merged MTZ/HKL bit-identical on the sets checked).
- SUPERCELL_POSSIBLE stays a key and a summary line; a warning only where the rocking part
reaches 20% (sub-cells that refine normally rocked at 2.1-16.2%, the accepted doubled cells
at 4.2% and 29.1%). The summary says where further lattice domains may put spots on the
half-integer nodes.
- LATTICE_TRANSLATION only where the Patterson at the vector reaches 90% of the origin
(UNDECLARED_LATTICE_TRANSLATION_PCT, new); the two admitted vectors on the battery read 79%
and 85% on crystals refining in the deposited cell, and are now reported as a very strong
pseudo-translation. PSEUDO_TRANSLATION warns from a 20% peak (xtriage convention; detected
peaks were 8.1% on one set, 24.8-63.2% on the rest).
- <|L|> outside 0.375-0.55 (on the adopted or the pre-search merge) is TWINNING_VERDICT=
NOT_READABLE with no warning, instead of a twin or SYMMETRY_SUSPECT reading.
- NO_LATTICE no longer fires on a rotation run that indexed the sweep with no frame indexed
on its own; INDEXING_RATE is --developer on rotation.
- A rotation run that finds no lattice writes a report (VERDICT= FAILED, NO_LATTICE) before
exiting 1, instead of an input-parameter error and no report (NoLatticeFound).
- Powder and Ice summary lines, ICE_* keys, and a POWDER_RINGS flag from a 0.5 spot fraction.
- SWEEP_GAPS warns only where the degraded ranges cover 10% of the sweep (77 -> 48 sets).
- A single rotation sweep no longer carries an INDEXING_AMBIGUITY warning (one orientation
matrix, consistent hand); INDEXING_AMBIGUITY_OPERATORS and an Indexing choice line instead.
- A DOMAIN more than 10 deg away is a second crystal; the MULTIPLE_LATTICES sum is documented.
- Mosaicity line and docs name XDS's suggested REFLECTING_RANGE_E.S.D. as the comparable
number (0.90x median over 34 in-house sets; per-image SIGMAR runs ~1.35x higher).
- HARMONIC_CONTAMINATION, SCALING_NOT_CONVERGED, POWDER_RINGS in the documented vocabulary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
After the existing mask is complete, a second step takes the pixels it left out that are
significantly dimmer than their ring, joins them through a 6 px bridge and across module gaps of
any width, and adds a piece whole when it holds >= 2000 dim pixels of which >= 200 are deep. The
existing mask is never touched, so a sweep with no such piece keeps its mask bit for bit.
Quick subset tests (battery --only, no model check) against the rc173 a0518abe6 full battery:
152 of 233 masks identical; 10 of 10 controls (including the sets where earlier shadow changes
regressed through marginal decisions) give byte-identical merges. On the 21 sets that gain a
piece the space group never changes, merge outlier rejections fall on 19, R_meas falls and ISa
rises on 17, and the shell-scaled model R improves on 16 of 18; a transmitting arm with an
over-subtracted background strip is recovered (R_meas 13.3 -> 12.0 %, ISa 13.4 -> 15.5).
Known costs: on one set with background bumps at ring radii the low-resolution agreement with
the model falls (CC 0.89 -> 0.85) while its internal statistics improve, and two sets cut
slightly coarser (1.50 -> 1.55 A, 1.72 -> 1.80 A) with a better model R.
Squashed from branch hq-beamstop (cfa080a59..60b6c6822), whose history records the variants
tried and dropped: a straight port of the earlier arm finder moved every mask, a two-shape
construction filled almost a whole sweep through hole filling, and a deep-fraction floor
rejected a real arm that is dim along its whole length.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The local radial background at ring radii (--background-radial) already
existed and is gated per image on the smooth-ice channel of the ice score,
but it was off unless asked for. Systematically negative merged
intensities at ring radii (I/sigma < -4 on 0.4-1.6% of reflections of the
affected open sets) come from the flat annulus mean over-subtracting a
sharp ring. rugnux now runs with =auto unless the flag says otherwise; the
library default (and so the broker) stays off.
Measured on the open arm against rc173 (commit 727bd0d6d as reference):
auto: 8agq (smooth ice, fires) CC_model .9327->.9476, R shell-scaled
.1987->.1795, I/sig<-4 1.55%->1.15%; 8rud R shell-scaled
.2507->.2456, ISa 10.0->13.3; 5reo, 7kcn, 9fhc bit-identical
(gate does not fire); wall time unchanged.
forced on (not adopted): same 8agq gain and 8rud .2414, 9fhc/8qq7
small gains, but 7kcn (no ice) CC_model .9491->.9411, ISa
11.0->7.5 - which is why the gate stays.
ISa falls on 8agq (15.6->13.3) while agreement with the model rises: the
merge-statistics vs external-model trade the settings comment describes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
A new correction surface, fitted after the time x detector surface and
before the goniometer-frame 8x8 grid: log A is a sum of real spherical
harmonics (l = 1..6, 48 terms) of the diffracted-beam direction de-rotated
into the crystal frame. The incident-beam path depends on phi alone and is
in the per-frame scale already.
It runs through ApplyCellSurface unchanged in everything but the update:
the cells are 32 x 64 equal-solid-angle direction bins, and each round the
per-cell sums (ref2, cross, the same damping) become one ridge-regularised
Gauss-Newton step on the coefficients (prior width 0.1/l per degree-l
coefficient) instead of independent per-cell steps. The half-set Fisher-z
gate adopts or refuses it exactly as it does the grids; where it is
refused, the grid after it sees what it saw before.
Why: the folded 8x8 grid (hemispheres share a cell) is the weak basis for
long-wavelength absorption. Offline, held out by unique reflection, this
basis lowered held-out scatter 4-8% on 6 of 11 long-wavelength sets where
no cell grid did, raised model-phased anomalous peaks 0.02-0.2 sigma, and
was neutral on hard-X-ray controls.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
Compared with ctruncate (CCP4 9, version 1.17.29) on rugnux's own merged
intensities of the open battery arm, three differences:
- ctruncate gives no amplitude to an intensity below -3.7 sigma (exactly
that bound on every set checked; up to 2498 reflections on one set).
Rugnux turned them into small, confident amplitudes (F/Fc ~0.3 on the
sets whose background is over-subtracted on powder/ice rings). They now
get F = NaN (missing in MTZ, '?' in mmCIF); IMEAN is kept. They are
also left out of the shell mean that sets the Wilson prior.
- The switch to sqrt(I) at I/sigma = 4 left 4-6 sigma amplitudes 2-3%
above ctruncate's on every set (1.019-1.028). The posterior now applies
up to 20 sigma (emulated: 0.995-1.000).
- A shell whose mean intensity is not positive gave a prior at the 1e-10
clamp and amplitudes of ~0 (one set's outer shell); it now takes the
nearest lower-resolution shell's mean.
The remaining gap (weak amplitudes 2-5% below ctruncate's in the outer
shells) is ctruncate's anisotropy-corrected prior; not attempted. An
offline R-free ablation put the whole ctruncate conversion at -0.0006
median (15/18 sets better) and dropping its rejected negatives at -0.0009
mean (-0.009 on the worst set).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
MULTIPLE_LATTICES and EXTRA_LATTICE_INTENSITY_PCT now count DOMAIN, TWIN_DOMAIN and SEGMENTED
lattices only. A FOREIGN lattice (an unrelated cell) is still listed but no longer warned about: it
is as often a wrong main lattice or tNCS as a second crystal (one of the four such sets in the
battery census is a tNCS crystal). On the census this names 25 of 200 rotation sets instead of 29.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
-z (now also --reference; --reference-mtz still works) takes an SF-mmCIF
(e.g. a deposited -sf.cif, gzipped or not) as well as an MTZ. The format is
recognised by content: a file starting with "MTZ " is an MTZ, anything else
is parsed as CIF. The first merged reflection block with the requested
column (or, by default, one the auto choice accepts) is converted to a
gemmi::Mtz in memory with GEMMI's CifToMtz and then read by the unchanged
MTZ loader, so the in-memory reference is exactly what the MTZ path yields.
Unmerged (_diffrn_refln) and anomalous-only blocks are passed over; the log
names the block used.
The R-free set comes from _refln.status (f -> FreeR_flag 0, o -> 1: the
CCP4 convention the loader already reads) and is preferred to
_refln.pdbx_r_free_flag, whose convention varies by program; a status
column with no 'f' is ignored and pdbx_r_free_flag is used instead.
Checked on two open-arm sets in the deposited setting (one with F_meas_au
only, one with intensity_meas): reference loaded with the deposited cell and
group, the inherited free set agrees with the deposited status 'f' on every
common reflection, and the merge correlates at CC 0.99 with the deposition.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
RUGNUX_ADVANCED (new section "The setting the files are written in", -z and
--model paragraphs), RUGNUX_REPORT (SETTING_* keys, REFERENCE_MISMATCH, model
change-of-basis keys), CPU_DATA_ANALYSIS_DECISIONS / _INTEGRATION, CHANGELOG
entries under rc.173.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The magnifier moves from a helper window into its own dock below the
inspector. It has three fixed zoom levels (x64 and x32 with the pixel
values written on the pixels, x10 without) instead of wheel zoom, and
follows the cursor only while Shift is held over the diffraction image.
Meanwhile the diffraction view draws a frame around the area it covers,
just outside that area so it stays visible at low zoom without hiding the
pixels; releasing Shift (seen by the application-wide key filter, so
wherever the focus is), moving without it or leaving the image hides it.
A "Pop out" button moves the magnifier into a separate window and back.
Qt's own dock floating stays disabled: a floated dock is placed
off-screen on WSLg. Whether it was popped out, and the window geometry,
persist across sessions. The inspector's image statistics become a
collapsible section so a small screen can give the space to the
magnifier. kLayoutVersion is bumped for the new dock.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Advisory only: the class is measured (<I/sigma> >= 0.5) and its rocking part is at least 2% of the
lattice's, three standard errors clear. The warning asks the user to process both settings - the
run's cell and the reported doubled cell with -C - and compare them in refinement. Of 105 battery
rotation sets it names 8: both crystals whose accepted cell is the doubled one, one pseudo-translation
whose doubled description is an accepted alternative, and five correct sub-cells with weak ordered
half-integer intensity.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The report-only supercell probe now reaches the results report: SUPERCELL_CLASS (parity of the
primitive indices), its occupancy and Bragg-like (rocking) part against the lattice's own reflections,
its <I/sigma>, and SUPERCELL_DOUBLED_CELL, the Niggli-reduced cell to give with -C. No decision is
taken on it: correct cells whose half-integer class is diffuse, and cells whose depositor kept the
sub-cell, cannot be told from a real doubling by the data alone. On a crystal with a real doubled
axis the reported cell matched the deposited one, and processing on it with -C brought R_free
against the deposited model from 0.60 to 0.29.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The between-pass geometry walk kept a round only when the realised held-out residual fell by more
than its noise, and started only on a fit move of a trust-region step or more. The residual is
dominated by low-resolution reflections, where a distance and the compensating cell scale move every
spot alike, and its centroids are taken inside a disc centred on the prediction, so it barely sees a
distance error that costs the high-resolution shells. On 8pqd the canonical pass ran at 96.456 mm; the
fit asked for 95.878 mm (0.6 %, less than a step, so no walk). Forced, that round read the residual
only 0.74 sigma lower but put 70.4 % of the validation spots on the lattice against 58.1 %.
Now a round is kept when either the residual falls beyond its noise (HeldOutResidualFell) or the
validation evidence prefers it (ValidationEvidencePrefers, z = 3.29). A move of less than a step is
first tried as two index-only probes on the validation frames (fit's geometry vs the one in hand) and
pays for a re-integrated round only where the fit's geometry scores higher; walks started by a large
move run as before.
8pqd: 96.456 -> 95.878 mm, d_min 1.374 -> 1.306 A, R_free .218 -> .206, REFMAC R_free .209 -> .202,
Wilson B 34.4 -> 30.7 (pool had 1.325 A / .209 / .203). 6vww 7n0i 7ris 8egn 8xtf 9qw8 9w3y 6qaj
6cdl 9i0a myob_x06da_split lyso_x06da_half_image unchanged (probes say the geometry in hand stands);
probe cost 1-10 s per run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
R_MEAS is what users set beside XDS/AIMLESS, so its meaning does not change; the merge-weighted value
from hq-cure-merge is reported beside it (R_MEAS_WEIGHTED, REFRES_R_MEAS_WEIGHTED). The per-hand
table keeps the ordinary statistic.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
The delta-CC1/2 disposition keeps a weak stretch in the merge (downgraded)
wherever removing it would not raise CC1/2, and the merge then carries each
of its observations at the small 1/sigma^2 its scaled-up counting error gives
it. R_meas counted those observations at full weight, so the frames that add
next to nothing to the intensities set the number. hq-pool battery: 8xtf kept
115 weak frames (scale 0.1-0.2 of the run's) with the same CC1/2, <I/sigma>
and CC_model per shell as rc173 with 84 frames rejected, and R_meas was
1.3-1.5x higher in every shell; 9w3y (33 deg rejected -> 0, every model
metric better) and lyso_x10sa_strong read the
same way. The disposition itself is not segmentation-dependent: conviction
is on the batch grid, not the ledger ranges, and those sets' dispositions
changed because the corrected data changed.
R_meas now weights each observation by its merge weight v = 1/sigma^2 under
the error model (corrected_sigma on the host, ModelSigma on the GPU, with the
error model of the last MergeAccum), normalised per reflection to Kish's
effective count, so equal sigmas give the ordinary formula
(WeightedRmeasTerms). Where the proportional term dominates - strong
reflections - frames are weighted alike, as in the merge: 8xtf's lowest shell
reads 15.0%, the rc173 run with 84 frames rejected 15.2%. The per-hand table
is weighted the same way. R_MEAS_UNWEIGHTED / REFRES_R_MEAS_UNWEIGHTED keep
the XDS/AIMLESS convention and are what to set beside XDS; the battery scorer
records them. MULTIPLICITY stays a count.
Effect (weighted / unweighted): lyso_x06da_ref 0.0454 / 0.0479, 8xtf
0.225 / 0.907, insu_I_x06da_ref REFRES 0.072 / 0.232. GPU and CPU paths agree
to the last printed digit on lyso_x06da_ref.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
