The device scaling's peak, measured with nsys on four large sets (33-123 M observations, 4-16 M
distinct hkl), is 81 bytes an observation plus 291 bytes a distinct hkl to within 5%. Once the
ingest has counted both, an engine whose estimate is over 80% of the card's total memory scales on
the CPU instead, with one warning line saying why; a resident ingest is made again on the host,
which the CPU path needs. Decided once, before anything is scaled, on the card's total memory: the
answer depends on the card, not on the moment.
On a simulated 8 GB card (total clamped to 8.2e9, 7.75 GB free) a 33 M-observation, 15 M-hkl set
whose final merge alone needs ~7.5 GB on the GPU ran out of memory there; it now completes, and its
merged MTZ matches the 16 GB GPU run to float rounding (max relative difference 9e-7 on any
intensity column, same reflections, cell, space group and resolution cut). The largest estimate on
the battery is 12.45 GB, under 80% of a 16 GB card, so nothing changes there.
The all-observations arm is copied from the engine's card arrays only when that engine is on the
GPU.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The ingest term was 90 bytes per reflection and 64 per run. Measured with cudaMemGetInfo at each
step of IngestUpload/IngestSort (relative to the free memory before the observation arrays):
- the second radix sort is the peak where runs are few: the 66 B of observation arrays, by_image,
perm, key, key_sorted and cub's alternate key/value buffers of a plain SortPairs - 102.3-103.0 B
per reflection on 7qij (51.1 M, 52.2 M, 78.1 M reflections) and 4nwv (32.7 M);
- the cut is the peak where runs are many: 98 B per reflection plus 56 per run (4nwv, 15.4 M runs:
125.8 B per reflection, 4106 MB against 3923 MB from the old formula);
- the upload holds the 66 B beside the two 64 MB staging buffers (~150 MB measured).
Now max(103 B per reflection + 58 B per run, 66 B per reflection + 160 MB): 2-4% above every
measured peak, where the old formula was 10% under on 7qij (7170 MB against 7986 MB for the main
engine) and 4.5% under on 4nwv. Its only consumer is whether the all-observation arm goes beside
the main engine, which changes when that merge runs, not what it gives.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
Every merge downloaded the fulls' frame and group keys and built both CSRs by serial counting
sorts on the host - four passes over the fulls, two of them scattering into an array as long
as the distinct hkl (15 M on 4nwv) - and uploaded them again. They are now stable radix sorts
of the resident fulls on the card, with the counts summed there; the host gets the counts per
frame and the group CSR it walks itself. The same permutations (stable sorts list each frame's
and each group's fulls in index order, as the counting sorts did): p.hkl identical on cytc
and 4nwv.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
On a set too large for the quality guard's engine to sit beside the main one (8qaw: 123 M
observations; 8a1a), the main engine was ingested, found not to leave room, dropped and
ingested again after the guard - several seconds of ingest for nothing. The pre-pass's
engine merged the guard's very frames earlier in the run, so its card memory per reflection
is now kept (prepass_engine_bytes_ / prepass_engine_reflections_) and both engines are
reckoned from it before either ingests: the guard at its frames' reflections, the main
engine at all of them (observations add in proportion; rocking events and distinct hkl at
most in proportion). Beside where the two fit the card less the 3 GB reserve, otherwise
the guard first. The re-ingest path is gone.
7qij, 9v16, cytc: beside; 8qaw, 8a1a: guard first. Same merges, same numbers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The rest of the resident ingest moves to the card: the delta_phi re-solve against each
frame's smoothed lattice, the one exact-Bragg angle per rocking event, the partiality
recomputed from the smoothed mosaicity (with corr brought to it) and the histogram of the
frames an event spans. The host keeps the per-frame work (the smoothed lattices, the
mosaicity window) and reads back only corr; nothing per observation crosses back, and the
host arrays the smoothing worked on are no longer built on this path.
The integer work is the host's to the bit (the event-span histogram). The floating-point
work differs in the last bits: every rounding on the card is explicit (no contraction, so
the same bits on any GPU), the host's compiler fuses some of the same products into sums,
and erf/atan2 come from the CUDA library. This is the one place in the lever where results
change; see the hand-back report for the sets and metrics it moves.
Measured on 7qij (main engine, 78 M reflections): the whole ingest 6.5 s host + 1.3 s device
before this lever -> 2.9 s, of which 2.4 s is the upload (memory-bound, beside the
post-refinement gather) and 0.25 s this smoothing. Same space group, error model,
resolution and reflection count; IMEAN differs at the 1e-7 level except on near-zero
intensities.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
With the GPU resident, the host ingest swept the source reflections four times (background
and h histogram, sort keys, the in-range build, the device staging), bucket-sorted the keys
and cut the raw-hkl runs: 5-6 s of host time per engine on a 78 M-reflection sweep, run
twice beside the quality guard. Now the frames' reflections are copied once, through two
pinned buffers, to the card, where the per-frame background is summed, the reflections
are ordered by two stable radix passes ((image, index), then raw hkl - the one order the
host's total comparator gives), cut into runs by run-length encoding, filtered by the
resolution range and compacted. The observations, the runs and their order are the host
ingest's to the bit; a new GPU test checks the layout against the host's sort and cut.
The geometry smoothing still runs on the host in this step, on arrays read back.
The all-observation arm's engine is cloned after the main engine's device ingest (its
DeviceBytes are known only then) and copies the runs on the card as well. DeviceBytes now
also covers the ingest's own peak (90 bytes per reflection, 64 per run before the cut).
Measured: p.hkl and MTZ data identical on cytc and 7qij.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The group reduction reaches every observation through the group permutation, and so fetched one
32-byte sector per field for I, sigma, partiality and corr. I, sigma, partiality and prescaling_corr
now sit together in one aligned record, so it fetches two sectors per observation; the grouping
kernels' finiteness test and the combine read the same record. The per-observation fit coefficient,
its flag and 1/sigma are no longer stored: the per-frame fit forms them as it reads each observation
(the same expressions, so the same bits), which removes PrepScaleObsKernel and 13 bytes per
observation of device memory. DeviceBytes' per-observation figure drops by those 13 bytes.
Every sum keeps its order: p.hkl md5 identical to the base on cytc, 8a1a and 8qaw.
RTX 5080, nsys, merge-engine kernels: 8a1a 2459 -> 1924 ms, 8qaw 3624 -> 3025 ms, cytc 501 -> 476 ms
(ReduceGroupMeans 1.5-1.6x faster, PrepScaleObs gone, FitPerFrameG absorbs it).
Measured and not taken: storing all per-observation arrays in group order makes the reduction 3-4x
faster on its own, but the per-frame fit then has to gather, and moving 15 arrays at every grouping
costs as much as the reduction saves (net 0 on the 5080); a record for the combine-only fields
did not speed the combine.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The partiality recompute reads d only to broaden the rocking curve by the energy bandwidth. On a
large set the final ingest sits at the process's peak resident memory (8a1a: max RSS rose by the
array's 4 bytes x 68 M observations), so the array is kept only where a bandwidth is set; with
zeta in and the predicted-partiality copy out, the ingest's peak is then where it was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The all-observation arm of the space-group search merges on a copy of the engine (CloneIngest), and
its IngestDevice staged every observation from the ~92-byte source reflections a second time -
concurrently with the main engine's own staging of the same reflections. Now the main engine
stages, and the copy takes its card arrays with cudaMemcpyPeer (a device copy when both engines
share the card, as on a one-card machine). CloneIngest no longer copies the predicted partiality,
which only that staging read.
Output identical (cytc p.hkl md5, and with --dump-observations the dump md5).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The ingest read every ~92-byte Reflection seven times per engine: the background/h-range sweep, the
h histogram, the key scatter, the observation build, the delta_phi re-solve against the smoothed
geometry, the partiality recompute, and the device staging. On a box with several sockets these are
remote, latency-bound loads (the owner's 4-socket profile puts RotationScaleMerge's ingest lambdas
at the top of the CPU time on most large sets). Now:
- the histogram of h is counted in the background sweep, per chunk of frames over the whole int16
range the sort key holds, so the h range needs no sweep of its own;
- delta_phi is re-solved against the smoothed per-frame lattice while each kept observation is built
(SmoothedFrameGeometry computes the per-frame bases first, SmoothedDeltaPhi solves one);
- the build keeps zeta and d beside delta_phi, so the partiality recompute reads compact arrays, and
it brings corr_ingested to the recomputed partiality in the same loop, which retires the
observation-length copy of the predicted partiality and its separate pass.
Output identical (p.hkl md5 on 9gdj and cytc; cytc also with --dump-observations, dump md5 equal,
and CPU-only). Transient host memory: +4 bytes per kept observation inside IngestHost (zeta and d
in, the predicted-partiality copy out) and nthreads x 256 KiB of histograms.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The gate for merging the space-group search's all-observation arm on a copy of the engine, beside
the first arm, judged an engine at a fixed 256 B per observation. It now counts what the ingest
knows - observations, rocking events (at most one full each) and distinct hkl (at most one ASU
group each) - at 80 / 260 / 200 B, the envelope of the measured per-engine peaks (helpers
included: error-model fit, Wilson outlier test, pair sort), plus a fifth, and keeps 3 GB of the
card for what else stays resident (context, the run's indexers: measured up to 2.3 GB at the gate
with no speculative pass running).
Measured per-engine peaks (scratch build tracking each engine's device allocations): 4nwv 7.98 GB,
8a1a 8.15, 8tyy 9.36, 8qaw 13.3 - two engines do not fit a 16 GB card on any of these, so they stay
serial. The old constant was not an over-estimate there (fulls cost ~260 B each at the surface
fit). What changes: 9 battery sets of 33-53 M observations at high multiplicity (8s38, 8sqt,
7ph1, 9min, 9i80, 9sl0, 9h0q, 8v4o, 6ze4) now merge the arm beside; none goes the other way.
Their merge-step VRAM peak is 11.0-14.1 GiB of 15.9, no out-of-memory, no failed speculative probe.
Every engine peak measured on 11 sets is at most 81% of the estimate.
Output identical (p.hkl md5 and MTZ data, 40 sets incl. ci tier). Gain is small: the two merges
share one GPU, so the arm beside slows the main P1 merge by ~0.5 s while saving ~0.7-1 s of wait
at the search; alternating A/B on 5 of the sets: -1.1 s output pass in total.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
The space-group search made its two P1 merges one after the other on one
engine: the Lorentz-filtered one, its search, then the all-observation one and
its search. Since every merge now starts from the same state, the
all-observation merge is made on a copy of the engine (CloneIngest: the host
half of the ingest copied, the device half staged on the copy's card), started
right after the ingest and read where the search needs it, so it runs beside the
P1 merge, the re-scaled search merge and the first arm. Its search then runs
beside the first arm's.
Placement: from three cards up the copy gets card 2 to itself (card 0 has the
main engine, card 1 the quality guard's). With one or two cards it shares the
last card, and only where two engines of this size fit the card's total memory
(DEVICE_BYTES_PER_OBSERVATION = 256; two engines measured at ~150 B/observation
each), so the choice never depends on what is free at the moment. On a 16 GB
card that admits sets up to ~32M observations; larger ones keep the serial
order. Nothing of the copy reaches the run before the first arm says the arm is
wanted: its log, its result fields and its per-frame write-back onto the
outcomes are taken at the point where the serial merge would have been made. A
copy that failed is merged again in series.
Results are bit-identical to the serial order (p.hkl md5 and MTZ data), at
default -N and -N 8.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Each merge restored corr to the ingest state but kept g_partial from the merge
before it. A frame the new merge did not fit kept the G an earlier merge had
fitted it with, and the fulls' CC1/2 weight (frame_scale_fulls) read it, so a
merge depended on which merges ran before it on the same engine. RunArm now
resets g_partial to 1 at its start, as the first merge on an ingest already had
it. That is what lets two merges run side by side on two engines and give what
they give in series.
The fits themselves never warm-started (FitPerFrameG and the device kernel both
solve each frame afresh), so the change only reaches frames that a merge leaves
unfitted after an earlier one fitted them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged observations already leave a run as the unmerged MTZ (and
the scaled fulls as the .hkl), and deposition does not come straight
from rugnux, so the _diffrn_refln block made the .cif up to ten times
larger for no consumer. The .cif is one data block again; the keys,
software rows, metadata, R_merge/R_pim/CC* and provenance stay.
ScaledFull loses the scale and frame it carried only for that block.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the
entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on
_reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/
_diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a
loop naming Rugnux for 'data reduction' and for 'data scaling'.
The experiment as the input states it: _diffrn.ambient_temp from the
sample temperature, the source's general class (from NXmx
/entry/source/type, which the HDF5 reader now reads), facility and
beamline, the detector make/model in _diffrn_detector.type (where
.detector used to carry it) and its class in .detector where the name
settles it, and the collection date from /entry/start_time.
R_merge and R_pim are formed beside R_meas from the same per-reflection
sums (stills and rotation merges) and written with CC* overall and per
shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the
_reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as
three new columns of the shell tables. R_pim following Weiss (2001),
CC* following Karplus & Diederichs (2012).
A second data block, data_sample_unmerged, carries the scaled unmerged
observations of a rotation merge as _diffrn_refln (measured index,
frame, scaled I and sigma, pdbx_scale_value) - the same records as the
.hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now
carries the scale and the frame. The unmerged MTZ stays unscaled.
Reflection data are unchanged: the merged _refln rows, the .mtz columns
and the .hkl are written as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two
threads that drive the merging tail were not: the quality guard took its card with set_gpu(1),
and the main engine runs on the pipeline's own thread, which goes on to everything else. On a
multi-socket server their pageable copies to the card went through a remote socket.
RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost,
IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous
device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding
covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads
the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use
std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not
squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses
pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask
themselves; the new test checks both that and that a pinned caller keeps its own mask.
No flag, no environment variable. On one NUMA node (this workstation) only the device setting
happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base =
rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa,
cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs),
-N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise,
e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for
multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The device correction-surface fit (gpu-tail2) keeps its per-term arrays - the
terms, their parity, the group CSR, the three subsets' orders and the per-term
weights, about 60 bytes a term - as engine members, and nothing freed them
after the last surface. They stayed on the card through the merge, where the
GPU Wilson outlier test (gpu-tail) then asks for ~96 bytes per full. On a
large rotation set (139.8 M partials, 13.35 M fulls, 16 GB card) the engine's
resident arrays left 1242 MiB free for the first Wilson test and 998 MiB for
the one after the automatic resolution cut, against 1222 MiB needed: the run
stopped with "Wilson outlier test ... run the CPU build" although nothing ran
beside it. The surfaces' buffers are released once the last surface is
fitted (the next pass's surfaces build their own), which gives back 905 MiB
there.
The up-front checks of the three dataset-sized tail steps (Wilson test, pair
sort, error-model fit) also read free memory as cudaMemGetInfo reports it,
which counts what the allocation pool holds with no buffer in it as taken.
They now hand that back and, only if it is still short, wait for GPU work
running beside the merge (GPUWorkBeside, the speculative probe) to end before
giving up - as the engine's own allocations already do. That changes when a
step runs, never what it computes; a set that does not fit still fails at
once with the same "run the CPU build" message.
p.mtz: the large set completes (and is identical with the Wilson test forced
onto the host); myob 9bdd6916 and 8a1a 309ea9a8 unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The R_meas accumulators per group and the device's per-group sums were eight
group-length locals plus a 56-byte record per group, value-initialised on one
thread for every merge that reports statistics. They are now members, reset in
parallel, and the device branch's pass that assigns each group its shell and
copies its sums runs on all threads; the per-shell observation counts are
integers, gathered per block. The shells' R_meas sums are still added in group
order, as before.
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Which groups are written is decided first (KeptIndices, in group order), then
each written group's MergedReflection is formed on its own, on all threads,
into a result sized once - where the serial loop pushed back up to millions of
records, growing a vector that each merge allocated afresh. The same records in
the same order.
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The rejected flags and the observation keys and halves (per full), the merged
intensities (per group) and the Wilson observation records were locals: on a
large cell every one of the run's merges allocated, page-faulted and zeroed
them again on one thread - hundreds of megabytes per merge on 8qaw. They are
now members, only resized, and reset with ParallelChunks where a pass needs
them reset. The error-model samples are no longer cleared before the device
download overwrites them (the host path still clears before it appends).
p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With the fulls resident (the GPU combine ran), ApplyCellSurface no longer builds
its terms on the host and uploads them: SurfaceBuildTerms makes them from the
resident fulls, this surface's cell per full and the current corr (the only two
uploads) - which fulls give a term, the terms in fulls order, the group CSR over
them and each subset's per-(block, cell) segments - and SurfaceShellWeights the
resolution gauge's inputs: the shell edges (order statistics of 1/d^2 read off a
sort), each group's shell and each (cell, shell) bin's weight. Every order the
host made is a stable counting sort, which a stable radix sort of the indices
reproduces; the per-bin weights are summed one bin per thread in fulls order
with the host's rounding. The existing exact surface kernels read them as
before. Without resident fulls (the observation dump) the host path runs.
p.mtz md5 and the surfaces' log lines unchanged on myob, cytc, thau, kdp, 8a1a
and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
With two GPUs or more the two-pass quality guard's merge of the pre-pass
frames gets a scaling engine of its own on device 1 (RotationScaleMerge::
SetGpuDevice), and the main engine's device ingest and first merge no
longer wait for it to hand the card back. The first merge waits only for
the guard's ingest, the last time the guard reads the outcomes: that merge
writes per-frame fields (mosaicity among them) back into them.
Neither merge reads what the other writes, and a merge gives the same bits
on any card of one model, so only when the guard runs changes. With one
GPU, and in the CPU build, the order is the one it was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
std::thread::hardware_concurrency() counts every CPU of the node, so a job
given a CPU set (taskset, a cpuset, a Slurm allocation) or a container CPU
quota started one thread per node CPU: the default -N, the shared worker
pool and every "0 = all threads" default. AvailableCpus() (ThreadAffinity)
counts the start affinity mask, capped by the cgroup CPU quota (v2 cpu.max,
v1 cfs_quota/period, the process's own cgroup and then the mounted root),
read once. Outside Linux it is hardware_concurrency(), so the viewer tree
stays portable. It replaces every hardware_concurrency() call outside the
tests and the vendored pocketfft.
Only how many threads run changes; the passes split their work by n alone,
so the results do not depend on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
FitErrorModelGPU puts the samples in the host's rank order with four stable
radix passes, so the equal-count bins hold the same samples; computes every
normalised deviation with the host's rounding, so every bin median is the
host's to the bit; and hands the per-bin terms to the host's own update step,
now ErrorModelUpdate (a pure extraction from FitErrorModel). The one thing
that differs is the order each bin's two sums are added in: a fixed tree on the
device, the order nth_element happened to leave the bin in on the host. a and
b^2 can therefore differ from the host fit in their last bits - measured on two
synthetic pools: relative difference 2e-16 and 3e-13 in a, 5e-16 in b^2. The
device fit is deterministic (same pool, same bits).
p.mtz md5 nevertheless unchanged on myob, cytc, 8a1a and 8qaw (GPU build): the
difference does not survive the float rounding of the merged sigmas there. A
knife-edge decision elsewhere could still see it, which is why this is kept as
its own commit.
The host fit took 0.4-0.6 s per fit on the large merges, about half of it the
initial equal-count split, with only 16-way parallelism in the iterations.
Device peak about 72 bytes per sample (the rank sort; each stage's buffers are
freed when it is done); the free memory is checked first and a device without
it stops with a message naming the CPU build.
Kept last on the branch so that it can be dropped on its own.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Every full's observation key and its half - the parity of the key's rank among
its group's usable fulls - are integer work, so the device gives the host's
halves exactly: one thread per full for the keys, one per group for the ranks.
The host receives both (the anomalous statistics rank the same keys again within
each Bijvoet cell) instead of walking every full's record for them.
p.mtz md5 and p.cif unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The strong-reflection asymptote is read from per-group sums of I*corr, its
square and the counting variance at the group mean - a walk over every full's
record on the host. In the GPU build a kernel takes them, one thread per group
in index order with the host's rounding (CountingVarianceExact); the host keeps
the strong-group selection and the median.
p.mtz md5, p.cif and the error-model lines (which print the asymptote)
unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The multiplicative spread of the outlier band is the weighted median of
(|ln(I/median)|, weight) pairs, one per usable full, and its sort was the bulk of
that step. SortPairsGPU sorts them on the GPU (by the second key, then stably
by the first) and leaves the sequence std::sort leaves: two pairs that compare
equal are the same two numbers. The values themselves (a logarithm) and the
running sum stay on the host.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build); a Catch2 case
checks the device order against std::sort.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The per-group sums the widening is measured from - each hand's inverse-variance
mean under the error model's sigmas, the group's d and the d range - are a walk
over every full's record on the host. In the GPU build a kernel takes them, one
thread per group in index order, with the host's model sigma to the bit
(ModelSigmaExact: every product rounded on its own); the host keeps the shells
and their medians. The host path is the same sums, now kept per group so both
paths hand the shell step one array.
p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
MergeEmSamples now forms each sample whole on the device - the counting
variance at the reflection's expected intensity included, every product rounded
on its own as the host build rounds it - and compacts them in fulls order with a
scan, so the host receives exactly the vector it used to build. That replaces
the download of three full-length double arrays and a flag array, and the host
pass that compacted them while reading every full's eighty-byte record.
p.mtz md5 and the error-model lines unchanged on myob, cytc, 8a1a and 8qaw
(GPU build).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
WilsonOutliersGPU runs the test of WilsonOutliers.cpp step for step on the
device and returns the same result to the bit: the two orders the host sorts
into are total orders, so a stable radix sort of the indices reaches the same
sequence; every sum is taken by one thread over its shell or reflection in the
host's order; each product the host rounds on its own (-ffp-contract=off) is
rounded with __dmul_rn; medians are read off sorts; and the logarithms (t, z,
u, the tail scale) are taken on the host from downloaded counts and medians.
MergeAndStats calls it where the merge itself is on the device.
8a1a final P61 merge, "rsm merge and statistics" 4.35 -> 2.82 s (the test runs
twice there, once per error-model fit). p.mtz md5 unchanged on myob, cytc,
8a1a and 8qaw (GPU build). A Catch2 case checks the device result against the
host one on a population that reaches every branch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ApplyCellSurface copied and sorted every term's 1/d^2 to read 19 order statistics off it, for each
of the four surfaces of the final merge. The edges are now selected: one pass counts the values
under each top-16-bit pattern (a non-negative float orders as its bits), a second collects the few
bins the ranks fall in, and nth_element picks the value inside each. An order statistic is a value,
so the edges are the ones the sort gave.
8a1a, GPU build: the shell set-up of the four surfaces 0.55 -> 0.26 s (instrumented run), the
correction-surface window 6.32 -> 6.12 s and the final merge 11.03 -> 10.82 s (two interleaved warm
runs each); whole run 55.78 -> 55.45 s. 8qaw surfaces 2.95 -> 2.83 s.
Tried in the same window and not kept, as they measured nothing: the serial stable counting sorts
(surface group order, decay terms, fulls CSRs, delta-CC1/2 frame order) on all threads; delta-CC1/2
workers uncapped from 8; measuring two rounds of the edge search at once. The delta-CC1/2 window
(2.4 s) is bound by its longest chain of ranges, one serial accumulation, not by the worker count.
p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The unmerged scaled fulls the final merge hands back for the .hkl were gathered by one thread
pushing back over every full. Which fulls is now KeptIndices (the same list, in the same order),
and each record is made side by side.
The same records in the same order: p.mtz and the .hkl byte-identical (myob, thau, cytc, kdp,
cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a, 8tyy, CPU build myob). 8a1a, no other
load, interleaved: the P61 merge's statistics stage 3.94 -> 3.71 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
RotationScaleMerge::Ingest is now IngestHost (read the outcomes, sort, build every host array,
smooth the geometry and recompute the partiality) followed by IngestDevice (allocate the GPU's
arrays and stage them). The host half reads the outcomes and nothing else, as the post-refinement
and the quality guard's merge do, so on the canonical pass it runs on its own thread beside both;
the device half runs where the ingest used to, after the guard has handed the GPU back, so the two
engines never hold the card at once. The device feed now stages the recomputed partiality in the
same slices as every other field, straight from the ingest array, instead of in a pass of its own
after the rest; the values on the device are the ones they were.
8a1a, no other load, interleaved runs: the first merge (P1) ends at 10.9 s after the image pass
against 11.3 with only the guard overlapped and 12.7 on the rc175 base. p.mtz byte-identical on the
gate sets (myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a,
8tyy, CPU build myob); --model, stills, calibration, --mode scale (GPU and CUDA_VISIBLE_DEVICES="")
outputs identical; --no-merge / --mode azint / --write-process-h5 _process.h5 differ only in the
datasets that differ base-vs-base (azint sums, bkgEstimate, profiling times, dates, git).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
ChooseLogScaleSmoothing picks the smoothing of the fulls' per-frame log scale by leaving out
every fifth block of frames: 25 lambdas x 5 folds, each a banded solve over the whole sweep, run
one after the other on every iteration of every fulls scaling loop. The 125 solves are independent,
so they now run on the engine's threads; the held-out error is still added up per lambda over the
folds and frames in the order the serial loop added it, from the same curves, so the lambda chosen
is the same. nthreads defaults to 1 for other callers (the test).
8a1a: the six "combine and scale fulls" stages of the tail 4.7 -> 3.6 s. p.mtz byte-identical
(myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a); [RotationScale]
and the scale_merge test files pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
The guard's merge of the first prepass_images frames read nothing the post-refinement
writes and the post-refinement reads nothing the guard writes, but the two ran one after
the other at the start of the tail. The guard now merges the first frames of the outcomes
in place (RotationScaleMerge::SetFrameLimit) with the per-frame scale not written back,
instead of moving the reflections into a copy whose write-back was thrown away, and it
runs on its own thread from just before the post-refinement; its log lines are held and
replayed where the guard's result is read.
Same observations in the same order, same settings: p.mtz is byte-identical (myob, thau,
cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a). On 8a1a the 3.7 s
guard is now inside the post-refinement, which itself goes from 4.5 to 6.0 s under the
shared load: about 2 s off the tail.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
A TimingMark (common/Logger.h) logs "TIMING <section> <seconds>" at debug level when a section
ends, so the lines go to <prefix>_log.txt and stay off the terminal. Marked: each pass of
RunPipeline (pre-pass, probe, output) and its first-pass indexing, image pass, scale/merge/symmetry
phase, output files and census wait; inside the scale/merge phase every merge (labelled by its
role: P1, P1 space-group search, P1 all observations, the adopted group, P1 cross-check, quality
guard), the post-refinement, the space-group search and its two SearchSpaceGroup arms, adoption,
analyses, the P1 cross-check wait, model validation and writing; inside RotationScaleMerge the
ingest and, per arm, partial scaling, combine and fulls scaling, correction surfaces and the merge;
the console's steps; and the results report. tools/rugnux_timing.py tabulates the tail of a log.
Only the log changes: the outputs are byte-identical.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
When a chain needs more fulls than the arrays hold, fulls and fulls_staging are released before the
resize. The gather overwrites every field, so growing in place only copied the previous chain's
records into the new allocation to be overwritten.
The expected saving (a few tenths of a second on 8tyy) is below the run-to-run noise of the wall
clock: 8tyy median of 3, 127.0 s without vs 129.6 s with, under different machine load. 8qaw max RSS
21.9 -> 21.7 GB.
p.mtz md5-identical to rc175 on myoglobin, cytochrome C (also at -N 4 and -N 8), thaumatin, 8a1a,
8tyy, 9gdj and 8qaw.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Floating-point summation order only (tier F): the seven global sums of RefineDecay (slope fits,
held-out disagreements), the per-batch sums of FitRelativeBCurve, and the per-shell and
per-(batch, shell) sums of MeasureRadiationDamageB were single-threaded walks over every full.
They are now FixedBlockSum: the split depends on the number of fulls alone (ReductionBlocks,
ParallelBlocks), the block partials are merged in block order, so the sums have the same bits at
every thread count and on every machine - but not those of the serial walk.
Checked: p.mtz, p_P1.mtz, p.cif, p.hkl and the report identical to production on myob/cytc/thau
(GPU) and myob (CPU), and identical between -N 4/8/default (GPU cytc, myob; CPU myob). On these
sets the decay correction is not adopted (negligible or not cross-validated) and the monitor's
printed values do not move, so the F-tier change is not visible in their outputs; a set where the
decay correction is adopted will move in the last bits of corr.
Clean branch: p.mtz md5 identical on myob/cytc/thau/8a1a (GPU), cytc -N 8, myob -N 4 (CPU).
Measured (prototype with spans, 8a1a, 2 interleaved pairs, load 8-10): decay sections 2.70 -> 1.08 s,
radiation-damage monitor 1.04 -> 0.86 s, RSM total 37.1 -> 35.6 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact (tier E): the device gets the same group ids and the same group-ordered permutation
(ascending observation index within a group) the host's counting sort produced.
ComputeAsuGroups still finds the groups on the host (one ASU reduction per raw-hkl run, the
(key, run) sort, the dense ids, the representatives - now unpacked from the packed key instead of
a second ASU reduction). With the GPU resident it then hands the device only the runs in group
order (two arrays as long as the runs, not the observations): BuildGroups stamps every
observation's group from its run (the device evaluates the same finiteness test on the same
uploaded floats), counts each group's observations, and fills each group's segment and puts it in
index order. The host no longer builds or uploads the two observation-length arrays; the device
keeps its per-observation group array across Runs instead of reallocating it beside the old one.
SetGroups (host-built upload) is gone; the CPU path keeps the host histogram CSR.
Measured (prototype, GPU RTX 5080, Run-span sections): ComputeAsuGroups summed over a run's
merges 8tyy 9.0 -> 4.2 s, 8a1a 2.2 -> 0.85 s, cytc 0.51 -> 0.19 s. md5 of p.mtz identical to
production on myob/cytc/thau/8a1a/8tyy and cytc -N 4.
Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact (tier E) for every written file.
- A search merge (for_search) is never written - the run always merges again in the chosen
group - and nothing reads rfree_flag except the writers and --model validation of the written
merge, so RotationScaleMerge no longer assigns a free set to it.
- AssignRfreeFlags takes a thread count: the per-reflection key (the lattice holohedry walk) runs
on fixed chunks, and the distinct Laue-key count is taken off a parallel sort (equal keys are
equal values, so every sort gives the same sequence) instead of an unordered_set. Rugnux passes
its thread count at its two direct call sites; the default stays 1.
Measured (prototype, GPU, 8tyy F432 cubic): the R-free/chi2 section of the merges 4.5 -> 0.65 s;
8a1a 0.76 -> 0.37 s; cytc -0.13 s. md5 of p.mtz/p_P1.mtz/p.cif/p.hkl identical to production on
myob/cytc/thau/8a1a/8tyy.
Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Exact (tier E): every per-group and per-pair sum keeps its index-order summation, so p.mtz and
every written file are bit-identical; nothing depends on the thread count.
- The fulls' group CSR (index order within a group) is kept per arm: on the GPU path it is the
one RunArm already builds for SetFullsGroups, on the CPU path BuildFullsGroupCSR.
- The four "every thread walks every full, keeps its own groups" loops (error-model sums on the
CPU path, Bijvoet sums, ISa asymptote, anomalous pair sums) become ParallelBlocks over groups /
pairs walking the CSR - O(n_full) instead of O(threads x n_full) traffic.
- The serial walks that build per-group/per-pair data (outlier-median sets, band spread pairs,
group d, anomalous-variance shells, per-hand R_meas rows, per-frame counts, the error-model
sample list, the Wilson observation list) are ordered-parallel: fixed-block compaction
(KeptIndices), CSR fills, per-block lists whose consumer is order-free (medians).
- CC1/2 and CCanom halves are assigned cell by cell from the CSR (the rank is a set property).
- Friedel pairs are numbered by first appearance through a parallel sort instead of an
unordered_map of millions of keys (NumberByFirstAppearance).
- CPU path: the fulls scaling loop's group means sum over the CSR on all threads
(ReduceFullGroupMeans replaces the serial ReduceGroupMeans), the rescale runs on all threads.
- The error-model sample download goes into uninitialised arrays (no serial zeroing).
Measured (perf-rsm-explore-wip prototypes, GPU RTX 5080, box load 6-14, Run-span medians):
MergeAndStats-side sections 8tyy -3.5 s, 8a1a -2.0 s, cytc -0.25 s. md5 of p.mtz identical
to production rc174 on myob/cytc/thau/8a1a/8tyy (GPU), myob/cytc (CPU), and at -N 4/8 (GPU
cytc, CPU myob).
Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
Every full-statistics rotation merge computed French-Wilson amplitudes - the
search candidates, the pinned low/high pair, the final, and the P1
cross-check - although only the written merge's amplitudes are used, and those
were recomputed anyway with the anisotropic prior. RotationScaleMerge gets
SetFrenchWilson(); Rugnux turns it off for its merges and makes the amplitudes
once after the anisotropy analysis (anisotropic prior when a tensor was
fitted, isotropic otherwise). --mode scale keeps the engine's own amplitudes.
p_P1.mtz is now intensities only (owner decision): the F/SIGF/F(+)/F(-)
columns are omitted; every intensity column is bit-identical to before.
French-Wilson was 29% of the CPU samples in the large-cell tail (8tyy P1
cross-check window, ~84 core-s). p.mtz md5 unchanged on myob, cytc, thau,
kdp, 8tyy (GPU build). Wall: 8tyy 173.9 -> 150.9 s; cytc/thau -0.2..-0.3 s,
myob unchanged (3 interleaved repeats).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
* Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results.
* Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results.
* Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior.
* Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals.
* Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4).
* Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery.
* jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies.
Reviewed-on: #84
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports.
* jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls.
* Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results.
* Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable.
* Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate.
* Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do.
* Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence.
* Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags.
* Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check.
* Rugnux: Clear error messages when a data set needs more GPU or host memory than is available.
Reviewed-on: #83
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection.
* Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100).
* Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused.
* Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image.
Reviewed-on: #82
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked.
* rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results.
* `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry.
* `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel.
Reviewed-on: #80
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases.
* rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures.
* rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination.
* rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands.
* rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings.
* The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`).
* `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed.
* Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself.
Reviewed-on: #79
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results.
* A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell.
* rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`.
Reviewed-on: #78
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
