Image buffer: the per-image CBOR metadata headroom had been re-derived from the
online reflection cap alone, which cut it from 4 MiB to 2.55 MB while the measured
worst case - reflections plus the capped spot list plus the three azimuthal arrays -
is 2.9 MB, so the receiver dropped the frames with the most to say. Restore it and
give it a name that both the code and its guard test read: written down twice, the
two had drifted and the test kept passing against the value the code had left.
Spot finding: an unset low_resolution_limit means no limit at that end, as an unset
high_resolution_limit already did. An optional rather than a zero sentinel, because
zero is not a natural "no limit" here - every pixel lies above it, so the plain
comparison masked the whole image instead of none of it, and nothing validated the
zero. The API field is no longer required; a zero is folded into the unset case at
the boundary, where older clients still send it, so one spelling reaches the
analysis code. The FPGA takes its fixed-point ceiling instead, since ap_ufixed<16,9>
wraps above 512 A and would have masked everything.
image_preprocessing: check the CUDA calls on the fused decode path - the one new GPU
file with none, and the path fed by bytes we did not produce. An unchecked
synchronise returned the host-written sentinel as if it were a measurement, so the
decode looked successful and the fallback to the host decoder never fired.
rugnux: --stride no longer writes one past the end of the per-image arrays, whose
count floored where the worker loop ceils, and the written process file links the
images actually processed rather than the first N - each frame's picture now sits
next to its own analysis.
Powder calibration: the face-centred calibrants no longer list their systematically
absent rings, so the distance fit starts from a reflection that exists rather than
an extinct one; the triclinic calibrant covers both signs of h and k instead of a
single octant, which is only valid for a diagonal metric. The test asserted the old
behaviour - one ring formula for every cubic standard - and is rewritten.
CBOR: skip an unknown tagged value in the end block, as the other four blocks
already do. One advance lands on the tagged item rather than past it, so an older
reader fed a newer end message threw and never finalized its file.
Viewer: a settings value the setter rejects no longer escapes as an uncaught throw
from a worker slot, and the field offers only what the setter accepts.
Space-group search: judge stage B on the same "present" cut stage A already computes.
Merged sigma is floored so no reflection reads above ISa, so on a low-ISa merge the
fixed cut left both stage B tests unsatisfiable - every screw axis passed unchallenged
and the centering rescue switched itself off on exactly the weak data it exists for.
Where the fixed cut is the smaller of the two they are equal and this is inert: over
the 37-crystal rotation battery every crystal reports the identical space group and
identical merge statistics, so it is a no-op there and the low-ISa case it targets
remains unmeasured.
rugnux: --polarization reaches --mode azint, which parsed the flag and then dropped
it; that mode also applies the same polarization default as every other mode.
Acknowledge the ACTS/traccc project, whose sparse connected-component labelling both
spot extractors take their algorithm from, with its citation and its license.
The rc.161 change list is brought back to one line per entry, and the user-visible
changes that were missing from it added.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The GPU engine copies six small per-ring arrays back to the host every frame - the
clipped raw sum/sum2/count that the threshold is computed from, and the plain
corrected sum/sum2/count that become the azimuthal profile. They were plain
std::vectors, so the copies landed in pageable memory, and a device-to-host copy
into pageable memory blocks the calling thread until it has completed whatever
stream it was issued on. The profile snapshot sits between the plain pass and the
two sigma-clip passes, so Detect() stopped there and the device then sat idle while
the host caught up and enqueued the rest.
Register them, as AzIntEngineGPU already does with its own, and the copies are
genuinely asynchronous. Measured on a 4.5 Mpixel frame: 0.647 -> 0.621 ms per
frame. Nothing else changes - the spot list and the profile are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removes azim_int_settings.sigma_clip / rugnux --azim-sigma-clip and the clipping
machinery in AzIntEngine. This is a partial revert of a6be35ccd - the ice-ring-mask
removal that commit also carried stays. Sigma clipping remains where it started and
where it is needed: inside the adaptive spot finder, at a fixed 3 sigma on raw
counts, feeding the detection threshold and the ice score.
The option made the workflow harder to reason about than the quantity was worth. It
gave azimuthal integration two meanings behind one setting - the bin mean and the
background under the peaks - which the azimuthal-integration workflows do not need.
It also did not compose with the fused GPU engine, which supplies the profile from
its PLAIN pass: on the default rugnux, viewer and receiver path the setting was
silently doing nothing (measured, the profile came out identical to the unclipped
run to 1e-6 with identical per-bin pixel counts). Making it correct is not a matter
of gating that one shortcut - it means separating the workflows (azimuthal
integration, MX rotation, MX stills, geometry calibration) and deciding per workflow
what the profile is for, which is a larger change than the option earns.
The default path is unaffected: over 20 images of a rotation dataset the radial
profile, the per-bin pixel counts and the spot counts are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The changelog is for users. It is not a place for commit messages or developer
notes, so: one line per entry, say what changed rather than why or how it was
arrived at, and no sample identities. Rationale and measurements belong in the
commit message.
Pushing gets its own section rather than a line inside Test, since it is a rule
about the repository and not about running the suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The same frame mismatch as the rot2/rot3 fix, in the other two fields. Our pixel
coordinates are pixel-centred - 948.0 is the CENTRE of pixel 948 - while pyFAI
measures from the edge of the sensor and puts the centre of pixel i at (i + 0.5) *
pixel size. Poni1/Poni2 went out as beam * pixel size, so anything reading the
file placed the pattern half a pixel (37.5 um at 75 um pixels) off ours. The
previous commit's "Poni1/Poni2 need no such change" was right about the axis
directions and wrong about the origin.
The proof was already in the tree. The pyFAI reference values in
DiffractionGeometryTest were computed for a .poni with Poni2: 0.150 and a 75 um
pixel, which the tests translate to beam_x = 2000 - but pyFAI's numbers are
reproduced only at 1999.5. At 2000 every one of them is out by 2.6e-3 nm^-1, which
the 1e-2 tolerance hid. The tests now use the beam centre those headers actually
mean, and agree with pyFAI to 1e-6 - float precision - across untilted q, azimuth,
rot1, rot1+rot2, rot3, rot1+rot2+rot3 and the solid-angle correction. Tolerances
drop to 1e-4 (1e-5 for solid angle): ~100x the observed float noise, and 26x
tighter than the half pixel they were blind to.
The viewer's calibration window printed "PONI x = ... mm" from the un-offset value
beside the path of the file it disagreed with; it now matches the file.
Also moves the viewer's beam-centre cross half a pixel down and right, where the
spot, prediction, top-pixel and saturation markers already are. Our coordinates
are pixel-centred and the Qt scene's are pixel-cornered, so the map between them
is +0.5, and DrawBeamCenter was the one overlay missing it.
The convention itself is now written down in docs/DETECTOR_GEOMETRY.md, with the
conversions to XDS ORGX/ORGY and to the edge-of-sensor programs, this being the
second bug to come out of it.
Only exported and displayed values change; the fitted geometry, spot positions and
integration were always self-consistent. A .poni written by an earlier build is
half a pixel off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The .poni file carried rot2 and rot3 with our sign, which is not pyFAI's. pyFAI
has the slow axis increasing bottom to top; the MX convention runs top to bottom,
so the two frames differ by a reflection in y. Conjugating a rotation by a
reflection gives R(n, theta) -> R(Mn, -theta), so for rot2 (about x) and rot3
(about the beam) the sense reverses, while for rot1 the axis IS y and the axis and
the sense reverse together and cancel. Negate the first two, leave rot1 alone.
Poni1/Poni2 are unaffected: they are distances from pixel (0, 0) along each axis,
which the direction the axis runs in does not change.
Caught by integrating a LaB6 image in pyFAI with the file we had just written.
Unflipped, the rings come out BROADER than they do with no tilt at all - peak
height 42 against 30, mean ring-position error 0.0045 1/A against 0.0027 - which
is the signature of a tilt applied the wrong way. Flipped, they sharpen to 132 and
0.0005, and flipping rot1 as well makes it far worse (peak 5), so the asymmetry is
real and not a fitting artefact.
The unit test pinned the old signs, so it passed throughout. It now pins the
verified ones and says why, since the stored values and the written ones
disagreeing looks like a bug unless the reason is written down.
Only the exported file was wrong. Nothing internal changes: the fitted geometry
and everything downstream of it in Jungfraujoch were always self-consistent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The powder panel could only calibrate the image on screen. Calibration is now a
third page beside MX and AzInt, so the dataset button runs it over every image the
same way it runs the other two - which is the point, since a powder ring is
measured far better by summing a run than by one frame.
The page carries the calibrant and the method (rings or spots); the interactive
Guess/Refine buttons stay where they were and now share the one calibrant
selection, so there is no second combo to drift. analyzeDataset() carries the
ProcessMode rather than a bool: a third state was coming, and two bools would have
had one combination that cannot be valid.
The calibrant list gains ICE, which it could not offer before: the widget worked
in unit cells, and hexagonal ice has none that generates its rings correctly
(P6_3/mmc would include systematically absent ones). FindCenter now takes the ring
list its first line used to derive, so the interactive path gets ice as well.
The result window leads with the residual rms rather than the fitted sigma. The
sigma is a formal scatter estimate and understates a bad fit badly - measured on
ice, 0.215 px reported against a 1.70 px residual - while the rms separates a
usable fit from one that has locked onto the wrong thing.
A rings run needs the profile binned in azimuth; below four sectors it returns
nothing at all. The viewer raises the count to 32 exactly as the CLI does, and
says so in the panel and in the job dialog rather than doing it silently.
Also fixes a CLI inconsistency this comparison exposed: rugnux's calibration
branch never applied the standard offline analysis defaults, so it measured the
rings in a profile built with the file's polarization factor while every other
mode - and the viewer - uses 0.99. Found because the two disagreed by 0.005 px in
PONI x, and confirmed by reproducing the viewer exactly with --polarization 0.99.
With it applied the CLI and the viewer write byte-identical .poni files on LaB6
by rings, LaB6 by spots, and an iced dataset over 1800 images.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Belongs with the previous commit - powder calibration reads the ring positions
off the accumulated profile, and without this accessor rugnux/Rugnux.cpp does not
compile. It was left out of that commit by a staging mistake, not by intent.
GetAzIntProfile() flattens the profile to an array; the calibration wants the
object's own GetResult(), which leaves a bin no pixel fell in as NaN. Flattened to
zero, such a bin reads as a deep hole in the ring rather than as no measurement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--azint-only and --scale are replaced by --mode mx|azint|scale|calibration, with
mx the default. The old flags are removed rather than aliased.
Calibration mode fits the detector geometry - PONI x/y, the two tilts and the
distance - to a calibrant's powder rings and writes a pyFAI .poni alongside a
report of how far each parameter moved from the header. Bragg data constrain the
beam centre worst, because it is gauge-coupled to the crystal orientation; a
powder ring has no orientation to couple to.
--calibrant takes lab6, agbh, ceo2, si or ice. A calibrant is a list of ring
positions rather than a unit cell, because hexagonal ice is P6_3/mmc: rings
enumerated from its cell would include systematically absent ones. So the
crystalline standards generate their rings from a cell and ice carries the
measured list, and RingsFromAzimuthalProfile, GuessGeometry and OptimizeGeometry
all take ring q. The calibrant table is shared with the viewer's powder panel,
which previously carried its own copy.
--calibration picks how the rings are measured: rings (default) sums the
(q x azimuth) profile over every processed image and fits the arcs in it; spots
pools the found spots and fits those. Both use the whole run, with -s/-e/-t
selecting images. rings defaults --azim-phi-bins to 32, since a profile with one
azimuthal bin has averaged the ring over every direction and cannot locate it.
Two fixes this exposed:
The extraction window is capped at half the gap to the neighbouring ring. The
background under a peak is taken from the ends of its window, so a window wider
than half that gap measures the next ring's flank as this ring's background -
and hexagonal ice has three rings within 0.06 1/A. Ice calibration was 3.5 px
out before this and 0.29 px after; LaB6 is unaffected.
RingOptimizer holds rot1/rot2 fixed when only one ring is present. A tilt and a
centre offset both move a ring as cos(phi) and are separated only by the tilt's
amplitude growing as the ring radius squared, so on a single ring they are
exactly degenerate.
Measured. LaB6 at five distances: the fitted direct beam is within 0.36 px of an
independent implementation out to 300 mm, and D = -0.046 + 1.000788 dtz with an
rms of 0.011 mm. At 500 mm one ring is fully on the detector and a second only
clips the corners, which is not enough to constrain a tilt - restricting the q
range to the resolved ring recovers 0.06 px. Ice: 5.53 -> 0.29 px on one crystal
and 4.71 -> 0.80 px on another, against XDS's refined direct beam. On an ice-free
crystal the fit is worse than the header, which is the correct outcome.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CPU_DATA_ANALYSIS describes how the pipeline works; the account of which bar was
tried first belongs in the commit that changed it. Same facts, no narrative.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three things had reached the code without reaching the documentation.
The azimuthal sigma clip had a RUGNUX.md row and a CPU_DATA_ANALYSIS section but
no changelog entry - and the only "sigma clip" the changelog mentioned was the
background ring's, which is a different thing at a different stage.
--index-ice-rings was in the options table but nowhere in the changelog, so the
entry describing the ice gate still implied that whether indexing uses the
ice-band spots is decided per run, which it no longer is.
CPU_DATA_ANALYSIS section 6 still described the Bravais class as simply "the
highest-symmetry class that matches within tolerances", which is the behaviour
that lost a crystal outright. It now records that the class is chosen from the
UNREFINED candidate against a fixed angular tolerance, that a pseudo-symmetric
lattice therefore gets promoted a class too far, and that the first pass settles
it on validation-frame counts with a clear-majority bar - including why the bar is
a majority rather than a margin, since that distinction is the whole reason the
check is safe. It also records that the first pass finds its own spots rather than
reading the acquisition's, which was not written down anywhere.
Also a build note: M_PI is not standard C++ and MSVC does not define it, so the
Bragg integrator's use of it broke the Windows viewer build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
update_version.sh at 1.0.0-rc.161. The only substantive change is the one that had
drifted: the spot-finding ice-ring half-width was still documented as 0.02 in the
generated Python client and its docs while broker/jfjoch_api.yaml has said 0.03
since the band was widened to the measured ring FWHM. Anyone reading the client
docs - or relying on the client's default when omitting the field - got a band
two-thirds the width the pipeline actually uses.
The TypeScript frontend client regenerates identically (the spec itself did not
move), and python-client/ is not tracked here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Auto rode in with the ice work rather than on its own evidence, and measured over
the 37-crystal rotation battery it does not carry itself yet. It TARGETS
correctly - it fires on ten crystals and every one is ice-positive, no failures,
no space-group changes - but it costs 1.35x the wall clock (median +3 s per
crystal, worst +29 s) and on the merge statistics it is the familiar sign-mixed
trade: high-shell CC1/2 worse on three of the four crystals that move materially,
mean -0.76.
The case for it is real but rests on agreement with a fixed external model - 43 %
of the ice bands' excess amplitude removed on smooth ice, the effect 7x stronger
inside the bands than outside - which is the better arbiter and also the narrower
one. That deserves settling on its own, not riding along with a set of ice
defaults. `--background-radial=auto` keeps it a flag away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The "radial curvature correction on/off" line was printed only when
--background-radial was given explicitly. In auto - the default - it said nothing,
so a run's log carried no record of whether the correction had been applied.
That is not cosmetic. Auto decides per image from that image's ice score, so two
runs of the same data with different flags can differ substantially with nothing
in either log to explain it: a crystal whose high-shell CC1/2 read 8.1 % with the
correction pinned off and 4.5 % under the default looked like a regression for
some time before the flag turned out to be the whole difference.
Log the effective mode unconditionally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The profile is the MEAN of each bin, so a few strong reflections landing in a bin
lift it exactly as a smooth powder ring does. That is the wrong quantity whenever
the profile is wanted as a background rather than as a measurement of what is in
the bin - the ice score being the case in point, where reading a plain profile
INVERTED the metric: over 37 rotation crystals the two highest-scoring crystals
had no ice at all.
The adaptive spot finder already computes the right thing, a sigma-clipped
per-resolution-ring background, as a byproduct of its own threshold. Where it
runs, the ice score uses that. Where it does not - --no-adaptive-spots,
--azint-only, and anything reading the profile the broker wrote - there was no way
to get it. This adds one: azim_int_settings.sigma_clip (rugnux --azim-sigma-clip),
0 = off, minimum 2 because a tighter clip rejects a large part of a clean Gaussian
bin and biases the estimate low rather than removing outliers.
Two clip passes follow the plain one, matching the finder's recipe - the first
pass's standard deviation is itself inflated by the peaks being removed, so one
pass leaves a threshold that is still too generous. A bin with fewer than eight
pixels is left alone: at the detector edge and behind the beam stop there is no
spread to clip on.
Both engines do it. On the GPU the accept range is computed by a small kernel and
stays resident, so a clip pass is one more read of the same pixels and no round
trip; the two accumulation kernels take the range as a pointer that is null on the
plain pass. Measured on a JUNGFRAU rotation dataset, non-adaptive path: azimuthal
integration 0.02 -> 0.06 ms per image, exactly the 3x the extra passes predict,
against a 0.34 ms per-image total.
Note what the result IS: the smooth background under the peaks, not the bin mean.
It should not be switched on where a ring's integrated intensity is wanted - the
powder-ring geometry fit reads ring peaks, and those are what a clip is designed
to remove. Off by default, so nothing changes unless it is asked for.
Not exposed over the REST API - that needs the generated model regenerated, which
is a separate step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Changelog entries for the three changes above, and a new CPU_DATA_ANALYSIS
section on determining detector geometry from powder rings: why a ring is an
independent constraint on the beam centre (it has no crystal orientation to be
gauge-coupled to, unlike everything else that fits geometry here), what a ring
can and cannot determine, and how the ring points are obtained.
The section states the harmonics correctly, which is worth writing down because
the intuitive version is wrong: a detector tilt shows up as cos(phi), the same
harmonic as a beam-centre error, and the two are separated by the amplitude
growing as the ring radius SQUARED - so it takes at least two rings, and on one
ring they are exactly degenerate. The genuine cos(2 phi) term is hundredths of a
pixel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Bravais class is decided from the UNREFINED FFT candidate against a fixed
3 degree angular tolerance (LatticeSearch). A lattice that is pseudo-symmetric to
a few tenths of a degree is therefore promoted a class too far, and the constraint
then snaps a real angle to the ideal one - which throws nearly every reflection of
every frame out of tolerance. Measured on a monoclinic crystal that is
pseudo-C-orthorhombic to 0.42 degrees: the promoted cell indexes 2 of 60
validation frames and the run dies, where its own primitive cell indexes 39. It is
the same lattice in a different setting, b_oC = -(a + 2c), volume exactly 2.00x.
The perverse part is that BETTER SPOTS MAKE IT WORSE. LatticeSearch applied to the
true cell returns the promoted class deterministically; runs that succeed escape
only because the raw FFT candidate is inaccurate enough to miss the promotion
window. So it is bistable and non-monotone in every knob - 190 spots per image
gives 44/60, 195 gives 12/60, 200 gives 2/60 - and it will bite harder as spot
finding improves.
The indexer already refines a free triclinic cell alongside each constrained
candidate, but decides between them on the fraction of the accumulated first-pass
cloud that indexes, where the two differ by less than a factor 2 (measured 0.243
vs 0.135, missing both of that guard's bars). The caller has a far sharper
statistic: it already counts how many of 60 validation frames a candidate indexes,
and there the same pair differs by more than 20x. So keep the triclinic cell
instead of dropping it, and let the first pass settle it.
The bar is a clear majority, not a margin, and that is the part that took a
battery to get right: the unconstrained refinement holds NO cell parameter fixed,
so it can only index at least as many frames as the constrained one, and on
genuine symmetry it does index a few more. A 10 % margin - the bar a later scheme
needs to displace an earlier one - demoted a real I-centred orthorhombic crystal
to P1 (47 -> 54 frames) and perturbed an F-cubic one (49 -> 58). Only a
constrained cell that fails outright while its unconstrained cell works is
evidence of a false promotion, so demand exactly that. It is the same "fails to
index half the frames" test the long-axis rescue below already uses.
Battery over 37 rotation crystals: 33/37 space groups matching XDS with one hard
failure becomes 34/37 with none, and the other 36 crystals are identical in every
printed statistic (checked against a repeat run of the previous binary, which
itself differs on one crystal by one observation). The extra validation pass runs
only where the constrained cell already failed - 71 ms in a 15 s run - and not at
all on the 34 crystals whose constrained cell indexes a majority.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A detector tilt does NOT appear as a cos(2 phi) modulation of the ring radius, as
the previous comment claimed. To first order a misalignment beta gives
r(phi) = R + (R^2 / F) (beta_x cos phi + beta_y sin phi)
which is a cos(phi) term - the same harmonic a wrong beam centre produces. What
separates them is the radius dependence: the centre's amplitude is the same on
every ring, the tilt's grows as R^2. So they are told apart across rings, not
within one, and on a single ring they are exactly degenerate. Measured on a powder
standard the true cos(2 phi) term is of order R^3 beta^2 / F^2 - hundredths of a
pixel, at the noise floor - so it carries nothing usable.
Also add the tilted round trip, which was missing. It doubles as a check that
RingOptimizer's open-coded rotation agrees with DiffractionGeometry's: the fitter
applies Rx(-rot2) Ry(+rot1) by hand rather than going through the geometry's
Rz(-rot3) Rx(-rot2) Ry(+rot1), and those had never been held against each other.
They agree - 0.020 / -0.015 rad recovered as 0.0197 / -0.0148. Dropping rot3 is
right rather than an omission, since rings cannot constrain in-plane roll.
The tilted case yields fewer ring points than the centred one, which is expected
and worth knowing: the extractor searches a window centred on where each ring is
EXPECTED, so a large enough geometry error carries part of a ring out of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ring calibration already here (AssignSpotsToRings + RingOptimizer, driven from
the viewer's powder panel) is given a SPOT LIST from a single image. A powder ring
is not a set of spots - it is a smooth arc - so a spot finder samples it wherever
its threshold happens to bite, and one image carries only the counts that image
collected. An azimuthally-binned profile summed over a run measures the same ring
directly, at every azimuth, with the whole run behind it.
RingsFromAzimuthalProfile turns such a profile into the (x, y, q_expected) triples
RingOptimizer already consumes, so nothing downstream changes: for each calibrant
ring and each azimuthal sector it fits the radial peak against a locally
interpolated background, and maps the measured (q, phi) back through the current
geometry to the pixel it came from.
What this is for is the BEAM CENTRE. A powder ring is a conic centred on the beam,
so a wrong centre makes its apparent radius oscillate once per turn and a detector
tilt twice - and neither depends on the calibrant's d-spacings or on the detector
distance. That matters, because the beam centre is otherwise the weakest parameter
we have: fitted from Bragg spots it is gauge-coupled to the crystal orientation,
which is why PostRefine has to restrain it toward the header and commit only a
sub-1 % move, and why XtalOptimizer carries a soft prior noting the beam is "only
LaB6-monitored to ~a few px". A ring does not know about the crystal.
Two things the peak fit is careful about, both of which would otherwise show up as
a spurious cos(phi) - i.e. as a beam-centre shift:
- the sector's CENTRE is used, not its lower edge. GetBin() floors phi into the
sector, so a bin stands for [j, j+1), and taking its edge rotates every ring
point by half a sector.
- a peak has to stand clear of the scatter of the background either side of it,
or a sector with no ring in it contributes its largest noise excursion as
though it were a measurement.
Refuses a single-azimuthal-bin profile outright: that is a plain radial profile,
the ring has been averaged over every direction, and there is nothing left to say
where its centre is.
Tested by round trip against the forward model, as the existing calibration tests
are: synthesise the profile the azimuthal integration would build with the rings
where a shifted geometry puts them but every pixel binned with the unshifted one,
then extract and fit. A 6.0 / -4.0 px beam offset is recovered as 6.13 / -4.03
from 192 ring points. Only the beam centre is exercised here; the tilt path is
covered by the existing DetGeomCalibTest round trips.
This is the extraction only - nothing calls it yet, and the run-scoped accumulator
it is meant to read (JFJochReceiverPlots::az_int_profile, already summed over a run
and written to /entry/azint/dataset) is still integrated with one azimuthal bin by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Two-pass rotation indexing found only a lattice that indexes 2/60 validation
frames" tells a user that something went wrong and nothing about what. The
commonest cause is a metric symmetry promoted one class too far: the constrained
cell then misses every reflection by the small angle the constraint snapped
away, and the run dies with no way to see that a centred supercell setting was
chosen over the primitive one it should have kept.
Print the centring, crystal system and cell that was rejected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
M_PI is not standard C++. MSVC defines it only when _USE_MATH_DEFINES is set
before <cmath>, so the radial background kernel's azimuth loop does not compile
there:
error C2065: 'M_PI': undeclared identifier
error C2737: 'phi': const object must be initialized (cascade from the first)
GCC and Clang define it anyway, which is why the Linux build stayed green.
image_analysis is viewer-reachable, so it has to build under MSVC.
common/JFJochMath.h already carries a constexpr PI for exactly this reason - its
comment names this case - so use that. Same value to the last digit, so the
integration results are unchanged; the CPU/GPU parity test passes unaltered
(9002 assertions).
This was the only M_PI left in the viewer-reachable tree. The remaining uses are
in tests/, which Windows does not build (JFJOCH_VIEWER_ONLY is forced there).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ice handling was gated on a measurement the run only made AFTER the images had
been processed, so the per-image pass could not use it. The flagging therefore
ran unconditionally: ice-band spots were ordered last in the --max-spots budget
and held out of the indexer seed and the geometry refinement on every crystal,
iced or not. The eleven bands are fixed geometry holding 16-26 % of the unique
reflections whether or not there is ice, so on a clean crystal that discards a
fifth of the spots - the strongest first - for nothing. Measured on a crystal
whose gate never fires, that moved the merged data by a mean of 0.85 sigma
against a run-to-run floor of 9.3e-5.
Measure it in the first pass instead. That pass already looks at ~100 images
spread over the sweep, and it already stops at the spot finder, so it sees the
azimuthal profile for the smooth channel and the unfiltered connected components
for the spot channel. Both counts SpotAnalyze takes are pre-filter, so pooling
them there is the run's own verdict, reached before anything has been discarded
and in time for the pass that acts on it. Where the sample sees no ice, the run
indexes on the ice-band spots too.
It has to be the whole sample: the spot channel is a ratio pooled over images,
because one frame carries a handful of control spots. A per-image gate is not an
alternative - two of the crystals whose indexing this rescues fire on that
channel alone, at profile scores of 1.12 and 1.22, so gating per image on the
profile score would drop exactly the cases that matter.
This also removes the first-pass spot reuse, and with it --redo-rotation-spots
and the reuse path. Finding the ~100 first-pass spots costs little, and reusing
was actively wrong here: the stored spots were found online at the acquisition's
threshold and have already had their ice-band entries ordered last and dropped
by its spot budget, so counting ice from them under-reads it by construction,
and the lattice search never saw the spot-finding settings at all. It also
removes the need for the machinery that re-found spots whenever a spot-finding
option was named, which made those options impossible to A/B.
IndexAndRefine cached index_ice_rings at construction, which happens before the
first pass; it holds a reference to the experiment, so it now reads the setting
where it uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First-pass rotation indexing reuses the spots the acquisition wrote, and does
NOT re-mark them - their ice flags come from the file. So --detect-ice-rings can
only invalidate those spots when it asks for something the file did not do.
It was flagged as a spot-finding option, which forced reuse_rotation_spots off
whenever it appeared, so naming it swapped the acquisition's spots for this
program's own and moved the first-pass lattice by itself. Measured over the
37-crystal rotation battery, passing the SEMANTICALLY NULL --detect-ice-rings=on
to files that already carry detect_ice_rings=1 changed the merged data on every
crystal, sent one crystal's ISa from 1.66 to 0.38, and lost MyoB_13 to indexing
failure outright. Both arms of any A/B on the flag therefore moved for a reason
that had nothing to do with ice, which made the flag impossible to test.
Re-find only when the requested value differs from the file's, and say so when
it happens. A file with no key at all counts as "did not mark", which is what
its stored spots show - such a dataset carries no per-spot ice flags to reuse.
Verified over the battery, with the merge mask and the radial background pinned
off so this is the only variable. --detect-ice-rings=on on the 36 keyed
crystals: all 36 log "using the spots stored in the file", the re-finding line
appears nowhere in the arm, unique counts are identical on every crystal, the
largest mean |dI|/sigma is 3.4e-5 against a repeat-run floor of 9.3e-5, and not
one of R_meas, CC1/2, CC1/2_hi, ISa, completeness, SigAno, d_min or space group
differs anywhere. MyoB_13 indexes again. The one file carrying no key reuses
under =off and re-finds under =on, as it should.
--detect-ice-rings=off still re-finds, since it does differ from those files,
and two crystals still fail to index there. That is not this change: a control
that re-finds with ice marking ON indexes both. With the marking off, ice spots
are no longer ordered last, so they consume the --max-spots budget and the
first pass collapses.
With the confound removed the flag can finally be measured, and on a comparison
whose spot source is identical on both arms it is clearly worth having - though
the win is at INDEXING rather than at the scale fit. Three crystals are saved
outright (one would otherwise collapse to P1 at CC1/2 19%, one loses half its
completeness and its screw axis, one loses its F-centred cubic lattice), two
more only index with it on, and the remaining eight gate-fired crystals differ
by well under 1% in R_meas and CC1/2 in both directions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three defaults, each settled by measurement rather than by argument. The
arbiter throughout is structure-referenced - anomalous peak height where a
crystal can carry it, and otherwise the agreement of the ice bands with a fixed
external model against resolution-matched DECOY bands carrying no ice. The
band-versus-decoy contrast is used because R-free here tracks completeness, and
every one of these switches moves completeness.
The damage is real and it localizes: over the rotation battery the ice bands'
excess amplitude reaches +9.6% on a smooth-ice crystal and +35% on the worst,
while a clean control sits at +0.6% (z +0.45). On the worst crystal, nine of the
ten largest excess peaks in a q scan land on hexagonal ring positions. Turning
ice handling off leaves the contrast unchanged and forcing it on a clean crystal
does not create one, so it is the ice and not the machinery.
MERGE-TIME RING MASK -> OFF. It deletes reflections, which no other program does
by default - AIMLESS, DIALS, xia2, XDS and CrystFEL all keep ice-band
reflections in the merge and exclude them only from the model fit; autoPROC is
the sole exception. On the one battery crystal where the mask fires and an
anomalous arbiter can score it, dropping the band moved the mean peak height at
the known sites by -0.001 +- 0.018 sigma, 2% of the site height, while removing
1149 unique reflections whose mean I/sigma was 3.62 against the dataset's own
3.05 - better than average data - and costing 17 completeness points in that
shell. It fires on 5 of 37 crystals, changes no space group, and those 5
disagree in sign: it clearly helps the two most heavily iced, is a wash on two
and costs a third. So it stays as a switch, worth setting by hand on a badly
iced crystal where it shows in the high shell, but it is not a default.
RADIAL BACKGROUND -> AUTO, gated per image. The correction models the background
as a function of radius alone, and that is exactly when it works. On a crystal
with pure smooth powder ice it removes 43% of the bands' excess amplitude, with
the improvement 7x larger inside the bands than outside; on a crystal whose ice
is discrete crystallite spots - no smooth ring to model - the excess amplitude
GREW by half; on clean data it is inert to four decimals. The two ice channels
already separate those morphologies, so --background-radial takes on|off|auto
and auto applies it to an image when that image's peak-excluded score reaches
--ice-min-score. Auto never engages without such a score, because the plain
profile carries the Bragg peaks and cannot support an absolute threshold.
Per image rather than per run, and that was tested rather than assumed: the
gate fires on 100% and 94% of frames on the two crystals that want it, and on
1.5% of frames - 32 blocks, 23 of them single frames - on the textured-ice
crystal. A seam statistic against off + f*(on - off) is null on both mixed runs,
every merge statistic is bracketed by the pure arms, and the textured crystal's
auto arm lands on `off` rather than on `on`'s harm. A run-level gate would need
the score before the pass that integrates, i.e. rotation-only plumbing, and buys
nothing measurable.
The kernel was already built unconditionally, so flipping the flag per image is
free - except on the GPU, where the launches were gated on a construction-time
n_rad. That is why the buffers are now allocated whenever the correction could
run, and Run() decides per image.
DETECTION -> the geometry's default when the file is silent: on for rotation,
off for stills, with the command line and then the file taking precedence. A
rotation sweep sits on the same rings for the whole run, so ice there is a
coherent systematic and the presence gate keeps it inert on a clean crystal; a
serial stills run has too few spots per image to spend any on flagging. The
master file's key is kept as written rather than collapsed to a bool, so "the
file said nothing" is distinguishable from "the file said no" - it used to fall
silently to off, taking the exclusion from the scale fit with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Whenever the merge-time ice-ring mask dropped a band, the per-shell observation
count and hence the reported multiplicity were wrong. On one crystal the lowest
resolution shell read 40780 observations over 1932 unique reflections - 21.1x -
where the truth is 27007 and 13.98x, and the overall redundancy read 12.52
against 12.29. Only counts were affected: intensities, sigmas, R_meas, CC1/2,
completeness and ISa were right throughout, because a masked group carries
merged_I = NaN and never enters those sums.
It looked like double counting and was not - it is a MOVE. Two independent
faults, both in three lines:
total_obs rides on the R_meas re-walk, whose filter deliberately ignores the
ring mask (and, on a search pass, the ice flag) so that R_meas is computed on
the same reflections either way. RmeasUsable therefore differs from MergeUsable
by exactly those two tests, and the observations they admit were being counted
against a `unique` that excludes them.
On the GPU path that count is binned by the GROUP's resolution, and a group
every one of whose observations is masked never has one written - acc[g].d stays
NaN. ResolutionShells::GetShell(NaN) then returned shell 0 rather than nothing:
NaN fails both bound comparisons, falls through to the arithmetic, and
static_cast<int32_t>(NaN) is INT_MIN, which the clamp maps to 0. So the masked
ring's observations were re-labelled into the lowest-resolution shell, four
shells from the ring they came from.
The two paths disagreeing on the same run is what settled it: with the mask on,
the GPU statistics gave shell 0 = 752 and the CPU statistics 423, while the
merged intensities were identical.
Count the merged population instead - acc[g].nh, which the merge already
accumulates per group - and guard the CPU increment with usable_merge. The
rnusable skip stays: any group present in the merged output has at least one
observation passing MergeUsable, and MergeUsable is a subset of RmeasUsable, so
it cannot drop a group that contributes to `unique`.
With the mask off and for_search false the two predicates are identical, so this
is provably inert on every shipped configuration - demonstrated on four
configurations, including one where ice handling is active but the mask does not
fire: the statistics blocks are unchanged. (The reflection lists differ in the
last ulp on 3-12% of lines, but so do two runs of the same binary; that is the
known rotation nondeterminism, and the statistics block is what is stable.)
The NaN guard also removes a silent contamination nobody was looking for. Four
call sites validate a resolution with `d <= 0`, which NaN passes: the Wilson-B
fit and per-shell <I/sigma> (CalcISigma), the per-image resolution plot
(SpotUtils) and the shell Wilson prior (FrenchWilson) were all binning
non-finite d into their lowest-resolution shell. French-Wilson now falls back to
the global mean rather than to that shell's, which is the worst prior available.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A stage that turns further than commanded is invisible in the file, because the stored
omega values ARE the commanded ones - both XDS and rugnux then read the discrepancy as
the crystal drifting. Measured on one dataset in 37, a ~1.2 % over-rotation costs it
unique 9.9k -> 29k and CC1/2 68 -> 98 % when corrected by hand.
No new degree of freedom is added, because the one needed is already there and being
thrown away: step A's residual rotates by -angle_rad * axis[] with axis an UNNORMALISED
3-vector, so the length it fits IS the factor by which the stage actually turned.
GoniometerAxis::Axis then normalises it away (with the `increment *= len` line sitting
commented out). This only reports it.
Guarded by the same cross-validation that gates the cell move - a fold that merely
soaked up noise cannot raise the flag - and by a 0.5 % tolerance, which is where a
direct scan of this factor puts 36 of 37 datasets (all at exactly 1.0000). The known
fault reads 1.00604 and warns; clean controls read 0.99958 and 0.99954.
It UNDER-reads the true magnitude: the fit only sees reflections already indexed at the
nominal angle, per-frame orientation refinement has absorbed part of the error, and the
axis components are bounded. Treat it as a detector, not a calibration - nothing here
corrects the data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-image ice score was read off the PLAIN azimuthal profile. That profile is a
per-ring mean, so a few strong Bragg reflections landing in a ring's q bin lift it
exactly as ice would. Measured over 37 rotation crystals, that did not merely add
noise - it INVERTED the metric: the two highest-scoring crystals had no ice at all
(4.23 and 4.06), while a clean control read 1.57. A decoy null - the identical
statistic evaluated at q positions where hexagonal ice cannot be - reaches 1.51 at its
99th percentile and 2.70 at its maximum, so that metric cannot support any absolute
threshold whatsoever.
The adaptive spot finder already computes the right input for its own threshold: a
sigma-clipped per-resolution-ring background, in the same bins. A powder ring is
azimuthally smooth and survives the clip; Bragg peaks do not. On the clipped profile
the clean population tightens to 1.00-1.22 and the crystals with confirmed ice sit at
2.08-2.37, against a decoy null that never exceeds 1.29.
That channel is blind to one thing: ice in large crystallites diffracts as DISCRETE
spots and leaves the radial profile flat. So a second channel counts found spots on the
rings against the same q width of ice-free flanks beside them. The two barely overlap -
the smooth-ice crystals read 2.1-2.4 / ~1.0 and the textured ones ~1.1 / 3.8-17.6,
while a clean crystal reads 1.04 on both.
Both are then used as a GATE (--ice-min-score 1.5, --ice-min-spot-ratio 2.0, both
calibrated on the battery, 0 disables): the eleven fixed hexagonal bands cover 16-26 %
of the unique reflections at typical resolutions whether or not the crystal has ice, so
flagging, the exclusion from the scale fit and the merge-time CC1/2 ring mask are now
all skipped when neither channel sees any. The gate is applied in the full pipeline and
in --scale, which reads the stored per-image values back out of the _process.h5.
Also fixes the merge-time mask's control: the shoulder now excludes reflections that
are themselves on an ice ring. The rings are not evenly spaced - 1.947/1.916/1.882 A
sit 0.05-0.06 apart in q - so for those three the [w,3w) shoulder landed squarely on
the neighbours and the test compared ice against ice. Measured, that is the only thing
this changes: it removes firings on those three rings and leaves every other firing's
CC pair identical to three decimals.
And the online ice half-width, which was 0.02 in the API against 0.03 offline, so the
same data got a narrower band online than the measured ~0.06 ring FWHM justifies.
Battery (37 rotation crystals, against the previous behaviour): space groups 34/37 in
both and NO crystal's space group changes; 6 crystals gain unique reflections, 1 loses.
Best of them gains 7082 unique reflections with R_meas 16.0 -> 14.3, CC1/2 95.9 -> 97.3
and ISa 13.7 -> 19.0; another goes R_meas 54.9 -> 42.9, CC1/2 84.0 -> 90.4, ISa
3.9 -> 5.5; a third reaches CC1/2 99.4 from 95.7 at an unchanged reflection count. The
one crystal that loses reflections improves on both R_meas and CC1/2.
Not done here: the ScanResult/API/plot-type/frontend/viewer layers for the new
spot_count_ice_control (they need the OpenAPI regeneration). Message, CBOR, HDF5
write/read and the receiver plots are.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two independent pieces in the same code path.
The background-estimate variance was never propagated. A reflection's background comes
from a finite ring of n_b pixels, so subtracting it adds var(B)/n_b per signal pixel -
sqrt(1 + n_d/n_b) = 1.109 with the shipped stencil. Both engines omitted it, which is
exactly the 1.11-1.19 gap measured between the off-ring scatter and the reported sigma.
Three lines each; it affects every dataset, not only iced ones.
The radial correction is new and OFF by default (--background-radial). The signal disk
and the background ring are concentric, so for any background LINEAR in position
<B>_ann == <B>_disk identically and a plane fit buys nothing; the leading error is the
CURVATURE of the radial background, which on a sharp ice ring reaches +26 counts on a
single reflection. Since every reflection uses the same stencil, that error is a fixed
kernel over radial offset - one short dot product per reflection and no extra pixel
reads. Validated on empty apertures before any C++: mean |bias| over 9 bands / 3
crystals 4.33 -> 0.79 counts with the scatter unchanged.
Three things it cost a battery each to learn, all now in the code:
- the radial curve must be accumulated from CLIPPED annulus pixels, inside the clip
pass, or it carries neighbour tails and zingers (so it is inert under --integrator
boxsum, which has no clip pass);
- the GPU version was a 1.8x slowdown from atomicAdd contention on a small radial
array - staged in shared memory per block it now costs nothing measurable;
- it is battery-NEUTRAL as a default, because the reflections whose bias it fixes are
the ones the ice handling already excludes. Hence off by default.
CPU/GPU parity extended with two radial sections: 9002 assertions.
Also fixes a latent French-Wilson quadrature collapse: j_max = I + 8 sigma on a fixed
400-point grid degenerates to a single cell once sigma >> 50 <I>, giving F = 0.1 sqrt(sigma)
with sigmaF -> 0. Harmless today, but any sigma-inflation scheme detonates it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SmoothGeometry de-rotates each frame's lattice to a common reference, averages
in frame order and rotates back. A frame that never indexed keeps a
default-constructed CrystalLattice whose vectors are all ZERO - and zero is
finite, so the isfinite guard let it through. Those zero vectors were averaged
into their neighbours' smoothed orientation, pulling it toward the origin, and
they were scored in the leave-one-out cross-validation that picks the smoothing
window.
On a crystal where 374 of 900 frames fail to index, the effect on the window
choice is not subtle. Measured:
before n_scored 900 (only 526 indexed) CV score ~504-542 A^2 window +-12
after n_scored 516-526 CV score 0.160-0.175 window +-2
The score was inflated ~3000x and the choice among windows was noise. It settled
on +-12 frames - 9.6 degrees of goniometer rotation - on a crystal whose
orientation genuinely drifts by ~8 degrees over the sweep, so every partial's
delta_phi was recomputed from an orientation averaged across that drift.
Require a real cell. Exactly inert when every frame indexes, and no threshold is
touched.
The crystal that exposed it goes P1 -> P2_1, observations 60107 -> 77021,
completeness 64.1% -> 93.0%, multiplicity 1.10 -> 2.0, CC1/2 70.0% -> 84.9%,
R_meas low shell 37.3% -> 22.1%, and its 2-fold operator CC 0.330 -> 0.669,
comfortably clear of the 0.5 gate. Battery over 37 crystals: space groups
33/37 -> 34/37, and that crystal is the ONLY flip - no losses. Another crystal
is rescued from near-total collapse (4402 -> 139213 observations) because the
two-pass "going back to the header geometry" fallback stops firing. Anomalous
peak height +0.043 +- 0.022 sigma over 7 crystals, so the background clip's gain
is intact. Merged quality is otherwise neutral (CC1/2 6 better/6 worse,
R_meas_lo 9/6) with observations up on 18 crystals.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ANODE names its ranked peaks after the SFAC element it was given, so the labels
read S1, S2, ... on a sulfur case but FE1, MN1, ZN1, SE1 as soon as the model
carries a heavier scatterer. The parser matched "S" followed by digits, so on
any such dataset it dropped the ENTIRE peak list and the dataset failed the gate
as "no peaks" - however strong its signal actually was. One heme case reported
no peaks when its true top peak is 13.79 sigma at 3.02x the off-site floor.
The bug is silent and it mis-gates exactly the datasets most likely to widen the
arbiter set, since a heavy scatterer is what makes a weakly diffracting crystal
usable as an arbiter in the first place.
Byte-identical on the standing all-sulfur set, verified by diffing the gate table
before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A change that touches partiality - a mosaicity estimator, a rocking-curve
model, a background change - cannot be judged by the statistics we normally
reach for, and this was learned the expensive way. ISa is anti-correlated with
external accuracy and is the largest mover of any statistic; last-shell R_meas
moves with its denominator, i.e. the wrong way by construction; `--model`
R-free tracks its own zero-information floor, which shifts ~22x more than
R-free itself over the same sweep; and per-shell agreement with XDS is biased,
because XDS never divides by partiality, so "divide less" moves us toward it
mechanically - measured to put the optimum ~1.4x too low.
Anomalous difference density at known scatterer sites has none of those
problems. It is read in units of the map's own sigma, so the uniform intensity
rescale a partiality change produces cancels exactly, and it is referenced to
the structure rather than to another program's partiality model.
The script runs SHELXC + ANODE per arm against a model placed ONCE and then held
fixed, and reports the mean site height, the off-site noise floor, and the
paired per-site change between arms. Numeric arm labels turn a set of arms into
a curve with a per-dataset optimum. The dataset table lives outside the
repository, as rugnux_vs_xds.py already does for the battery, because dataset
and sample identities are not committed.
It reproduces the measurements it was built from: all nine points of three
pooled curves, every per-crystal optimum, the site heights, the paired t
statistics, and the adversarial control in which a model refined against the
worst arm reproduces the curves to <=0.005 and the same optimum on 4/4.
Four things the ad-hoc scripts it replaces got wrong, all now handled:
* Keying sites on the ANODE atom label silently drops an alternate conformation
sharing that label - one dataset class has 18 sulfur sites, not 17, and the
uncorrected mean read 15.03 against a true 14.49.
* The off-site floor skipped any peak within 1.0 A of ANY atom, so a ripple
sitting on a light atom was not counted as background; requiring 1.5 A from an
anomalous scatterer raises one floor from 7.12 to 9.48 sigma.
* Special-position peaks are Fourier ripples, not background. Excluding them is
load-bearing on 3 of 9 datasets and they are now reported in their own column
rather than dropped silently.
* Enantiomorph care turned out to be unnecessary - passing the merged file's
screw label to ANODE while the model sits in the other hand gives byte
identical peaks. What does matter is the pair whose absences are identical,
I23 vs I2_13, which phaser's automatic hand test does not cover.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The search already knows when several groups share an absence pattern - it
scores them identically, marks them all in the candidate table and prints
"Best space group: I23 or I213 (indistinguishable from these data)". The
one-line summary then dropped that and reported only the representative, so the
run's headline answer claimed a decision the diffraction had not made.
Carry the alternatives through to the summary. It already has them:
ProcessResult holds the whole SearchSpaceGroupResult.
Space group: I23 (No. 197) or I213 (No. 199) - indistinguishable from these data
Some of these pairs are enantiomorphs (P4_1 vs P4_3), where the choice needs
phasing or anomalous signal. Others are not, and are worth naming because they
surprise: I23 vs I2_13 and I222 vs I2_12_12_1 differ only by a screw whose
condition h00: h=2n is ALREADY implied by the I-centering condition
h+k+l=2n, so the screw has no observable signature at all. Checked over 35936
reflections with gemmi, the two absence patterns are identical - not nearly, but
exactly. Of the 65 chiral space groups, 13 classes are indistinguishable this
way, the largest being the four-way P3_112 / P3_121 / P3_212 / P3_221.
The representative stays the lowest space-group number, which is why a cubic
insulin comes out I23 where the deposited convention is I2_13. That choice is a
convention and the summary now says so instead of implying it was measured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mask drops a hexagonal-ice ring when its merged half-set CC1/2 falls a fixed
0.05 below its resolution shoulders. That margin is not a significance level: at
the populations these rings actually have, 0.05 spans 1.1 to 7.3 sigma across
firings, and a nominal Fisher-z error understates the real scatter of these
heavy-tailed intensities by ~2.7x, so the null has to be measured rather than
derived.
Measured it with decoy bands - the identical ring/shoulder statistic evaluated
at q positions carrying no ice ring - over the 37-crystal rotation battery: the
gap's empirical null is p95 +0.032, p99 +0.095. So 0.05 sits near the 96th
percentile, about 4% of ice-free bands clear it, and roughly half the 22
observed firings are indistinguishable from bands with no ice in them. The
firing gaps are continuous, not bimodal, with 12 of 22 in [0.05, 0.10).
Raise it to 0.10, the 99th percentile of that null. Firings 22 -> 10, crystals
12 -> 5, decoy false-positive rate 3.4% -> 0.8%. An independent check against
XDS - which integrates through ice rings and so measures exactly what we delete
- agrees: of the firings with a usable comparison, 9 true / 9 false becomes
7 true / 2 false.
Battery: space groups 34 OK / 3 DIFF, the same three crystals as baseline, and
no other discrete decision changes on 37/37. The heavily iced crystal keeps all
five of its rings and its CC1/2 of 96.6; eight others recover 3.9-11.9% more
unique reflections and up to 10.4 completeness points. Cost is CC1/2 -0.84 on
one crystal, -0.35 on another, and agreement with XDS on the common reflections
worse by a median 0.0004.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--detect-ice-rings did two unrelated jobs at once: flagging ice spots so
indexing de-prioritises them and keeping ice reflections out of the scale fit,
AND gating the merge-time mask that drops a decorrelated ice ring and re-merges.
Turning it off to de-confound a merge-stage experiment therefore also changed
how the data were indexed - measured, that breaks indexing outright on two of
the 37 rotation battery crystals - while leaving it on lets the mask land
differently between two arms of an experiment and contaminate the comparison
(measured on up to 19 of 37 crystals in response to a small intensity change).
Add --ice-ring-mask[=on|off], default on, gating only the merge-time mask.
Verified with =off: ice-spot flagging and the scaling exclusion still log and
still apply, no mask line, no second merge, and the first error model is
bit-identical to the =on arm. The full pipeline and the offline --scale path
reach the same verdict on the same data, as they must.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A
symmetric trim is not a consistent estimator of the mean of a right-skewed
(Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true
mean at every level, and with ~50 signal pixels in the r1 disk that
under-subtraction adds ~5 counts to every partial on every frame. Measured two
independent ways on four rotation datasets - stored background_mean against a
plain ring mean over the same pixels on reflection-free frames, and directly on
apertures that provably hold no reflection. Empty-aperture pedestal, counts:
plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip
+0.02..+0.54.
Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for
monochromatic data. It rejects the same one-sided contamination the trim was
there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim
by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds
~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean
ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The
trim stays reachable with --background-trim for back compatibility; setting
either estimator clears the other, so they can never stack. --integrator boxsum
does not take the clip (matching what the shipped clip already did), so it now
uses the plain ring mean unless --background-trim is given.
The intensities get measurably more accurate: per-shell agreement with an
independent processing of the same images improves on 14 of 16 crystals
(weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR
- absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and
CC1/2 in the outer shell improves on 14/7.
EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by
information-free counts, so both get worse when the bias is removed; neither is
evidence against this change. That fingerprint is exactly how the trimmed mean
was accepted in the first place.
Known cost: over the 37-crystal rotation battery the de-novo space-group count
goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal
whose merge fails the absolute-sanity gate under either background (R_meas
63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the
very operator being scored, so its operator CC is diluted by construction. No
other crystal changes space group, and twin protection is not weakened - the
H-ratio veto that refuses genuinely twinned crystals gets MORE decisive
(1.63 -> 1.84, 2.83 -> 3.99).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running bare `./jfjoch_test` costs far more time than it is worth on a
developer machine, and CI runs the full suite on every push regardless; the
useful local run is the cases or tags covering the code that changed. Same
reasoning for the 37-crystal rotation battery - it belongs on changes that
plausibly move merged results, not on every edit.
Also record that pushing is the maintainer's decision, not part of "make the
change".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The correlation stage kept only reflections with I/sigma >= present_i_over_sigma
(3.0). That statistic is taken on the P1-MERGED intensities, whose sigma is
floored at b|I| (Merge.h, SigmaWithSystematicFloor) so that ISa = 1/b is the
asymptotic I/sigma ceiling - no reflection in a merge can read above it.
Verified over the rotation battery: max I/sigma equals 1/b on every merge.
So a fixed cut is not a per-reflection test at all. Every reflection sitting at
the floor reads 1/b exactly, however strong, and on a merge whose ISa falls
below the cut NOTHING passes: every operator is left with no pairs, its CC is
NaN, and the point group collapses to 1. The predicate "search-merge ISa < 3"
identifies the affected crystals exactly.
It is latent today - no crystal in the battery starves on the shipped
integration background - but it fires on four as soon as an additive intensity
bias is removed, and it is not a data-quality verdict: the crystals it silences
have final merges at ISa 19-22 while their low-multiplicity search merge sits
at 3.5-4.0, just above the cut.
Cap the cut at the merge's own I/sigma quantile so the correlation stage always
keeps at least its strongest quarter. A no-op wherever the fixed cut already
keeps that many - the cut stays exactly 3.000 on healthy merges. Battery
unchanged at 34 space groups matching XDS / 3 differing, with merged
observations identical to 0.000% on all 37 crystals.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts the profile-radius part of 457b1bfd1; the comparison-script and
mosaicity-column changes from that commit are kept.
The cap was validated on the rotation battery, which cannot test it: the profile
radius feeds `ewald_dist_cutoff` in IndexAndRefine, and that is read only by the
STILLS predictors (BraggPrediction/BraggPredictionGPU). The rotation predictors
gate on the mosaicity window instead and never look at it. So "no space-group
changes, 36 of 37 crystals bit-identical" showed the quantity is inert for
rotation, not that capping it is safe - and the one regime where it does act was
never exercised.
Validating it needs the serial-stills battery, which is a much larger exercise.
Until then the arbitrary constant is not worth carrying in a code path nobody
measured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Goes in §10.3 next to the per-frame scale and mosaicity smoothing, since it is
the same mechanism applied for the same reason, and trims the changelog line to
one sentence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Geometry is re-refined independently on every frame, against that frame's spots
alone - as few as a dozen on a sparse crystal, where XDS fits its equivalent to
about sixty times more data. Measured over ten datasets the per-frame orientation
carries two components: a slow drift that is real, with rugnux and XDS agreeing to
R^2 0.83-0.88 on the two crystals that genuinely slip by 1.5 and 0.54 degrees, and
a fast jitter that is fit noise, scaling with spots-per-frame at exponent -0.79
where counting noise alone would give -0.5. The jitter is worth 1-8% on merged
intensities, 24% on the sparsest crystal.
It cannot be fixed by refining less. Turning per-image refinement off entirely
loses six space groups and a whole crystal, and even a 624-spot-per-frame crystal
collapses; dropping the beam-centre terms holds the space groups but is worse on
31 of 37 crystals. The freedom is earning its keep, so keep it and suppress only
the band that cannot be physical - a crystal does not re-orient and snap back from
one frame to the next.
So smooth the orientation in frame order after integration and recompute each
partial's delta_phi, and hence its partiality, from the smoothed lattice. Batching
at integration time was not an option: frames are processed independently and the
online path depends on that. This runs before the GPU upload, so the device path
picks it up with no separate kernel.
The window is chosen per dataset by leave-one-out cross-validation, because the
two components vary far too much for one number - drift spans 0.018 to 1.288
degrees and jitter 0.005 to 0.221, so any fixed window over-smooths one crystal
while under-smoothing another. Chosen windows range from +-1 to +-20 frames. It is
capped: cross-validation scores how well neighbours predict a frame's orientation,
which on a barely-drifting crystal keeps improving with width, but the per-frame
fit is also absorbing a real per-frame systematic and smoothing too wide destroys
it - uncapped, one crystal chose +-60 and lost 16% of its ISa.
Battery over 37 crystals: space groups unchanged at 34 matching XDS, R_meas better
on 31 and worse on 6, low-resolution R_meas 30/7, ISa 26/10, high-resolution CC1/2
23/12.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same defect as the mosaicity in 2c94f3013, in the same file's sibling fit. The
profile radius is an RMS of the excitation error over whatever spots were kept,
and weaker spots sit further off the Ewald sphere, so it grows with the depth of
the list: measured over a 150 -> unlimited spot budget it rises ~60%, and on a
clean dataset as much as on a hard one, so this is general rather than something
one awkward crystal provoked. That made it a function of --max-spots, which is
an indexing budget, rather than of the crystal.
Its one consumer treats it as a membership gate (ewald_dist_cutoff is twice it)
where reflections at the cutoff carry near-zero partiality and are removed
downstream anyway, so the integrated data barely notices: with the mosaicity
already pinned, the partial count moves 0.2% across a 34% change in the radius.
It is also reported per image as a diagnostic, though, and a number that slides
with an unrelated setting is misleading to anyone comparing two runs - and it
would stop being benign the moment anything used it as a width rather than a
gate.
Battery over 37 crystals: no space group changes, 36 of 37 bit-identical, no
failures, one crystal marginally better.
Also in the comparison script: report XDS's mosaicity next to rugnux's. XDS has
two and they are not interchangeable - CORRECT.LP's REFLECTING_RANGE_E.S.D. is
post-refined, while INTEGRATE.LP's per-batch SIGMAR is its integration-stage
estimate, and the two differ by up to 2.3x. The XDS cell now prints both as
postrefined|MLE so a per-image estimate is compared against the one measured the
same way. Fixes a scoping bug in the same addition where every crystal read the
last directory's INTEGRATE.LP.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-image mosaicity MLE ran over the whole indexed spot list, so it rode
on --max-spots, which is an indexing budget. A spot is detected when
I_full * R(tau) clears the finder threshold, so selecting by intensity censors
on R(tau): a deeper list holds proportionally more large-|tau| partially
recorded spots and the fit widens with it. Raising the budget 250 -> 1000
widened sigma_M 0.059 -> 0.075 deg on a rotation dataset whose measured rocking
width says 0.054.
That is not cosmetic. An over-wide mosaicity mis-states every partiality in
scaling: forcing the mosaicity across that range moved the merge error model
from b 0.039 / ISa 26 to b 0.167 / ISa 6, and the space-group search lost a
genuine 422 with it, merging the crystal in 222 instead.
Cap the fit at the strongest 250 spots. FilterSpotsByCount leaves the list
strongest-first, so this selects exactly the spots a smaller --max-spots would,
and the mosaicity becomes invariant: 0.0538 deg at 250, 500, 1000 and 2000
spots, with the correct space group at each. Trimming or down-weighting the
tau tail does not work - the censoring is multiplicative in R(tau), so it
widens the whole distribution rather than adding a tail.
Battery over 37 crystals: exactly one change, the demoted crystal repaired
(33 space groups matching XDS -> 34). 23 of 37 are bit-identical, never
reaching 250 spots. Unaffected elsewhere: the default spot count is 250, and
stills have no goniometer so they return before the fit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Offline reprocessing is not bound by the online spot budget, and the cap is
applied at the end of SpotAnalyze, so it is exactly the spot list the indexer
and the per-image refinement see. jfjoch_viewer already sends 1000, so the two
front ends now agree on the same file.
Measured as a paired A/B over the 37-crystal rotation battery, de novo, with the
resolution and Friedel setting matched to the XDS reference, both arms from the
same binary bar this constant:
R_meas low shell 16 better 0 worse 19 unchanged
R_meas 14 better 4 worse 17 unchanged
ISa 14 better 6 worse 15 unchanged
CC1/2 6 better 3 worse 26 unchanged
Low-resolution R_meas is a clean sweep. Around half the battery is bit-identical:
those frames never reach 250 spots, so the cap never bound. Wall clock is
unchanged (10m00s vs 10m44s, uncontrolled for page cache).
Known cost, and the reason this is its own commit: one crystal in the battery
reproducibly loses symmetry, tetragonal 422 -> orthorhombic 222, doubling its
asymmetric unit. Its R_meas and ISa "improve" there, but that is what merging in
too low a symmetry always does, and the lower symmetry then admits a merohedral
indexing ambiguity. An intermediate cap of 500 demotes it too, so it buys none of
the safety. This is the known point-group-decision-moves-with-data-amount
fragility of the space-group search rather than an argument for starving the
indexer of spots - the search is the thing to fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two front ends disagreed on the fixed-threshold spot finder: rugnux started
from 3.0, jfjoch_viewer from the SpotFindingSettings default of 4.0, so the same
file processed either way could give different spots.
Inert on the default path - the adaptive finder derives its threshold from each
image's own per-resolution-ring noise and never reads signal_to_noise_threshold
(only ImageSpotFinderCPU/GPU and DetModuleSpotFinder do). It changes behaviour
only under --no-adaptive-spots, and there it now matches the viewer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bitshuffle_hperf is an x86-only implementation - its entire vector body sits behind
__i386__/__x86_64__, so on aarch64 every entry point compiles down to the scalar
fallback. Measured against its own SIMD path that costs 8.3x on encode and 3.8x on
decode, and it is the transform behind every compressed image the writer produces and
every one the reader, preview and XDS plugin take apart again.
The classic bitshuffle vendored beside it does have an aarch64 NEON path, and is already
compiled into the same target, so this costs nothing new. BitShuffleBlock.h picks
bshuf_trans_bit_elem/bshuf_untrans_bit_elem there and keeps bitshuf_encode_block /
bitshuf_decode_block everywhere else, where hperf is about twice classic SSE2 and remains
the better choice. The expected aarch64 gain is ~2.5x encode / ~1.7x decode: classic NEON
is 128-bit and carries an extra pass, so it recovers part of the gap rather than all of
it. The condition mirrors USEARMNEON in bitshuffle_core.c exactly, because with NEON off
the classic scalar path is slower than hperf's and must not be selected.
Swapping implementations is only safe while the two agree bit for bit - otherwise an ARM
build would write files an x86 build could not read. They do: verified byte-identical
output and mutual cross-decoding for elem_size 1/2/4/8 over block sizes from 8 to 65536
elements. Both are always compiled in, so the new test holds them to it on every
architecture, not just the one that would notice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers the Debian package rename and the DKMS scripts, the writer's --verbose, the
version plumbing, and the documentation pass - the release-contents page, the corrected
repository URLs and package names, and the rugnux, viewer and writer pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rugnux gained --model - R-free and 2Fo-Fc/Fo-Fc maps against an atomic model, and with it
the resolution of the enantiomorph and of a merohedral indexing ambiguity - without the
page ever mentioning it. It was the only option missing; the two lists now agree in both
directions, checked against the usage the binary prints.
The viewer page still said results are never saved and that no Windows package exists.
Both have been false for a while: the Processing panel runs full rugnux jobs on the open
dataset, writes _process.h5 and the merged reflections, registers each run as a
selectable view so runs can be compared, and can hand out the equivalent command line for
a cluster; and the installer is published with every release. The mask menu also loads
TIFFs now, and the View menu has layout presets.
The writer page documented -R for the root directory, which is the back-compatibility
alias for -d, and an HTTP status interface that no longer exists - status reaches the
broker over the writer notification socket, and a writer is stopped with a signal.
The test page pointed at .gitlab-ci.yml and at jfjoch_offline_process, which is not a
binary any more; the CrystFEL fixture pointed at HDF5DatasetWriteTest, which is not
either. The broker page linked ../broker/redoc-static.html, which MyST resolved by
copying the 700 kB file into _downloads/ rather than using the copy already in _static.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing said what is in a release or what it needs of the machine it lands on: that the
Linux binaries are built -march=x86-64-v3 and the Windows ones /arch:AVX, so each has a
CPU floor; that the portable .tgz is built on RHEL 8 for its glibc; that the Windows
installer is MSVC (Visual Studio 2026), CUDA 13.3, Qt 6.11 and carries the Qt runtime;
and above all what the CUDA variants need. Only cuFFT is linked dynamically, and it has
no link-time dependency on the driver library, so a CUDA build starts on a machine with
no NVIDIA GPU at all and falls back to the CPU path - as long as cuFFT can be loaded,
which the .tgz and the installer arrange by shipping it and the distribution packages
arrange through the distribution's own CUDA packages. Collected into a new page rather
than scattered over the install instructions.
The repository page had the RHEL 9 rows pointing at el8 paths under the wrong slsdet
number, no rows at all for the two slsdet9 repositories the pipeline uploads, a driver
package named jfjoch-driver where it is jfjoch-driver-dkms, and a note that RPMs are
unsigned from before the pipeline started uploading them with sign=true.
The FPGA page had a paragraph that stopped mid-sentence, in the middle of a link, and a
section describing a firmware build triggered by commit message. The firmware is stable
and carried from version to version now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rugnux_scale and azint have not been separate binaries since they became rugnux --scale
and rugnux --azint-only; the viewer component carries rugnux, jfjoch_extract_hkl and
jfjoch_recompress.
Two job conditions tested github.ref_type against 'workflow_dispatch'. ref_type is only
ever 'branch' or 'tag', so that test never matched and never did anything: build-rpm's
whole condition was that test, and half of the unit-test one was. Both jobs ran on a
dispatch, as the release flow needs them to. Drop the dead tests rather than repair them
- the behaviour they read as intending is not the behaviour that is wanted - and write
down why unit-tests is skipped on a tag, which is the part that is deliberate: the
dispatch run is the one that tests, and the tag it creates only rebuilds and uploads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--no-scaling-corrections said it disabled the decay and absorption surfaces. It disables
the modulation surface too - GetCorrectionSurfaces() gates all three - and the flag has
worked that way since the detector-plane modulation was added; only its description did
not follow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--verbose was declared required_argument while -v carries no colon in the getopt string,
so the long form consumed whatever followed it. "jfjoch_writer --verbose tcp://host:5400"
swallowed the address as the flag's argument and then failed for want of a data source -
the short form was fine, which is presumably why it went unnoticed.
The usage line for the root directory asked for <int>. It is a path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The openapi-generator invocation still passed --git-host=git.psi.ch and a user id of
jungfraujoch, from before the move to gitea.psi.ch/mx/jungfraujoch. Those properties are
not cosmetic: they become the source URL in the generated README and pyproject, so the
published client documentation - docs/python_client/README.md, which is copied out of the
generated tree - told readers to pip install from a host that no longer answers.
Regenerating with the corrected flags changes those two lines and nothing else, verified
against the committed tree.
update_version.sh, make_doc.sh and gen_python_client.sh were all mode 644, so the
"run update_version.sh" the documentation asks for fails on the shebang. CMake and the CI
both work around it by invoking them through bash.
make_doc.sh builds a throw-away venv in the working tree and deletes it on the last line,
which set -e skips whenever pip or sphinx fails - so a failed docs build left tmp_venv/
behind. Delete it from a trap instead, and ignore it along with the default output
directory and the sdist directory gen_python_client.sh creates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PROJECT() carried a hardcoded 1.0.0 next to a JFJOCH_VERSION read from the VERSION file,
so PROJECT_VERSION was free to drift from the version everything else uses. It cannot
simply be handed the same string - project(VERSION) accepts numeric major.minor.patch
only, and rejects a pre-release suffix such as -rc.161 - so cut the numeric part out of
the same file instead of writing it down a second time.
common/ then read ../VERSION again into PACKAGE_VERSION, purely to interpolate it into
GitInfo.cpp. That is the same file read twice with two variable names, one of them a
common enough name to be set by something else in the parent scope. Use JFJOCH_VERSION,
which is already in scope there.
The CUDA architecture note claimed the list "embeds no PTX". A bare entry in
CMAKE_CUDA_ARCHITECTURES emits SASS and PTX both, so the newest entry has been the
forward-compatibility path all along: on a GPU newer than anything listed, the driver
JIT-compiles that PTX at first launch. Adding sm_121 still buys native code on Spark
instead of a JIT, which is what the comment should have said.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CPACK_DEBIAN_MAIN_COMPONENT does not exist. Only CPackRPM.cmake has a MAIN_COMPONENT
variable, which is why the RPM came out as "jfjoch" while the .deb of the same component
came out as "jfjoch-jfjoch" - the DEB generator names every component package
"<CPACK_PACKAGE_NAME>-<component>" unless CPACK_DEBIAN_<COMPONENT>_PACKAGE_NAME overrides
it, and the line we set was read by nobody. It also named a component ("broker") that
does not exist, so it could not have matched anything either way. Set the name the
generator actually reads, and declare Replaces/Conflicts on the old one: the new package
owns the same files, so dpkg would otherwise refuse to unpack it over an installation
that already has jfjoch-jfjoch.
The DKMS component's postinst and prerm are the driver's postinstall.sh and
preuninstall.sh, configure_file'd into place. They were mode 644, and
CPACK_DEBIAN_PACKAGE_CONTROL_STRICT_PERMISSION is off, so they were packaged as they are
- and dpkg cannot execute a maintainer script it cannot execute. The RPM path is
unaffected: it reads the same files as text into the spec.
The viewer's Freedesktop menu entry and its icon are installed on Linux only now. On
Windows the Start Menu shortcut comes from CPACK_PACKAGE_EXECUTABLES and on macOS from
the .app bundle, so on those two the installer was carrying a share/applications entry
and a share/pixmaps icon that nothing reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers the GPU decode work (fused un-transpose, the memory it frees, corrupt-chunk
detection, large bitshuffle blocks, host fallback), the two reproducibility fixes
(integer ring statistics, striped geometry-refinement sampling), the connection-teardown
and written-extent fixes, and the thread-count, max_hkl and compression-time repairs.
Folded into the existing entries where they belong rather than added as new ones.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cache returned a device copy for a (device, host address) pair and cast it to
whatever the caller asked for, with nothing checking that the bytes behind that address
were still the same bytes. A host buffer can be mutated in place - PixelMask::LoadMask
does exactly that - or freed and reallocated at the same address, and either hands the
caller a device copy of something else. Nothing would report it: the tables are read-only
geometry, so the engine would simply mask the wrong pixels for the rest of the run while
the azimuthal mapping, the written pixel_mask dataset and the viewer overlay used the new
one. Today that is unreachable, but only because of two guards in unrelated files that
neither state nor assert the requirement.
The byte length and an FNV-1a checksum of the bytes being uploaded are now part of the
key. Both are computed once per engine construction, over a buffer that is about to be
copied to the device anyway, so the cost does not show. Expired entries are pruned on
insert, since distinct content now means distinct entries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each image is written at its own ordinal, so a frame that fails to load or analyse leaves
a HOLE - the frames after it keep their positions rather than shifting up. The end
message nevertheless reported the number of frames that SUCCEEDED as the image count,
and the writer sizes /entry/data/data and every per-image array from that.
So one failed frame in the middle of a run made the declared extent one short, and the
image it dropped was the LAST one written, not the one that failed. Two failures dropped
two, and so on: the file quietly ends before the data does, with the per-image metadata
still carrying rows for images the VDS no longer maps. It also under-counted the data
files when the run was split.
Track the highest ordinal actually written and use that. A hole then reads as the fill
value, which is what a frame that was never written should look like, and the counts of
collected and written images stay counts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`frac > RATIO * best_frac` cannot be satisfied once best_frac passes 1/RATIO - above
0.667 for a ratio of 1.5, which is ordinary for good rotation data. Above that the two
guards do not raise the bar, they close the branch: no axis multiple and no
lower-symmetry setting can displace the incumbent however much better it fits, so a
genuine superstructure is kept as its sub-cell and its satellite rows go unindexed,
silently.
The obvious repair - restate the bar on the fraction left UNINDEXED, which is well
defined over the whole range - was implemented and measured. It regressed the
37-crystal battery from 34/37 to 32/37 correct space groups: a C2 lattice fell to P1,
and a P2 case went to C222 keeping 2923 of 22440 reflections with CC1/2 in the last
shell at -35%. The indexed fraction is too noisy to carry a looser test.
So the unreachable-but-safe form stays, and the limitation is recorded at the comparison
rather than left to be rediscovered. Fixing it properly needs the selection to be
decided on something better than the indexed fraction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-ring sums were floats reduced by atomics, so the ring sigma - and with it the
detection threshold - depended on the order the blocks happened to arrive in. Detection
compares an INTEGER pixel value against that threshold, so a threshold that drifts
across an integer flips every pixel of that value in the ring at once, which is how a
last-bit difference turned into a different spot list.
A preprocessed pixel is an exact int32 and the masked and saturated sentinels are
skipped, so v and v*v are exact in 64 bits, and integer addition is associative: the
sums no longer care about arrival order. Both engines now accumulate the same way, so
they agree exactly rather than approximately, and the GPU spot list is bit-identical
across runs. The corrected sums that feed the reported azimuthal profile stay float -
a pixel value times a float correction has no exact integer form - but they do not
enter the detection decision.
Cost: the ring reduction needs 28 bytes per bin instead of 20 in the plain pass, which
drops it from eight co-resident blocks per SM to seven and costs about 11% of that
kernel (0.582 -> 0.650 ms/frame on a 4.5 Mpx frame). End to end it does not show:
alternating runs on three rotation crystals came out the same or slightly faster, and
the battery is unchanged in every number. The CPU engine got 30% faster (32.2 -> 22.6
ms/frame), integers being cheaper than doubles.
Tests: exact CPU/GPU agreement on the spot list, and 50 repeats of bit-identical output
where there were four.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stills first pass drew frames from a shared cursor and stopped when a shared counter
reached its target, which got two things wrong at once. The cursor walked the equally
spaced sample in ascending order, so stopping early read only its leading PREFIX - the
beam centre, distance and cell were fitted to the beginning of the run, not across it,
and the comment claiming otherwise was wrong. And where the stop landed depended on how
the workers happened to interleave, so the set of frames varied run to run: on the same
data at -N 32 and -N 8 the pass examined 483 and 457 frames and refined the detector
distance to 168.0481 and 168.0530 mm.
The sample is now cut into a fixed number of interleaved stripes, each stopping once it
has contributed its share. Every stripe spans the whole run, so an early stop no longer
biases the fit, and a stripe is processed identically whichever worker claims it - so
what gets examined depends only on the data, not on timing and not on -N. The same three
runs now give 451 frames examined and 168.0452 mm, identically.
The bundle selection was order-dependent too: frames are collected in worker-completion
order and sorted by spot count with a non-stable sort, so equally strong frames swapped
places between runs. They carry their image ordinal now and it breaks the tie.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two futures per connection are joined from more than one path: the acceptor reaping a
dead connection, and the control plane starting or ending a run. Calling get() on one
future from two threads at once is undefined and invalidates it, and the guard against
it was a non-atomic test-then-set of an atomic flag - `if (!c.active) return; c.active
= false;` - so both callers could pass it.
Holding connections_mutex across the joins is what the previous commit removed on
purpose, and rightly: the joins block on a writer that may be inside a send. So the
lock is per connection and covers only the teardown. Nothing it joins takes it, so it
cannot deadlock.
The comment claiming a detached connection is unreachable by anyone else was wrong: a
control-plane call that copied the pool before the erase still holds a shared_ptr to it.
That is precisely how the two teardowns meet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The raw-bytes path assembled each element a byte at a time, which on a full frame cost
about 4x against writing the 8 contiguous elements a thread owns through an
element-typed pointer. They are 8*ES-byte aligned, so the compiler merges them.
72.4 MB frame: 1.524 -> 0.406 ms for upload plus both kernels.
The test now also times the LZ4 pass on its own, so the bounds and validity checks in
the hot loop can be costed rather than guessed at. They are free: 0.231 ms against
0.2297 ms measured for the kernel before any of them existed - the restored offset == 1
and power-of-two fast paths pay for them. compute-sanitizer memcheck reports no error
over 400 single-bit-corrupted payloads and nine malformed containers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-image image-scale B factor was dropped from the CBOR stream and from the written
HDF5, which is a change for anything reading those files, but the changelog listed it only
under the OpenAPI breaking changes.
The GPU adaptive finder test claimed both finders sum the rings in double. The CPU one
does; the GPU one stages a block's contribution in float before reducing across blocks in
double, deliberately, to keep the hot loop's shared footprint down. Say so, and say what
follows from it - detection compares integer pixel values, so a threshold that crosses an
integer flips every pixel of that value in the ring at once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two leftovers from earlier fixes of the same shape. BraggIntegrationEngineGPU still read
device 0's shared-memory size to decide whether its profile grid fits; workers are pinned
round-robin across GPUs, so on a heterogeneous node that check can pass on a different
card than the one the kernel launches on. SpotExtractorGPU still uploaded its default
resolution mask with a pageable copy on the NULL stream, which is not ordered against the
engine stream now that streams are created non-blocking.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
How far the predictor walks the lattice became a setting defaulting to "derive it from
the refined cell", which is what is wanted offline. Online it is not: per-image
prediction cost then depends on whichever crystal is mounted, and the cube (2n+1)^3
grows quickly. The broker was only ever handed a concrete value when a client sent a
bragg_integration block, and none of the shipped configs has one, so the derived path
was the normal deployment.
Four places in the tree - the conversion, the predictor, the OpenAPI description and the
changelog - already state that the broker keeps a fixed bootstrap. Make that true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RefineThreads rejects anything above 64, and the two-pass rotation path - the default -
passed nthreads/2 straight from the machine. On a host with 130 or more hardware
threads rugnux therefore exited 1 with the bare message "Candidate-cell refinement
thread count" before reading a single image, and the only workaround was to discover -N
and pass a smaller value. Dual 64-core parts are squarely in range.
The limit is now a named constant the setter and the callers share, so the two cannot
drift apart again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The device decoder was byte-exact on every valid input - 994 production-compressed
images, 927 hand-built LZ4 blocks covering engineered (offset, matchlen) pairs across
the overlap branch boundary, 18000 repeat decodes, sanitizer-clean - and an audit
against LZ4_decompress_generic could not construct a valid block it mis-decodes. What
it did not do was notice when the input was NOT valid, and that mattered more than it
looks: the decode buffers are reused frame to frame, so a block that stopped early left
the PREVIOUS image in place, and in the bitshuffled layout the untouched tail is the
most significant byte-plane. A corrupt chunk therefore did not look like a missing
corner. It looked like thousands of real pixels several powers of two too bright, fed
to spot finding with no diagnostic, where the host decoder had raised an error.
So the kernel now flags a block that fails to reach its declared length while consuming
exactly its payload, and the host turns that into an exception once the caller has
synchronised. Reads are clamped against the end of the payload as well as the output,
both length chains are bounded exactly as read_variable_length bounds them, the two
offset bytes are bounded, and LZ4's parsing restrictions are enforced. On the host side
a block size that is not a multiple of 8 elements is rejected (it made the un-transpose
read uninitialised shared memory), the block count is bounded by what the chunk could
hold before it becomes an allocation (twelve header bytes could demand hundreds of MB
of pinned memory, permanently, per worker), trailing bytes are rejected, and the stream
is synchronised before any throw that happens after work is queued. An image of fewer
than 8 elements is all verbatim tail and now decodes rather than throwing. When the
device route fails for any reason the host decoder gets its turn, so it costs speed
rather than the acquisition.
The lanes cooperate on the copies and a later match can read bytes another lane wrote,
which since Volta needs an explicit __syncwarp(); it worked only because ptxas happened
to reconverge at the post-dominator. The prototype's offset == 1 and power-of-two fast
paths are also restored - the shipped kernel ran a runtime modulo, an emulated 32-bit
division per output byte, on the path its own comment calls the common case.
The un-transpose is now fused with preprocessing. One thread owns one group of 8
elements across every byte-plane, so once it has transposed its 8 bytes out of each
plane it holds 8 complete elements and emits 8 finished int32 pixels with the mask, the
error marker, the saturation cap and the statistics applied. The decompressed image is
never materialised: 0.623 -> 0.411 ms/frame at 18 Mpx, 0.523 -> 0.340 with 8 concurrent
workers. Staging nothing in shared memory also drops the 48 kB ceiling, which had made
any file whose bitshuffle blocks exceed it a hard failure; 64 kB blocks now decode.
gpu_compressed is sized from the chunk with grow-on-demand instead of from the
uncompressed size - it was reserving ~73 MB per worker to hold ~4 MB. Measured on a
1630x1553 uint32 rotation set at -N 32, peak GPU memory falls 3756 -> 3084 MiB; the
same model gives ~144 MB per worker on an 18 Mpx frame.
Decoding on the device also stopped reporting a decompression time, which blanked the
broker's compression plot trace and filled /entry/profiling/compressionTime with NaN.
The decoder brackets the decode with CUDA events and reports it again.
Tests: a differential fuzz suite against the CPU decoder - incompressible and highly
compressible data, engineered offsets, a size sweep hitting every rem%8 value twice,
all six element sizes, an 18 Mpx frame, decoder reuse, concurrency, hand-built LZ4
blocks across the overlap boundary, 26 foreign bitshuffle block sizes from 128 B to
64 kB, corrupt payloads and malformed containers, with a coverage report that proves
which LZ4 paths were reached rather than assuming it. Plus the fused path held byte for
byte against ImagePreprocessorCPU, statistics included, and against the host-upload
path on the same frame.
Battery: 37 crystals, every merged number identical to the host-decode run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Earlier releases put breaking changes in their own paragraph after the bullets
(rc.139, rc.29) rather than as one item among them. Follow that: the OpenAPI
changes now sit under their own heading below the list, with the client-side
action in the lead line, and are listed one per change instead of run together.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rc.161 changelog had grown entries several hundred words long and listed the
same area three or four times over. Collapse them by subject - spot finding,
resolution limits, space-group search, scaling, performance, correctness - and
hold each to one line, keeping the actionable detail in the breaking API entry.
Add the performance work that had not been written up: device-side image
decoding and the parallel first-pass candidate-cell refinement.
Section 0 of the CPU analysis document was the longest thing in it after two
core algorithm sections, and most of that was a profiling narrative and the
measurements that motivated the change rather than a description of what runs.
Cut it to the two kernels, the host-side block scan and the fallback rule. The
attribution stays; it is also in the reference list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three independent changes to the CPU-bound parts of an offline rotation run, none
of which alters a result.
Candidate-cell refinement now splits across threads. RefineCandidateCells already
took a (block, nblocks) partition, but the only call site passed nblocks=1, so the
whole first pass of a two-pass rotation run sat on one thread per scheme - two
threads, unchanged at every -N, for a third of the run. A block touches only its
own scores(j) and cells rows and holds its own scratch, so the split is exact.
The budget is a new IndexingSettings::RefineThreads, left at 1 by default and set
only where few indexer threads exist: raising it unconditionally would
oversubscribe the paths that already run one indexer per image across all workers.
The mmCIF writer built a std::ostringstream per formatted number, twelve per
reflection. snprintf gives the same digits for 0.535 -> 0.220 s per file.
The space-group search built the same orbit mapping twice per candidate point
group - once for the merge chi^2 and once for the systematic-error b, an
apply_to_hkl and Canonicalize per observation per operator each time. Build it
once and hand it to both.
18 Mpx rotation set 24.6 -> 18.7 s, 2.5 Mpx 13.0 -> 10.7 s, and the 37-crystal
battery 13m55s -> 10m47s with no failures, the same 34/37 space groups, and
statistics unchanged on 30 of 37 (the rest drift within the run-to-run spread the
binary already had, which a control build with the split disabled reproduces).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EnsureCapacity resized its 13 device arrays to exactly the current image's
predicted-reflection count, so every image that set a new record freed and
reallocated all of them. cudaMalloc and cudaFree take a device-wide lock in the
CUDA driver, so those images stalled every other worker: sampling the worker
threads during the per-image loop found 21-24 of 32 parked in cuMemAlloc_v2 or
cuMemFree_v2, all called from this one function, and the running maximum makes
32 workers do far more allocator work than one does.
Grow by half again instead. All transfers and kernel launches are sized by the
per-image reflection count rather than by the capacity, and the member is
already documented as holding at least that many, so over-allocating changes no
result. On an 18 Mpx rotation set the integration stage drops from 1.37 to
1.25 ms per image at 32 workers; merged statistics, error model and adopted
space group are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pipeline decompressed each image on the host and uploaded the result. On
an 18 Mpx rotation dataset that made the host-to-device copy the bottleneck of
the whole per-image loop: nsys puts the copies at 78% of the loop against 39%
for every kernel combined - 3600 transfers of 72.4 MB - and they ran at only
12.5 GB/s of an available 27-28 because the host-side decompression was itself
saturating host memory bandwidth. The GPU was mostly waiting.
So the compressed chunk goes across instead, about 4 MB rather than 72 MB, and
is decoded on the device. That removes the transfer and the host decompression
that was throttling it, in one change. Measured on an idle machine, a run goes
from 45.11 s to 24.97 s - 1.81x - with the merged output unchanged.
THE APPROACH IS JON WRIGHT'S (ESRF): "Experiences with GPU decompression for
bitshuffle + LZ4 data", HDF5 User Group 2021, and github.com/jonwright/
bslz4decoders. The kernels here are ours, but the idea and the demonstration
that it is worth doing are his. Cited in docs/ACKNOWLEDGEMENT.md and in the new
section 0 of docs/CPU_DATA_ANALYSIS.md.
Two kernels mirror the CPU decoder. LZ4 runs one WARP per bitshuffle block:
every lane parses the same sequence stream (a broadcast read, no divergence)
and the literal and match copies are split across the 32 lanes so the stores
coalesce; an overlapping match is treated as a pattern of period offset sourced
from bytes that already precede the write position, which keeps it parallel
rather than a serial byte loop. One thread per block instead measured 13x
slower. The bitshuffle inverse then un-transposes each byte-plane through
shared memory and interleaves the planes back into elements.
Only BSHUF_LZ4 is decoded on the device. The zstd variants have no device
decoder, and neither has an uncompressed or float image; Supports() returns
false for those and the caller decompresses on the host exactly as before. The
fallback is explicit, so a format we cannot decode on the device is a slower
path and never a wrong answer.
Tests hold the device decoder against the CPU one byte for byte, on data from
the production compressor, for every element size the detectors emit -
including the 8-bit DECTRIS modes, which take bitshuf_decode_block's separate
elem_size == 1 branch - plus a many-block frame, the formats it must decline,
and malformed containers, which must throw rather than run off a buffer.
Battery: 37 crystals, no failures, identical to the host-decode run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reduce_rings_shared is the largest kernel in the per-image loop - 73% of GPU
kernel time on an 18 Mpx rotation run, launched three times per image - and it
is bound by shared-memory atomic replay rather than by bandwidth: it reaches
156 GB/s against a measured 913 GB/s ceiling, and removing the atomics while
keeping the same loads makes it five times faster.
That is the case that wants resident warps to hide the serialisation, and four
blocks per SM left only 512 of the 1536 threads an SM can hold. The per-block
histogram is nbins * 20 B, about 9.6 kB at the default 0.01 1/A spacing, so
eight blocks fit in shared memory with room to spare. Both kernels are
grid-stride loops, so any grid is correct and a device that cannot co-schedule
eight simply queues the rest.
Measured: 9.21 s -> 5.33 s of kernel time over a run (852 -> 493 us per
launch), cutting total kernel time from 12.57 s to about 8.85 s.
flag_strong keeps four. It is bandwidth-shaped rather than atomic-bound and
eight measured no better (181 vs 175 us).
Wall clock is unchanged, and that is expected rather than disappointing:
kernels are 39% of the image loop while the host-to-device copy is 78%, so
faster kernels idle the GPU more without shortening the loop. This is
groundwork for the transfer work, not a speedup on its own.
The shared accumulators are float and summed with atomics, so the block count
changes the summation order and with it the last bits. The 37-crystal battery
is identical crystal for crystal except one observation in 925850 on a single
dataset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs in analyze_pixel, both confined to the middle stage of the wave.
The kernel walks each wave's rows in three stages. The priming and drain
loops read prev_out and substitute INT32_MAX for a pixel the previous pass
found strong, exactly as the CPU finder's value_at() does on every read. The
main loop did not - it read the image raw. So in the second pass the pixels
the first pass found strong stayed in the background statistics, inflating the
local mean and variance, and the halo of every broad spot failed the
signal-to-noise test. The two engines therefore did not agree, despite
1a1e05ad1 having set out to make them.
Separately, shared_sum2 is an int64 accumulator but val*val and old*old were
computed in int32 at three sites. That wraps above 46340 while the detector
overloads around 1e6, so any window containing a bright pixel got a corrupted
variance. pixel_result already did the same arithmetic in 64 bits.
The reason this survived is worth recording: numberOfWaves is fixed at 32, so
a wave owns ceil(height/32) rows, and the main loop only runs while
front < rmax with front starting NBX+1 = 16 rows ahead. At the existing test's
100 rows a wave owns 4 rows and THE MAIN LOOP NEVER EXECUTES - every row goes
through priming or drain, and the parity test passed on the broken kernel. On
a 4362-row detector frame a wave owns 137 rows and the main loop carries about
121 of them, so the bug covered roughly 88% of a real image.
The new tall-image test is sized to the partitioning rather than to
convenience: 1024 rows, spots placed inside the main-loop region, one core at
100000 to exercise the overflow. Against the unfixed kernel it reports GPU 25
pixels where the CPU finds 49 and fails six assertions.
The default rugnux path is unaffected because it uses the adaptive finder, and
the 37-crystal battery is identical crystal for crystal. The classic finder is
what SpotFindingSettings defaults to, so this is the online receiver's path;
measured there with --no-adaptive-spots, R-meas improves 11.2% -> 10.9% and
CC1/2 98.8% -> 98.9% on one crystal, with multiplicity up on both tried.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Candidate cell filtering called acos three times per candidate to turn dot
products into degrees, then compared those against the min/max angle bounds.
acos is strictly decreasing on [-1, 1], so "angle outside [min, max]" is
exactly "cosine outside [cos(max), cos(min)]" with the ends swapped - the
bounds convert once, and the three acos calls per candidate disappear.
The same loop also re-derived every already-accepted candidate's unit cell on
each new triple, inside the duplicate scan: three more acos each, for every
candidate accepted so far. Those cells are now kept alongside the candidates.
Measured on de-novo serial stills, where the indexer runs once per image:
34.43 s -> 14.17 s on one dataset and 21.92 s -> 6.59 s on another, with the
indexing rate and the merged reflection count unchanged (one gained 0.25
points of indexing rate). acos had been 40% of the whole process there.
Scope is narrower than that number suggests, and worth stating: the win is on
the de-novo path, which Auto selects for stills only when NO cell is known.
With a known cell Auto picks ffbidx, which reaches the same filter but feeds
it few candidates - measured neutral there (+0.5% instructions, -1.6% wall,
identical output), and that path already runs 14x faster in absolute terms.
Rotation runs the indexer twice per dataset rather than per image, so it is
unaffected: the full 37-crystal battery is identical, crystal for crystal.
Comparing cosines instead of angles can only move a candidate that sits on the
bound, so the filter's behaviour is unchanged except at that measure-zero
boundary.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CMake variable is the tidier spelling and would cover the MSVC job too, so
it is the obvious thing to reach for and worth saying why it was not.
It builds and links, CUDA included - and it does not reach .cu targets either
way, so there is no -dlto risk on either route. But it optimises less: 396.9 G
retired instructions against 384.9 G for -flto=auto, three runs each, with a
0.45% run-to-run spread, so a 3% gap is not measurement luck. Of 107 static
libraries the two routes agree within 5% on 105; the flag additionally covers
FFTW and libzmq. And CMAKE_AR stayed plain ar under the variable, so the
archive-handling argument for it did not hold here either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
update_version.sh had not been run for the adaptive spot-finding schema
change. Running it leaves the C++ server model and the TypeScript client
byte-identical to what the generators produced directly, but it also
regenerates two artefacts the direct calls do not touch and which are tracked:
the Python client's published documentation and the Redoc bundle. Both now
carry adaptive_threshold and false_pixels_per_frame.
LTO joins -march in the CI flags, which is why MARCH_CMAKE_FLAGS is now
LINUX_CMAKE_FLAGS - it no longer describes only the architecture. Measured on
rugnux against an otherwise identical build: 7-10% fewer retired instructions
and a 9% smaller binary, but only ~1.5% off the wall clock, because the
pipeline is GPU- and I/O-bound. It costs about 3x on an incremental rebuild
(9.8 s -> 30.1 s for one file plus link), so it stays out of CMakeLists and
out of a developer's edit cycle: CI builds from scratch and ships the result,
paying the link once. It links against CUDA with no special handling, and both
CI images already put gcc-toolset-13 on PATH, which -flto=auto requires.
MSVC is left alone: its LTO is a different flag (/GL + /LTCG) and nothing here
measured it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The self-calibrating detection threshold was reachable from rugnux and the
viewer but not online: spot_finding_settings carried no adaptive_threshold, so
the receiver always ran the fixed-threshold finder and the fused GPU engine
sat unused behind it.
adaptive_threshold and false_pixels_per_frame are now part of the schema, both
optional so an existing client that sends neither is unaffected, wired through
OpenAPIConvert in both directions and surfaced in the frontend panel, where
turning the mode on greys out the count threshold it replaces and reveals the
operating point it uses instead. The C++ server model and the TypeScript
client are regenerated from the spec; the Python client is generated but not
tracked.
Enabling it is REFUSED where spots are found on the FPGA - the JUNGFRAU and
EIGER workflows - rather than accepted and ignored, because a detection
setting that silently had no effect cannot be told apart from one that did.
The DECTRIS/SIMPLON workflow, which analyses images in software, accepts it.
Verified against a running broker: adaptive_threshold true is rejected with
that message and leaves the stored settings untouched, while false and omitted
both succeed.
It stays off by default online, unlike rugnux and the viewer. The broker
serves both workflows and the default has to be the one that works on either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The round-trip test checked h, k, l, I and the two predicted coordinates. The
other ten fields were written and read back unasserted, which is how the
offline --scale path came to lose image_number without a test noticing - it is
the field 3D-integrated reflections carry a fractional value in, and rocking
events cannot be grouped without it.
Reflections are now built by a helper that puts a distinct value in every
field that is meant to survive, keyed on the image and the reflection index so
a value read back from the wrong place cannot match, and checked by one that
asserts all of them. Verified by reintroducing the image_number loss, which
fails six assertions and passes none of them silently.
dist_ewald, observed and on_ice_ring are deliberately excluded and the test
says why: the first two are prediction/integration scratch that is never
written, and the third is recomputed from the resolution by whoever scales.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--scale did none of the ice handling the run that wrote the _process.h5 had
done, so re-scaling a stored dataset silently produced a different - and
flatteringly more complete - answer than the pipeline it was meant to
reproduce. Three separate gaps:
* --detect-ice-rings was accepted and ignored. The --scale block returns
before the line that applies it.
* Reflections were never flagged as sitting on an ice ring, so the per-image
scale fit included them. The flag is not stored per reflection, so it has
to be recomputed from the resolution.
* RotationScaleMerge was constructed with the ice half-width hardcoded to
zero. That is what turns a resolution into a ring index, so every ice test
inside the merge was a no-op whatever was passed to it.
The CC1/2 ring test that decides which rings to drop moves into
FindDecorrelatedIceRings, shared with the full pipeline so both reach the same
verdict on the same data, and --scale now re-merges with the mask the way the
pipeline does. The stills branch re-runs only the merge: the scaling has
already been applied to the reflections and repeating it would compound it.
Measured on a rotation dataset with three decorrelated rings, --scale went
from 8765 unique / 36.3% completeness / R-meas 18.5% / <I/sig> 1.1 to
7638 / 31.6% / 18.0% / 1.3, against the full pipeline's 7692 / 31.8% / 17.9% /
1.3 - the reported completeness had been inflated by reflections the pipeline
drops. The full pipeline is bit-identical across the refactor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ConvertToImageCoordinates now transforms the photon-weighted sums instead of
the centroid, which is only equivalent because a module's raw -> image map is
a signed axis swap plus an integer translation. Check that against the map
itself on every module of a detector whose modules do not share an
orientation, and either side of the 256-column multipixel gaps where the
translation changes. A wrong sign or a dropped gap term on any single module
would otherwise only show up as mispositioned spots on that module.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Candidate selection promoted a later cell whenever it indexed 0.05 more of
the accumulated spots. That margin is not meaningful when the candidate is a
near-integer volume multiple of the incumbent: multiplying an axis halves the
reciprocal spacing, so the multiple has a lattice point wherever its sub-cell
has one and another in between, and it collects spots the sub-cell leaves
unindexed for reasons that have nothing to do with the crystal. The indexed
fraction is biased in its favour, and a small lead is not evidence.
On one rotation dataset the true cell and a spurious 5x supercell were
separated by 0.003 of indexed fraction against a bar of 0.05 - close enough
that the -march flags the binary happened to be built with decided it. The
baseline build kept the true cell and merged to an R-free of 0.24 against an
external model; an -march=x86-64-v3 build (what CI uses) took the supercell,
carried it into a doubled cell and a different space group, and merged to an
R-free of 0.58, which is noise. Both were reproducible, five runs each, and
independent of thread count.
An integer multiple now has to index 1.5x the incumbent, the same shape the
lower-symmetry-setting guard next to it already uses. A real superstructure's
satellite rows are a large share of its spots and clear that comfortably.
Both builds now agree on the true cell with a wide margin.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The photon-weighted position sums were floats, so the centroid's last bit
depended on the build rather than on the data: gcc contracts the multiply-add
in AddPixel into an FMA under -march=x86-64-v3 and cannot at the baseline,
and MSVC does not contract at all under /fp:precise. The GPU extractor had to
match with __fmaf_rn, and the parity test still needed a two-ulp slack for
hosts that do not fuse.
Column, line and the per-pixel count are all integral, so the sums are exact
in int64 and both implementations reach the same bits with nothing to match.
The parity test now demands exact equality unconditionally and gets it,
including on a baseline build.
ConvertToImageCoordinates keeps the sums integral too: the raw -> image map
is a signed axis swap plus an integer translation, so it is applied to the
sums instead of to the centroid.
Drops the SpotToSave constructor, which had no callers and could not have
been converted without quantising the stored centroid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MSVC does not define M_PI without _USE_MATH_DEFINES, and rugnux is part of
the portable subset that JFJOCH_VIEWER_ONLY builds. JFJochMath.h already
carries PI for exactly this reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Lite workflow built its analysis with the fused GPU engine disabled, which is
also what decides whether the preprocessed image is copied device-to-host after
every frame. So on a machine with a GPU the online path was moving the whole image
back - 72 MB on a large detector, every frame, per worker - for a host reader that
does not exist on that path.
It was left off deliberately when the fused engine was added, to keep the online
path unchanged in that commit, and never revisited. Nothing depends on it: the FPGA
workflow uses a different analysis class, and strong-pixel values are read through a
device gather rather than from the host image.
Turning it on changes no result, and cannot: adaptive detection is unreachable
online, because the REST schema exposes no way to enable it, so the classic GPU
finder runs either way. Measured anyway, both engines on the same frames across five
datasets including very weak ones: 2400 frames, 638260 spots, not one difference -
identical lists, identical indexing rate, identical merge statistics to every
printed digit.
On a large detector with eight workers the median per-image cost falls from 94 to
59 ms and preprocessing from 21 to 6 ms; throughput rises from about 48 to 55 Hz. No
percentile regresses, which is what matters for a service - the ninetieth improves
from 128 to 74 ms and the tail with it. Spot finding gets faster too, because the
large copy no longer contends with the device gather.
Correct two statements while here. The flag's comment and the data-analysis
document both said the online receiver uses the CPU adaptive finder; online never
runs an adaptive finder at all, and the copy the flag really controls was not
mentioned. That copy would be better expressed as what it is - whether a host engine
will read the image, which the constructor already knows - rather than inferred from
which spot finder is wanted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-reflection frame number is written to the process file and always has been,
but the reader's designated initialiser simply omitted it, so every reflection came
back at frame zero. Nothing complained, because zero is a valid frame.
It matters because the 3D combine splits a reflection's partials into rocking events
by frame contiguity. With every observation claiming frame zero there are no gaps to
split on, so a reflection's entire rotation range collapses into ONE event: measured
on a rotation dataset, 216066 fulls against 216705 distinct reflections, where the
pipeline finds 367416. Forty-two per cent of the observations disappear, the
goniometer-frame absorption surface evaluates every observation at a single angle,
and the radiation-damage estimate is computed over a run that appears to last no
time at all.
The reason this survived is that the damage flatters: fewer, better-agreeing
observations per reflection give R_meas sixteen per cent lower, ISa thirty per cent
higher and a slightly better CC1/2 than the real merge. Anyone re-scaling a stored
file was reading numbers that looked better than the pipeline's while standing on
less than two thirds of the data, and one radiation-damage figure that was pure
artefact.
Read it as mandatory rather than optional-with-default, like h/k/l and the
intensities: a silent zero is precisely the failure being fixed, and every file this
function can read carries the dataset.
After the fix the combine reproduces the pipeline exactly. The normal path does not
go through this reader and is byte-identical before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spot finder flagged strong pixels on the device and then labelled them on the
host, so every frame sent the packed bitmask back - 2.26 MB on a large detector -
and the host walked all of it to recover a few hundred pixels. Do the labelling on
the device instead: compact the bitmask into a flat-index-sorted list, find each
pixel's backward neighbours by binary search, union them lock-free with path
halving, then label, accumulate and filter in one kernel. Only the spot list comes
back, and only one stream synchronisation per frame.
The gain in the ordinary case is modest - about a quarter off per-image spot
finding - because the host algorithm is genuinely fast on a normal frame. What
justifies it is the frame that is not ordinary. The host labels a sorted sparse
list through a window spanning two detector lines, so its cost is quadratic in how
many strong pixels share a line. A lit band of detector rows - a hot module, a
panel edge - costs 33 ms at two rows and 377 ms at fifteen, all of it under the
pixel cap that was supposed to bound this, and none of it maskable when the cause
is a diffraction ring rather than a defect: a ring runs tangent to a row at its
top and bottom, which is exactly the shape that hurts. The device version is flat
at 0.05 to 0.64 ms across every geometry tried, so an online run no longer stalls
a quarter of a second on an ice ring. Rejecting an over-cap frame is now free too,
since the count is known before any pixel is written.
Also label once and filter three times. The per-image minimum-pixel search runs the
extraction at three settings, but that setting only decides which components are
kept - it does not change the components - so the search itself need not be
repeated. This helps the host path as much as the device one.
The resolution mask moves to the device as a bit mask, uploaded when the limits
change rather than per frame, since the compaction needs it there.
Parity is asserted permanently rather than argued: five cases covering realistic
frames, occupancy from a hundred pixels to past the cap, the pathological
geometries including rings, the resolution mask, and a hundred-repeat determinism
check - requiring the same partition, the same spot order, and identical counts.
The centroid is a float sum and therefore order-dependent, so the device walks each
component from its root in ascending order and fuses its multiply-add the way the
host's does; note that whether the host fuses at all depends on the architecture
flags, so exact centroid equality is asserted where the compiler fuses and a
two-ulp bound otherwise. Making those accumulators integer would remove that
dependence entirely and is worth doing separately.
Regression set: all 37 crystals identical to the last printed digit. Unit suite
passes with the new cases.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
operator[], size(), data() and getBuffer() are one-line accessors that were defined
in the .cpp. The build sets no link-time optimisation, so out of line each of them is
a real call - once per pixel, from the CPU preprocessor, the CPU azimuthal integrator
and the CPU spot finder - and they stop those loops vectorising at all. They show up
in a profile directly: about six per cent of a whole azimuthal-integration-only run
is spent in the call overhead of two accessors that do nothing but index a vector.
Moving them into the header retires 30% fewer instructions on that run and takes the
per-image CPU cost on a GPU-less pass from 34.6 to 24.2 ms, with the output bit for
bit unchanged - same observation count, same cell, same merge statistics. It is worth
nothing on the GPU path, where the image stays on the device, and everything on the
paths that have no GPU to fall back on.
This also explains a measurement that had been blamed on the pixel mask being a
vector<bool>: a microbenchmark of that loop indexed a raw pointer and came out far
faster than the same loop in the binary, and the difference was this call, not the
mask. Measured properly the mask costs about 14% single-threaded rather than the 41%
claimed, and at the thread counts this actually runs at the bit mask is FASTER than
the byte mask it was proposed to become, because it moves eight times less traffic
and the loop is bandwidth bound. That change should not be made.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When more reflections are predicted for a frame than the output can hold, the
surplus was dropped by keeping those closest to the Ewald sphere. On the rotation
path that quantity is identically zero by construction - the rocking coordinate is
chosen so the scattering vector lands exactly on the sphere - so the comparison
fell through to h, k and l and the survivors were whichever came first in
lexicographic order. Measured on a large cell: every value within one float ulp of
zero, and the kept set had a MEAN PARTIALITY BELOW that of the full set, i.e. worse
than choosing at random. Rank by partiality instead, which the predictor already
computes and which is what the header always claimed was being kept. On the one
regression crystal large enough to cross the cap this lifts completeness from 84.8%
to 90.2% on the same observations; multiplicity and R_meas move the way they must
when the same measurements cover more of reciprocal space.
The online path asked for a cap of ten thousand but the truncation was hardcoded to
the offline limit, so the broker predicted and integrated up to six times what it
could transport and discarded the rest after paying for it. Honour the caller's
limit, which also makes the post-integration re-truncation dead code.
Indexer pool construction becomes a policy. The online service needs every indexer
resident before data arrives, because a cuFFT plan built on the first frame is
planning time inside the measurement; spending memory to be ready is the intended
trade there and stays the default. Offline there is no such deadline, and a stills
run with a known cell was holding a fully allocated FFT indexer per worker that the
algorithm resolution can never dispatch - 2.8 GB where 0.4 GB is needed. rugnux and
the viewer opt into building on first use; the broker, the receiver and the tests
are untouched. This also removes a dangling reference that was latent: the worker
held the settings by reference although the pool is routinely constructed from a
temporary, which only survived because eager construction finished inside the
constructor call.
Finally, refuse a first-pass lattice that indexes fewer than a sixth of the
validation frames. It fires on nothing in the regression set - the weakest real
crystal sits at 22 of 60, more than twice the floor - so it is a backstop, but the
failure it prevents is one the set does contain: a dataset with no crystal at all
adopts a lattice from its powder rings, integrates every image against it, and dies
much later inside the merge complaining about resolution. It now stops in the first
pass and says what to try.
Regression set: 36 of 37 crystals byte-identical, the exception being the
completeness gain above; 34 of 37 space groups, no failures. Full unit suite passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Linux jobs already pass -march=x86-64-v3; the MSVC viewer job passed nothing,
so it built at the x64 baseline. MSVC has no spelling for the x86-64-v2 level, but
/arch:AVX is the nearest and implies SSE4.1/4.2 - which is the part that matters,
because below SSE4.1 Eigen has no vectorised round and falls back to one libm call
per element. AVX is Sandy Bridge and up, a safe floor for a desktop viewer.
The architecture flags stay OUT of CMakeLists on purpose, so a site can build
x86-64-v4 on an AVX-512 cluster, or -march=native, or the plain baseline. That is
easy to mistake for an oversight and "fix", so say it in CLAUDE.md - together with
the consequence that catches anyone profiling: a default local Release build is not
what CI or production runs, and the gap is not uniform. GPU-bound work is
unaffected, but the CPU and Eigen bound phases - first-pass indexing and
scaling/merging - measure about 26% slower without the flags. That is enough to
make rounding look like a tenth of all cycles when a real build has it nearly free,
and to send a reader at the wrong code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three independent costs, each measured, none changing a result. Across the
37-crystal regression set the run time halves (median per crystal 2.0x, total
2.3x) and every crystal's merge statistics are unchanged.
The image copy back from the device moved the whole preprocessed frame - 72 MB
on a large detector, every frame, per worker - to serve a single host consumer
that reads only the strong pixels, at most a few hundred kilobytes of it. Give
the buffer a Gather() so that consumer asks for the values it actually wants (a
host loop on the CPU, a small kernel on the GPU), and copy the frame back only
when a CPU spot finder will genuinely read it. The copy the other way was worse:
it came from an unregistered vector, so the driver staged it through its own
pinned pool with a host-side memcpy on the calling thread, which does not overlap
and collapses under concurrency - 11.6 GB/s at one worker, 1.6 GB/s at eight.
That, not any hardware limit, is why throughput stopped improving past four to
eight workers. Pinning the decompression buffer once per worker fixes it: on a
18 Mpx dataset the image loop goes from 13.6 to 7.9 ms per image at 32 workers,
and 32 workers now beat 8 instead of losing to them.
Ceres was computing seventeen partial derivatives where five are free. The
per-image rotation refinement frees the beam and the orientation and holds
distance, detector angles, rotation axis and cell constant, but the cost
function declared all seven blocks, so every residual evaluated in Jet<17>
arithmetic. A residual exposing only the two free blocks - the same arithmetic,
the constants baked in - halves refinement, and it is exact rather than merely
close: dual coordinates evolve independently, so the residuals and the free
Jacobian columns are unchanged bit for bit.
The merge sorted an index array with a comparator that dereferenced a 1.6 GB
array of 72-byte records, i.e. a random walk over memory, single-threaded, twice
per two-pass run. Sorting a packed key instead is 2.4x. French-Wilson allocated
its integration scratch per reflection and ran serially; it now takes caller-owned
scratch and runs over chunks, 4.2x. The correction surfaces re-tested every
observation for usability and parity on each of ~22 passes and re-allocated their
accumulators each time; bucket the indices once and hoist the buffers.
Also convert std::round to std::rint where the rounded value only ever enters a
squared residual. The tie rules differ - away from zero against to even - so this
is safe exactly where a tie flips the sign but not the magnitude, and unsafe
wherever the value becomes a Miller index; those sites keep std::round. Verified
over all 2^32 float bit patterns: 8388608 exact ties exist, and the squared
residual is bitwise equal for every one of them. Worth little on its own here,
because the rounding that dominates is in candidate refinement, where the value
is an index and the substitution is not available.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each stage timer measures wall time inside one worker, so it counts whatever that
worker spent blocked on a contended resource - above all the single GPU - as well
as its own work. Those waits overlap across workers, so the mean was printed as if
it were the per-image cost when it is roughly the per-image cost times the worker
count. At the default thread count on a large detector the reported total came out
more than twenty times the truth, and single stages were printed as several times
the entire per-image budget of the run. That is the one output anyone tuning
performance reads, and it sent this investigation at the wrong stage for a while.
Divide by the worker count. It is a lower bound - a worker idle rather than blocked
is not counted - so rather than hide the remainder, report the image loop's own wall
time next to it, and with it the time spent OUTSIDE the loop. Nothing measured the
latter before, yet on a rotation run the first-pass indexing and the scaling and
merging can be more of the run than the per-image work is: on a large-detector run
here it is 5.1 s against 3.0 s. Both figures are for the last pass, and a two-pass
rotation run does all of it twice.
Also stop printing nan. The per-image indexing and scaling timers are never fed on
the two-pass rotation path, because the lattice is forced rather than searched per
image and the merge happens outside the loop, so every default rotation run reported
"indexing nan scaling nan". A stage that did not run is now simply absent.
Measured against the loop's own wall clock on a 18 Mpx dataset: 5% at one worker,
11% at eight, 29% at thirty-two, versus 23x too high before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The FFT shortlist could be rank-deficient, and then no cell could be formed at
all. FilterFFTResults takes the strongest max_vectors RAW directions and only
then prunes ones within 5 degrees of each other, but a single lattice row is
sampled by many neighbouring directions of the 16k half-sphere, so thirty raw
peaks routinely prune down to four or five distinct directions - the strongest,
hence shortest, rows. When a crystal's densest rows share a plane, every
surviving direction is coplanar, every triple the reduction forms is degenerate,
and the indexer returns nothing. On such a crystal the weak third axis was the
eighth distinct direction, at raw rank 78. Keep walking the same magnitude order
for up to four more directions that are 5 degrees clear of everything kept,
appended after the length sort so the earlier entries hold their positions and
the reduction still forms every triple it formed before - the shortlist only
gains candidates at its end.
That exposed two ways a change of SETTING was mistaken for a different lattice.
A centred conventional cell is an exact integer multiple of its primitive one,
so the same lattice described two ways differs by that factor: comparing
conventional volumes reads a setting change as a sub-cell or a supercell. Both
the candidate selection in the rotation indexer and the pass-2 comparison in the
driver did exactly that, and between them they discarded a correctly-classified
cubic F cell in favour of the body-centred tetragonal description of the very
same lattice. Compare primitive volumes in both, as the scheme comparison
already did.
Fixing the volumes alone was not enough, because the indexed fraction is also
biased across crystal systems: a subgroup setting holds fewer cell parameters
fixed than its supergroup, so it can never index fewer spots and will always
look better by that measure. Where a candidate has a lower lattice point-group
order at the same primitive volume - the signature of the same lattice in less
symmetry - require it to index markedly better, not merely better, before it
displaces the incumbent.
A general metric-symmetry promotion was implemented and rejected on evidence: it
raised a correct body-centred orthorhombic cell to triclinic and a monoclinic
one to C-centred orthorhombic, and no threshold separates the cases, because a
false pseudo-orthorhombic degeneracy measured tighter than a true cubic one on
obliquity and on alternative-basis axis excess alike. Metric alone cannot decide
this; only the intensities can, which is what the space-group search is for.
Measured over the 37-crystal regression set: one crystal goes from failing
outright to 91% indexed with 91% completeness and a better R_meas than the
reference, one keeps the cubic setting it had before, and every other crystal is
byte-identical. Full unit suite passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three small honesty and cost fixes on the two-pass rotation path.
Spot-finding settings did not reach the step that determines the unit cell. The
two-pass first pass reuses the spots stored in the file whenever it has them, and
reuse is the default, so both sampling schemes and the validation loop ran on
acquisition-time spots while only the per-image pass saw the command line. Every
--spot-* option was therefore a no-op for the lattice search on any file written by
this software, silently, and the lattice was cross-validated against one spot set and
applied to another. Giving spot settings now implies re-finding them for the first
pass as well, and plain reuse says so in the log.
The summary printed a space group and unit cell even when nothing indexed. With a
zero indexing rate the cell is whatever the lattice search happened to return, no
reflection was ever measured on it, and no output file is written - so stating it as
the run's answer claims a result the data do not support. Say that no lattice was
determined instead.
The second pass re-indexes de novo so the cell comes out self-consistent with the
post-refined geometry, and its result was already checked against the first pass -
once by the supercell test and once by the centring test - but only after every image
had been integrated with it, so a disagreement cost a whole extra pass on a dataset
that ended up on the first pass's lattice regardless. Compare them at the point the
lattice is adopted instead, using the same two tests and the same fallback. A
triclinic de-novo cell is left alone, being the demotion the merge reindexes.
Measured over the 37-crystal regression set: merge statistics are unchanged on every
crystal (the two that move are the known rotation-indexing non-determinism - one
observation in 2.9 million, and a zero-score lattice landing on no partials instead of
a few). Crystals whose data were already cached in the reference run are unchanged in
wall time. The one dataset that was burning a discarded pass went from three passes to
two, 509 s to 198 s, against 0.81x for the same-detector dataset that was already
running two passes - so about 214 s of the saving is the removed pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One analysis engine is built per worker thread, and each uploaded its own copy of
tables that are pure functions of the detector geometry: the pixel -> azimuthal bin
map and the per-pixel corrections (both in AzIntEngineGPU AND again in
AdaptiveSpotFinderGPU, from the same mapping), plus the pixel mask. On an 18 Mpx
detector that is ~224 MB per worker; with 32 workers ~7 GB of device memory held 32
identical copies.
Upload each table once per GPU instead and hand every engine on that device a shared
pointer to it. The cache is keyed by (device, source-vector address) because workers
are pinned round-robin across GPUs, so on a multi-GPU node each device keeps its own
copy - a kernel may only read memory resident on the device it runs on - and the
table is freed on the device that allocated it. Entries are held weakly, so a table
goes away with the last engine using it.
Measured on an 18 Mpx detector, 32 worker threads, 16 GB card: the stills path went
from exhausting the card (OOM in de-novo indexing) to 8.6 GB peak, and a normal
rotation run from 14.6 GB to 7.4 GB - it had been running within 1.6 GB of the limit,
so any larger detector or second GPU consumer would have tipped it over. Per-worker
footprint drops 403 -> 173 MB. Merge statistics are unchanged on a six-crystal
regression subset, including two-pass runs where the second pass rebuilds the mapping
on refined geometry, and wall time is unchanged (13.5-13.8 s vs 13.8-14.1 s).
Also take the launch configuration from the current device rather than device 0 in
AzIntEngineGPU and ImagePreprocessorGPU: with round-robin pinning, device 0's SM count
and shared-memory size can belong to a different card than the one the kernels use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ImageSpotFinderGPU::Detect launches its kernel twice, feeding the first
pass's strong-pixel bitmap back in so the second recomputes each local
background with those pixels excluded and keeps them strong. The CPU
finder ran a single pass, so the two returned different spot lists for the
same frame and a dataset processed without a GPU did not match one
processed with it.
It matters for any spot wide enough to reach into its own 31x31 background
box: the spot inflates the mean and variance it is then tested against, so
its outer pixels fail the SNR test. On the test image added here - a 5x5
core at 300 counts with a one-pixel ring at 25 - a single pass returns the
25-pixel core and 7500 counts where two passes return the full 49 pixels
and 8100.
pxl_val also becomes int64_t, matching the GPU's pixel_result signature.
It was int32_t, so pxl_val * pxl_val overflowed above 46341 counts even
though the surrounding sums were already 64-bit.
The new parity test compares PixelCount and Count, not just the centroid,
which does not move for a symmetric spot whether or not the ring was
picked up; it was confirmed to fail against the old single-pass CPU.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The header has said since it was written that blocking queue operations
must never run under connections_mutex; three code paths did exactly that.
KeepaliveThread held it while sending a keepalive to every connection,
which blocks until the peer-liveness or backpressure timeout - so one
half-dead writer socket could stall SendImage and every /statistics poll
for up to a minute, from an idle-time heartbeat. AcceptorThread and
StartDataCollection held it across RemoveDeadConnections, which joins a
writer thread that may itself be inside such a send.
RemoveDeadConnections is split in two: DetachDeadConnections unlinks them
from the pool under the mutex, which is quick, and CloseDeadConnections
tears them down afterwards with the mutex released - safe because they are
no longer reachable by anyone else. The keepalive loop copies the pool out
and sends outside the lock, the pattern EndDataCollection already used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deactivate() called measurement.get() inside the try that guards the
power-off, so an exception stored by a previously failed run was rethrown
before services.Off() ever ran: the detector stayed powered while the
state reported Error, and the operator had no way to turn it off. The
future is still reaped - it has to be - but its failure is logged and
dropped. It was already reported when it happened, and leaving the
detector on is the worse outcome.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The declaration still said "stills on, rotation off". Adaptive detection
has since been turned on for both workflows - adaptive_spots.value_or(true)
- which the comment at the assignment already explains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BraggPrediction.h claimed the buffer "GROWS to whatever a frame actually
predicts, so a large cell is never truncated here". Only the two GPU Calc
overrides call GrowCapacity; both CPU predictors stop at max_reflections.
The cap is applied inside the h/k/l walk and before the resolution test,
so what survives is the low-|h| block, not the reflections nearest the
Ewald sphere - a cell large enough to overflow 20000 gives different
merged reflections with and without a GPU. Documented rather than
silently claimed otherwise.
Also removed a paragraph describing a once-per-predictor overflow warning
that no longer exists, and fixed the rugnux_cli.cpp path in HDF5.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The azimuthal bin limit is FPGA_INTEGRATION_BIN_COUNT = 2048, not 1024.
There are 16 ROIs, not 64, and the map is a 16-bit per-pixel mask, so a
pixel belongs to any subset of them rather than to exactly one.
The lossy transform is round(sqrt(N*N*X)) = round(N*sqrt(X)): the HLS
squares the sqrtmult register before multiplying. The doc said sqrt(N*X),
which is off by sqrt(N), and the register comment claimed the value was
"minus one" and "should be square of the coeff" - both wrong, the host
writes N and the FPGA squares it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four of the seven ceres::Solve calls in image_analysis obtained a
Solver::Summary and never looked at it, so a solve that failed numerically
had its parameters written back and was reported as success.
StillsPartialityRefine and both PostRefine solves already gated on
IsSolutionUsable(); this brings the rest to the same contract.
IsSolutionUsable() is the right test rather than checking for CONVERGENCE:
it accepts a solve that ran out of iterations or wall-clock time but still
descended, which is exactly what the real-time callers depend on when they
set max_solver_time instead of max_num_iterations. Only FAILURE and
USER_FAILURE are rejected.
XtalOptimizer checks before the write-back, so a failed refinement now
leaves the caller's geom and latt untouched instead of half-updated.
GeometryRefiner folds it into result.ok, which previously reported success
from spot and frame counts alone. RingOptimizer returns a geometry by
value that both callers assign straight back over their input, so it hands
back the unchanged reference rather than a diverged beam centre.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified section by section. The corrections that matter:
- image_analysis/pixel_refinement/ is gone (38ea0ec23). The style rule it
anchored - no defensive or unrequested code - stays, now attached to the
experimental analysis code generally.
- rugnux_cli.cpp lives in rugnux/, not tools/ (f737424bd).
- There is no .clang-tidy in the tree and never has been; the naming
conventions are kept, described as what the code already does.
- compression/ has no sqrt codec - the algorithms are BSHUF_LZ4 and the
three BSHUF_ZSTD variants. The square-root transform is an FPGA pipeline
stage.
- jfjoch_hdf5_test is defined in tools/, not tests/.
- JFJOCH_VIEWER_ONLY was undocumented, and is forced ON on Windows/macOS.
- The per-image-scalar recipe pointed at reader/JFJochHttpReader.cpp, which
does not exist (it is viewer/), and missed that EndMessage carries both a
per-image vector and a run-mean scalar, so the CBOR END block needs two
keys. Added the camelCase-dataset vs snake_case-field trap.
- The portability notes described work already done (libjpeg-turbo) and
recommended fetching Eigen, which CMakeLists explicitly rules out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The queue-level fix for the live-follow OOM bounded how many datasets are
in flight, but not what each tick costs. Three handlers did full-dataset or
full-detector work per tick regardless of whether their window was open:
- the calibration window copied the whole pixel mask (GetMask returns a
reference; it was taken by value), memcpy'd it and ran a full-resolution
recolour on the GUI thread;
- the image-list window rebuilt one row of eight QStandardItems per image,
and then repainted every cell of the model on every frame to move a
one-row highlight;
- the dataset-info plot was rebuilt twice per tick, because setCurrentIndex
fires currentIndexChanged -> comboBoxSelected -> UpdatePlot and the
caller then called UpdatePlot again.
The first two now defer to showEvent while hidden, following the pattern
JFJochViewerReciprocalSpaceWindow::rebuildGL already uses; the highlight
repaints only the two rows that change; and the combo is blocked around
setCurrentIndex so the plot is built once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DrawSaturation walked the whole saturated set adding two QGraphicsLineItems
each, with none of the viewport culling DrawSpots and DrawPredictions do
directly above it, and no upper bound - and it is rebuilt on every pan,
zoom and frame change. Unlike spots, that set is not bounded by a setting:
an over-exposed frame or a missing beamstop saturates a large fraction of
the detector, which meant hundreds of thousands of scene items and a
multi-second freeze on each mouse drag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The GPU merge kernel rejects outliers on the device and keeps a per-full
flag there, but only returned the per-group counts. The host array the
CPU path fills stayed all zero, and the anomalous I(+)/I(-) accumulator
is host-side and unconditional - so with --reject-outliers and a GPU
present, the observations the merged IMEAN dropped were still averaged
into I(+) and I(-). The same command on a CPU-only host excluded them:
the exported anomalous differences depended on whether a GPU was there.
R_meas was unaffected, having its own device-side path that reads the
flags in place. MergeAccum now hands the per-full flags back so every
host-side reduction sees the same rejections. The comment claiming
reject_outliers was excluded from the GPU path was never true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was the only panel firing the generated call directly, with the
rejection routed to console.log. Because the poll then kept returning the
unchanged server value - which still equalled lastDownloadedS - the
resync effect never fired, so the dashboard showed a threshold the broker
was not using, indefinitely and silently. It now goes through useUpload
like its siblings, so a failure raises the snackbar, and onError puts the
server's value back in the panel.
The eight sliders also applied from onChange, which MUI fires for every
intermediate position while dragging: one drag across the ice-ring width
sent ~200 PUTs plus ~200 forced /statistics refetches, each reconfiguring
spot finding on the running acquisition. Dragging now only moves the
panel; the value is sent from onChangeCommitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
JFJochDecompressHperfPtr took source_size and never looked at it: every
per-block length was read out of the stream and passed straight to
LZ4_decompress_safe/ZSTD_decompress as the source length, with src_ptr
advanced by it. The only check happened after the whole buffer had
already been walked. A truncated frame, or a block header claiming
0x7fffffff, read far past the end of a heap buffer - reachable from the
ZeroMQ CBOR path and from any HDF5 chunk the XDS plugin is handed.
block_size == 0 satisfied the "% BSHUF_BLOCKED_MULT" test and then
divided at nelements / block_size, and source_size < 12 underflowed
source_size - 12 to about 2^64. Both are now rejected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deactivate() holds m for the whole power-off sequence, which is right -
nothing else should touch the detector while it is being turned off - but
it had no state check, unlike every other entry point. Called during a
measurement, calibration or initialisation it waited on measurement.get()
while holding m, and those threads re-acquire m to finish: a deadlock
that wedged every endpoint, /cancel included.
IsRunning() is exactly the set of states with a live background thread,
so deactivating stays possible from Error - otherwise a failed initialise
would leave no way to power the detector down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The processing VDS mapping is built from the number of images a run set
out to process, while total_images comes from the end message and is the
number it actually finished. Those differ whenever a run is cancelled or
skips an unreadable frame, and the mismatch was a hard throw - which
NXmx::Finalize catches by deleting the temporary master, so rugnux lost
the entire _process.h5 and every completed image with it. Ctrl-C after
5000 of 100000 images produced no output file at all.
Map what was written and drop the remainder instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rowsPerWave is rounded up, so with 32 waves the last waves can start at
or past the last row: rmin was never clamped and only the drain loop
checked front against height. On any detector below about 1500 rows -
including the module-converted 500K and 1M geometries and the kernel's
own unit tests - the priming and steady-state loops read whole rows past
the end of the image buffer, and those garbage rows entered the sliding
background window of the bottom rows.
Blocks with no rows to write now return before the first __syncthreads
(rmin depends only on blockIdx.y, so the block leaves together and the
collective ops stay well formed), and both remaining reads are bounded by
height. Rows past the end keep the INT32_MIN sentinel, which the window
already treats as "not counted".
The raw read in the steady-state loop is left as it is: making it apply
the prev_out substitution that the other two read sites use would change
which pixels are found, which is a separate question from this fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Character 40 carried a verbatim copy of character 35's matrix
(0-10 / -100 / 00-1), whose determinant is 1. A C-centred conventional
cell needs determinant 2, so a genuine oC lattice was returned as its
primitive monoclinic cell while still being labelled Orthorhombic 'C':
the refiner then clamped a ~117 degree beta to 90 and prediction dropped
half the reflections of a cell that has no centring.
International Tables A 3.1.3.1 gives 0-10 / 012 / -100 for character 40.
Character 35 is correct as it stands and is left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
International Tables A 3.1.3.1 gives 100 / -110 / -1-13 for character 9;
the last element was -3. With a negative determinant the transform is
left-handed and the "conventional" rhombohedral cell is not hexagonal -
beta came out around 110-134 degrees instead of 90 and c was far too
long. Any R lattice tall enough to reduce to character 9 was affected,
and the downstream Trigonal->Hexagonal promotion then forced 90/90/120
onto that wrong cell.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
release() published the handle through ReleaseSlot and only then wrote
status = InPreparation. ReleaseSlot makes the handle available to
GetImageSlot immediately, and GetImageSlot hands back this very object
without resetting it, so the receiver could observe the stale Sending
status and throw "Trying to send image that is not in preparation",
aborting the collection - or take the opposite interleaving and leak the
slot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SendZeroCopy closes the message when zmq_msg_send fails, and closing a
message built with zmq_msg_init_data runs its free function - here
zmq_socket_free, which already calls release(). The writer thread then
released the same slot a second time, under a comment claiming the
callback would not run.
The second release put a slot back on the free list while the receiver
had already taken it for the next image, so two threads wrote the same
buffer and the sending/preparation counters drifted permanently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-image ice ring score was encoded for every DataMessage but never
for the END message, although docs/CBOR.md has always listed it there and
both NXmx::EndResultVectors and HDF5MetadataSource expect it. Any dataset
written over the stream therefore had no /entry/MX/iceRingScore, and under
NXmxIntegrated - where the whole-run vector is the only copy - the score
was lost entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BraggIntegrationSettings::DMinLimit_A had a setter that nothing anywhere called, so
it was always its 1.0 A default - in rugnux, the viewer and the broker alike, with
no option or API field to change it. It feeds the predictor as high_res_A, which
discards any reflection with |q| > 1/d_min, so integration simply stopped at 1.0 A
however far the detector reached.
Five of the 33 rotation test datasets have detectors reaching past it, down to
0.981 A. On one of them, run with no resolution limit, the shell table ended dead
at 1.00 A with that shell still at CC1/2 55.6% and <I/sig> 3.4 - cut mid-shell
rather than fading out. This branch had already made the sibling limits
detector-driven (spot finding, scaling), so the pipeline was finding spots the
detector could see and then refusing to integrate them.
Make it a std::optional: unset means as far as the detector reaches, a value limits.
The limit is only a bound on how far the lattice walk goes, never a second opinion
on what is measurable - both predictors independently drop reflections that miss the
detector (BraggPrediction.cpp, BraggPredictionRot.cpp) - which is what makes the
detector's own reach the right default. rugnux gains --integration-high-resolution
(0 = no limit, as for --spot-high-resolution); the derived per-axis prediction range
resolves against the same number, so the two cannot drift.
Full battery: 30/33 space groups, unchanged from before, 0 failures and the same
three known mismatches; 22 of 32 crystals bit-identical and nothing worse than 5
observations in ~500k. The datasets that gain do so because their detector reached
past 1.0 A - the effect is understated here because the harness caps each merge at
the XDS resolution anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each Miller index is bounded by its OWN axis - |h| <= a/d_min, |k| <= b/d_min,
|l| <= c/d_min - so a single half-width has to be sized for the longest axis and
then walks the short ones far past anything the resolution cut can keep. Give the
predictor max_h, max_k and max_l instead, in all four implementations (CPU and GPU,
stills and rotation), and derive each from its own axis.
On a 149/83/226 A cell that is 23.1M candidates per frame instead of 94.2M, 4.1x
fewer. Results are bit-identical, as they must be - the candidates removed are only
ones the |q| <= 1/d_min cut rejected anyway: over six rotation crystals every merged
observation count, high-shell CC1/2 and space group matches the cube exactly, 6/6
space groups correct.
It buys almost no time, and the earlier claim that the cube cost 22% of that
crystal's wall clock was wrong. Removing 4.1x of the candidates moves it 1m58s ->
1m57s, so the whole prediction sweep is ~1% of the run. The 22% that crystal costs
relative to a fixed max_hkl of 100 is genuine extra work at max_l = 227: real
reflections inside the resolution sphere along the long axis, predicted and
integrated either way. Per-axis limits do not reduce that and cannot.
The user-facing setting stays a single number: it exists to bound the work, not to
describe the crystal, and applies to all three indices when set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The working tree accumulates merged reflection files, models and per-image dumps
while testing, and they sit in the repository root next to the source. They are
user data: a merged .mtz/.cif carries a sample's measured unit cell and its
filename usually carries the sample's name, neither of which may enter this
repository. Only build*/ and python-client/ were ignored, so a `git add -A` would
have picked all of it up - which is exactly what happened while preparing this
branch, caught before the commit was made.
Ignore the file types rather than rely on everyone typing the right paths, and
un-ignore tests/ so checked-in fixtures still work (git add -f for anything else
that genuinely belongs). Also widen the rugnux_vs_xds.py output dir to rugnux_cmp*/,
which is where the ad-hoc comparison runs land.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to making max_hkl a setting: it is now an optional, and unset means "take
it from this crystal". The predictor keeps only |q| <= 1/d_min and h = a.q for the
real-space axis a, so |h| <= a/d_min exactly - and likewise |k| <= b/d_min and
|l| <= c/d_min. max(a,b,c)/d_min therefore bounds all three at once: nothing that
could be predicted lies outside it, and nothing inside it is reached by a shorter
axis. It applies to rotation and stills alike, both going through the one place the
prediction settings are built.
Offline (rugnux, viewer) the default is unset, so every crystal gets its own range;
--max-hkl overrides it. Online the broker holds a concrete number, because the cost
is the cube of it per image and a live acquisition should not have its frame rate
decided by whichever sample is mounted: max_hkl joins bragg_integration_settings in
the OpenAPI with a default of 100, so an omitted field arrives as that default (the
generated model carries it) rather than as "derive it", and the frontend exposes it
next to the integration model.
Measured against a fixed 100 on six rotation crystals: three are bit-identical, two
were being truncated and recover 419k and 5.8k observations with the high-shell
CC1/2 going 15.1 -> 25.8% and 52.1 -> 55.3%, and the space group is unchanged 6/6.
It reproduces a fixed 200 exactly, which is the bound being tight rather than merely
safe.
The sixth is worth recording: a 149/83/226 A cell derives 227, and because a single
scalar has to cover the longest axis the cube is ~16x what a per-axis box would be -
22% wall clock, for a net 22 observations out of 364k (the per-frame 65536-reflection
cap re-selects at the margin when more candidates are offered) and identical CC1/2,
ISa and space group. Per-axis limits would remove that; the predictors already map a
thread index to h, k and l separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
max_hkl was hardcoded to 100 at the one place production builds the prediction
settings, so the only way to change it was to edit and rebuild - and it is not a
constant of the method, it is a property of the cell. An axis is truncated once
a/d_min exceeds it: 100 covers a 150 A axis at 1.5 A, but the same axis at 1.0 A,
or a 250 A axis anywhere, loses its outermost reflections with nothing said.
Move it into BraggIntegrationSettings next to the other prediction/integration
parameters and add rugnux --max-hkl (1..511, default 100 - no behaviour change).
Like the integration radii and the background trim it stays out of the OpenAPI, so
the broker keeps the default it has today and live analysis cannot be handed a
range that would not finish; the offline front end, which knows its cell, can ask
for more. RugnuxCommandLine emits it when it is not the default.
Measured on five rotation crystals at --max-hkl 200: two are bit-identical at no
cost, and three were being truncated - one gains 419k observations (+17%) and
takes its high-shell CC1/2 from 15.1% to 25.8% for +14% wall clock, the other two
gain 12k and 5.8k observations with CC1/2 76.6->82.4% and 52.1->55.3% for +9% and
+1%. ISa is unchanged throughout, and no frame overflowed the prediction buffer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dropping the fixed spot-finding limit left the reader still generating the
spot-vs-resolution plot over shells that stop at 1.5 A, so a stored file reopened
in the viewer showed a plot truncated at exactly the limit that was removed -
GenerateSpotPlot drops every spot outside its shells. Pass the detector's own
maximum resolution, as SpotAnalyze already does.
That value is 0 when the geometry gives no scattering angle at all (no distance or
no wavelength), and ResolutionShells throws on a non-positive d_min, once per
image. There is no resolution axis to plot against in that case, so skip the plot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Making the worker streams non-blocking removed the implicit ordering that the
constructors were still relying on. Each engine uploads its static inputs - the
pixel mask, the pixel-to-bin map, the corrections, the ROI map - with a blocking
NULL-stream cudaMemcpy, and then reads them from kernels on its own stream. A
pageable host-to-device cudaMemcpy returns once the source has been staged, with
the DMA still in flight, and a non-blocking stream no longer waits for the NULL
stream. The failure mode is a silently unapplied mask or a stale mapping, not a
crash, so it would not have announced itself.
Put them on the stream the engine already owns, and synchronise once at the end of
the constructor - that is required for the preprocessor, whose source is a local
vector, and leaves the others settled rather than in flight for the cost of one
one-time sync. The GPU spot-finder test uploaded its image the same way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leaving it at G = 1 looked like the conservative choice and is the more damaging
of the two errors. The per-image scale enters as rlp/(partiality*G) and multiplies
intensity and sigma alike, so substituting 1 for a scale that was really 1/200 of
the run median puts the intensities in 200x too low with sigmas 200x too low too -
1/G^2 times the weight they deserve. The merge cannot defend itself against that,
because the number that is wrong is the number the weight is built from. And if
the collapsed value was instead a failed fit, G = 1 merges the image mis-scaled by
an unknown factor. Per-crystal scales on serial stills genuinely span orders of
magnitude, unlike frames of one rotation sweep, so both readings are live.
An image whose scale is not believable has no usable scale. Write NaN into its
image_scale_corr, which every merge path already skips on, so it drops out of the
merged intensities, the error model and the statistics consistently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The "keep what the crystal came in with" gate required std::isfinite(cc) before it
would reject, so a refined model whose CC could not be measured at all was adopted.
ImageReferenceCC returns NaN when fewer than 20 reflections clear the partiality
cut - which is exactly what a refinement that collapsed the partialities produces,
since the cut is on the partialities it just rewrote. The gate therefore failed
open on precisely the crystals it exists to catch, and wrote the NaN into
image_scale_cc, on which --min-image-cc then drops the image from the merge, the
error model and the statistics.
Treat a CC that cannot be measured as worse than one that can, so the crystal is
put back exactly as it arrived.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Analyze dataset" cleared the stored cell and space group unless "Use the stored
unit cell / space group" was ticked, and that checkbox defaulted off. But the
settings panel writes the user's own cell and space group onto the experiment, so
a cell typed into the panel was discarded too - while the checkbox label said
"stored", implying it came from the file.
It also contradicted the dialog next to it: "Refine geometry (stills)" is offered
and default-ticked precisely because a cell is present, and the run then removed
that cell. The default dialog state on a stills dataset with a known cell ran the
bundle adjustment with nothing to anchor on and dropped indexing off ffbidx, which
needs a cell, onto de-novo FFT.
Drop the checkbox and take the crystal from the panel, which already has exactly
the right semantics: "Unit cell known" ticked writes the cell and group, unticked
clears both, and a space group of 0 means none. So ticked = -C/-S, unticked =
bare rugnux, and what a run will use is always what is on screen. That also keeps
the copied command line honest, since RugnuxCommandLine emits -C/-S from the same
experiment. The panel is refilled from the file when one is opened, so a finished
job's _process.h5 becoming the active snapshot now shows its group and can be
cleared, instead of silently pinning every later run to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--polarization was applied with the other geometry overrides, but
configure_offline_output runs afterwards and calls ApplyRugnuxExperimentDefaults,
which sets the polarization factor unconditionally. Every full-analysis run used
0.99 whatever was asked for, so the Lp correction was wrong at a beamline with
different polarization. Apply it after the defaults instead, and stop claiming in
RugnuxDefaults.h that nothing here is user-selectable.
--scale built a bare ScalingSettings and re-derived the rotation/stills split by
hand rather than calling RugnuxDefaultScalingSettings, which is what the split was
factored out for. It got scale-fulls, smooth-G, min-captured-fraction and outlier
rejection right and dropped CaptureUncertaintyCoeff on the floor: 1.0 in the
pipeline, 0.0 here. So re-scaling a rotation _process.h5 gave different sigmas and
ISa than the run that wrote it - the exact failure the block's own comment says it
exists to prevent. Start from the shared defaults and apply the overrides on top,
which also picks up --mosaicity and --search-min-zeta, and let -C bind here too.
REJECT_OUTLIERS_DEFAULT_NSIGMA had no reader left afterwards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The kernel guards against 2*max_hkl+1 and maps thread i to h = i - max_hkl, but
the host launched a grid sized 2*max_hkl. The h = k = l = +max_hkl planes were
therefore never launched while -max_hkl was, so the GPU predicted an asymmetric
subset of what the CPU loop (inclusive on both ends) does. The same bug was fixed
on the stills twin when the whole hkl range moved to the GPU; the rotation
predictor kept the old expression.
It only bites where the cell actually reaches |h| = 100 inside d_min - a ~150 A
axis at 1.5 A - so most data never noticed. Over the 33-crystal rotation battery
29 crystals are bit-identical and 4 gain observations, all of them large-cell or
high-resolution: +8519, +4693, +901 and +758 observations, with the high-shell
CC1/2 up 15.0->15.1%, 52.0->52.2%, 76.3->76.6% and 51.6->52.1%. Nothing is lost
anywhere, and R-meas and ISa move by at most 0.01.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first pass sampled min(n, max(refine_frames * 50, 8000)) images to keep the
200 strongest, so any serial run of 10000 frames or fewer indexed every frame
TWICE - and 99.3% of the pass was that sampling, the bundle adjust itself taking
0.24 s. The budget is sized for a low-hit-rate dataset; on data that indexes well
almost all of it was wasted.
Stop once four times the bundle size has been found, which still leaves the
"strongest N" selection a real pool and still spans the run, because the sample
is equally spaced. On a lysozyme jet dataset that is 1500 frames examined instead
of 4000, the same 200 bundled, and the same refined geometry - beam and distance
to the pixel, cell to 0.01 A. Warm cache: the pass drops 33 s -> 9.1 s and the
whole run 65 s -> 35 s, with CC1/2 and R-meas unchanged inside replicate noise.
Stills only - rotation has its own two-pass and returns from this function early.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every per-thread stream was created with cudaStreamDefault, and the 20 MB raw
image upload went to the legacy NULL stream. A NULL-stream operation implicitly
synchronises with every blocking stream in the process, so with one engine per
worker thread no two workers' GPU work could ever overlap - the whole GPU
pipeline ran serially however many threads were asked for.
Create the streams non-blocking and put the upload on the engine's own stream.
Measured on 2000 serial stills, interleaved, medians of three: 24.6 -> 19.2 s at
-N 32 (-22%), 32.7 -> 21.0 s at -N 16 (-36%), CPU utilisation 436-570% -> 723-859%.
Output bit-identical - same observations, uniques, completeness, R-meas, CC1/2,
error model and cell. The stream is synchronised at the end of the same function,
so the ordering the code relies on is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ScaleOnTheFly's collapsed-scale guard ran, and then StillsPartialityRefine
re-fitted every crystal's scale with no floor and adopted it unconditionally
whenever the image had no prior CC - which is exactly the state the guard leaves
behind. So the guard was protecting almost nothing. Measured on a lysozyme jet
dataset: of 367 images it left unscaled, only 8 were still unscaled in the
output, and 53 reached the merge at or below a fiftieth of the run median, the
worst at a 4525th; on a second run of the same sample, 297 images, worst at a
75000th. Those intensities are what the merge saw - up to 94x too high in the
written file.
Run the guard again on the refined scales. It now reports 367 then 51 on that
dataset, and the merge improves: R-meas 117.4 -> 110.8%, CC1/2 96.3 -> 96.5%.
Measurements confirm the rest of the guard is right as it stands: 0.02 is ~5x
below the lowest scale ever seen on an image that correlates with the merge
(no image at CC >= 0.4 falls below a tenth of the median), and leaving the image
at G = 1 beats both dropping it and replacing its scale with the median - on a
run where 30% of images are affected, dropping costs 1.8 CC1/2 and 29%
multiplicity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Raising the per-image reflection limit to 65536 for offline reprocessing also
raised the image-buffer headroom derived from it, and that headroom divides a
FIXED total buffer - so every slot grew from compressed+4 MB to compressed+16.7
MB and the receiver's slot count, i.e. how much of a burst it can absorb, fell by
about three. Online never needed the raised limit: measured on three serial
stills datasets the worst frame predicts 1380 reflections, 14% of even the old
cap.
So split them, the same way the geometry refinement's stopping rule is split:
online keeps the transport-sized 10000, offline gets the full 65536, and the
buffer headroom derives from the online one. Both still come from BraggPrediction
so the cap, the prediction and the headroom cannot drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RefineOne re-measured the image's correlation to the reference after writing the
refined partialities - because --min-image-cc drops images by it - and then
ignored what it measured. A crystal the tilt model suits worse than the fixed
partiality it replaces kept the refined model anyway, and the refinement is on by
default. Compare against the CC the crystal arrived with and put it back
untouched when the refinement does not improve it, which is the same state a
crystal with too few reflections to fit ends in.
Also four things noted in review and left until now: AdaptiveThresholdTest.cpp
was listed twice in the test target, AdaptiveThreshold.h was the one header in
image_analysis/spot_finding not in its library's source list, CLAUDE.md said
update_version.sh rewrites VERSION when it only reads it, and the CHANGELOG did
not mention that image_scale_b is gone from the plot_type enum - which breaks a
client that asks for that plot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-image refinement stopped on a wall-clock budget (40 ms, and 20 ms for
the rotation-only extra pass). Online that is exactly right - the budget is real
and an image that overruns it costs the acquisition. Offline it means the same
file refines to a different lattice depending on what else the machine was doing
at the time, which is not a property reprocessing should have.
Bound it by iteration count instead when the caller is offline. IndexAndRefine
takes the workflow as a constructor argument: the receiver asks for the
wall-clock bound, rugnux and the viewer get the reproducible one. 50 iterations
is Ceres' own default; the per-image problem converges well inside it, so it
bounds the pathological case rather than the normal one - measured on five
battery crystals, every number is unchanged from the timed version.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second pass re-indexes de novo and can land in a different setting from the
first - most often on the PRIMITIVE sub-cell of a centred lattice. The reindex
that exists to undo that declines when the metric does not match, and the code
then went on to stamp pass 1's group onto the cell regardless.
That is not a small error. A C-centred group on an already-primitive cell means
the centring absence rule removes half the reflections that genuinely exist, so
the merge holds more unique reflections than its own cell can - measured here as
"117% complete" with CC1/2 0.62, against the first pass's 92.6% and 0.98, on a
cell of exactly half the C-centred volume.
Detect the conflict where it happens and feed it to the pass-2 credibility guard
rather than acting on it locally: letting the pass re-search its own group
instead produced a P1 answer on a crystal XDS and the first pass both call C2,
which is a worse outcome than simply not trusting the pass. The completeness and
CC1/2 tests stay as the symptom-side net.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Digging into the selection logic showed the caps were not deciding the science -
the two-pass geometry post-refinement was, and the caps only fed it randomness.
Caps. The prediction buffer now grows to whatever a frame predicts instead of
keeping an arbitrary subset of it, and the per-image reflection limit is raised
to 65536, with the image-buffer transport headroom derived from the same
constant so the two cannot drift. Measured: bit-identical output on five battery
crystals, because a normal cell never approached the old limits - only a large
cell (~2.8e6 A^3, ~30000-44000 predictions per frame) ever did.
Pass-2 guard. The refined pass is normally the better answer, which is why it is
the canonical output, but it was adopted whatever it produced. On that same
crystal it merged more unique reflections than its own cell can hold -
completeness "117%", which is arithmetically impossible - while the header-
geometry pass sat at 92.6% and CC1/2 0.98. Compare the two and, when the refined
pass is not credible, go back to the header geometry and re-run so the canonical
files are the ones that are kept. Both bounds are set where only a failure
reaches them.
Together on that crystal: 111639 unique against XDS's 118730 (was 88000-99000
and different every run), CC1/2 98.0% (was 96.9-97.7%), ISa 8.54, and two runs
now agree bit for bit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found while chasing a 12% run-to-run spread in the merged reflection count of one
crystal. The GPU kernels claim output slots with an atomicAdd and, on overflow,
undid the increment with an atomicSub - so the counter saturated at the capacity
and the host could not tell a full buffer from an overflowing one. Which
reflections survived was then decided by CUDA block scheduling and changed every
run. Measured on that dataset: every frame predicts 23000-44000 against a 20000
buffer, and the spread reached the merged output (161591 / 165193 / 166110 /
166479 unique across four runs of the same command). Single-threaded runs diverge
too - this is entirely GPU-side.
Stop clamping the counter, so the true number predicted reaches the host, and
warn once per predictor when it exceeds the buffer. Which reflections are kept is
unchanged: making that reproducible means deciding what to keep when a frame
predicts more than the pipeline carries, and the obvious answers are worse - the
capacity is not the real limit, kPredictionOutput (10000, selected by smallest
excitation error) is, and on this crystal both a bigger buffer and a strided
selection collapse the merge, because the rotation combine rebuilds fulls from
exactly the partials that a smallest-excitation-error cut throws away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Analyze dataset" and `rugnux` with no options are two front ends onto the same
library and are meant to agree, but they decided their defaults separately and
the two lists had drifted. The viewer was missing:
- the de-novo starting point. The CLI discards the cell and space group stored
in the input file before it does anything; the viewer left them on the
experiment. Rugnux only searches for a space group when none is set, so the
search was skipped entirely and the stored group was reported straight back.
That is self-reinforcing: a finished job's own _process.h5 becomes the active
snapshot, so a run that ended in P1 pinned every later run to P1 - which is
what "lysozyme keeps coming out P1 in the viewer" was.
- the polarization factor, so the Lp correction was omitted altogether. The
missing factor is azimuthal and intensity-proportional, and symmetry mates sit
at the same 2-theta but different azimuth - the exact "unequal intensities
forced together" signature the space-group search vetoes as pseudo-symmetry,
which can land a genuinely de-novo run in P1 on its own.
- five rotation scaling defaults: the smooth-G range, the minimum captured
fraction, the capture-aware sigma, outlier rejection and --search-min-zeta.
The CLI's own comments tie the captured-fraction default to a crystal
recovering its true space group instead of P1.
Put the policy in one place (RugnuxDefaults) and have both front ends start from
it. The CLI now takes its defaults from there and applies user options on top;
its output is unchanged, verified bit-for-bit on four battery crystals.
The stored cell/group is still available: the job dialog offers "Use the stored
unit cell / space group", off by default, shown only when the file has one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The error model says sigma -> b*I for strong reflections, so merged I/sigma
flattens off at 1/b - the number reported as ISa. Plotting I/sigma against I
with that asymptote drawn on it is what shows whether the reported ISa
describes the data or comes from a degenerate fit, which nothing in the window
could show before. A third page next to the per-shell plot and table.
The merge carries a few thousand strided (I, sigma) pairs to the viewer for it -
a shape, not a reflection list; the reflections themselves are in the .mtz/.cif.
The hero row gains the Wilson B and the radiation-damage Delta-B. Both were
already computed and already in MergeStatistics, so they only needed showing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The protection against a per-frame scale collapsing toward zero lived inside
ComputeSmoothGWindow, so it only existed when smooth-G did: --smooth-g=0, a
dataset whose oscillation width is unknown, and any caller that never sets a
smoothing range - the viewer among them - merged with no guard at all. A
collapsed G multiplies that frame's intensities by 1/G and its sigmas by the
same factor, so nothing downstream can see it; the merge's n-sigma cut scales
with the number that is wrong.
Pull it out into ReplaceCollapsedScales, called unconditionally right after the
partial scaling loop, and let the smooth-G window assume what it now guarantees
instead of computing its own median and floor.
The fulls guard built its median from every frame including those never fitted -
those sit at the combine's corr = 1, so a run with many unfitted frames dragged
the median toward 1 and the floor with it. It also reported the absolute
amplification where the message says "below the run median".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Also corrects the reader bit-depth entry: taking the depth from the file was
the wrong fix for the 32-bit EIGER2 case and broke 8-bit files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight pages under docs/python_client/docs describe schemas that appear nowhere
in jfjoch_api.yaml and are linked from no index - left behind because the
regeneration step never cleared python-client/, which these are copied from.
That is fixed in update_version.sh; this removes what accumulated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
None of this has a reader:
- ScalingSettings::scaling_regularize and its setter/getter
- ScaleOnTheFlyResult::succesful (never set) and ::time_s (set, never read),
with the timing that only fed the latter
- JFJochImage::last_fit_viewport_ (written twice, read nowhere) and the
comment claiming the retry uses it - the retry keys off initial_fit_done_
- JFJochDiffractionImage::ice_ring_width_Q_recipA, and a QtConcurrent include
in a file that uses none
- an unused gemmi::Op accumulator in the spindle-angle helper
- <random> in Merge.{h,cpp}, from before the half-set split became a hash
- an orphaned comment describing the Ceres B-factor residual deleted in
014e43a4c, and two trailing comments that had collided on one line
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script now removes the generated C++ model, the frontend client and the
published python docs before regenerating, but not python-client/ itself - and
docs/python_client/docs is filled by copying that directory. So a schema dropped
from the API kept its generated model in the PyPI package and its .md page in
the published docs, linked from no index. Eight such pages are in the tree
today, JfjochSettingsSsl among them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adaptive spot detection became the default for rotation data as well in
6f4917dce; RUGNUX.md still said rotation kept the fixed-threshold finder, which
is also the opposite of what the usage message and CPU_DATA_ANALYSIS say.
--search-min-zeta had no entry in the option tables at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
centerAt returns early while the window is hidden, and nothing replays the last
position when it comes back, so re-opening the magnifier showed whatever region
the cursor was over when it was closed - with current pixels, which makes it
look like a live view of the wrong place. Remember the position while hidden and
apply it on show.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
drawPixelLabels took its range from the whole viewport rather than from the
exposed rect it was given, so a 200x40 px hover repaint still walked up to 5000
cells doing mapFromScene + QImage::pixel + drawText for each, only to have the
result clipped away. With hover feedback now rate-limited to 15 Hz that ran ~75
times a second at high zoom, against once per overlay rebuild before the
rendering rework. Intersect with the exposed rect; same in the magnifier.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The "d = ... A" readout moved from a scene item flagged
ItemIgnoresTransformations to a fixed viewport position painted in
drawForeground, which is what made a hover update dirty a small rect instead of
the whole viewport. But QGraphicsView pans by blitting the viewport: the painted
text is shifted along with the image and left there, and the pending update for
its old position is translated away too, so dragging the image smears ghost
copies of the readout across the corner. Dirty the old and the new rect when the
view scrolls - still a couple of hundred pixels, not the viewport.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With merging on, the _process.h5 is skipped because the merged reflections are
the wanted output and that file is large (113 MB for 200 images here). But if
nothing indexes there are no merged reflections either, so the run finished
successfully having written no file at all - the one case where the user most
needs something to look at.
Write it in that case. The per-image messages have already gone past unwritten,
so this carries the dataset metadata, the mask, the azimuthal profile and the
summary scalars rather than the full per-image tables - and it is small for the
same reason it is needed (124 kB on a zero-index run). A run that does index is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
main had no enclosing try/catch, but plenty of ordinary input reaches a setter
that throws: --polarization 2, --detector-distance 0, -q 0, --azim-max-q 20,
--scale combined with a reference MTZ, and every failure inside the pipeline
itself. All of them ended as "terminate called after throwing an instance of
'JFJochException'" and exit 134, with the message nowhere to be seen. Move the
body into RunRugnux and let main report what was thrown, exit 1.
Two options also still bypassed the numeric parser that exists to prevent this:
--scaling-high-resolution used atof, which turns a typo into 0 and then throws
from the setter, and --integration-radius used std::stof, which throws on
non-numeric input.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Ceres summary was discarded, so a solve that diverged or aborted left its
last iterate in psi and that tilt was written onto the partiality of every
reflection of the crystal. Restore the tilt the crystal came in with and stop
refining it; the scale fit alone is still a usable model, which is what the
other three early returns in this function fall back to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SolveScaleIRLS returns whatever it converged to and both writers accept any
G > 0, so a fit that collapsed to ~1e-3 multiplies that image's intensities by
a thousand. Nothing downstream notices, because the sigmas are multiplied by the
same factor and the merge's n-sigma outlier test is therefore blind to it - only
a total collapse self-heals, by overflowing corr to inf.
The rotation path refuses a per-frame scale this far below its neighbours; the
stills path had no guard. Judge each image against the median of the images that
did scale, and put a collapsed one back to G = 1 - the same state as an image
with too few reflections to fit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two faults in the same block, both of which let a search pass corrupt the
production merge that follows it.
The device's corr was only copied back to the host for the diagnostic dump, but
the |zeta| filter runs on the host and then uploaded the whole host array - so on
a CUDA build it wrote the values ingested BEFORE scaling over the scaled and
smoothed corr the device had just computed. With the rotation default
--search-min-zeta 0.85 that means the space-group search was deciding the
symmetry from an unscaled merge. Copy corr back first, and upload once after
both filters instead.
Zeroing corr also has no owner: it is how an observation leaves the merge, but
the only thing that ever rewrites it is the scaling loop, which skips frames it
cannot fit. A frame left with too few well-measured reflections therefore kept
its dropped observations at zero for the rest of the object's life - and the
final production merge re-uses the same object without re-ingesting. Snapshot
corr before the filters and restore it at the start of the next pass, so each
pass decides for itself and the final merge keeps everything, as documented.
The frame rejection (--min-image-cc) is now applied on the host for both paths;
its separate device path did nothing whenever the CPU combine was in use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The existing cases plant blobs at 200 on a background of 8..12, so any threshold
between 12 and 200 passes them - replacing RingThreshold with a constant leaves
them all green. Two cases that do not:
- the CPU threshold has to track the background: a frame and the same frame
scaled ten times must give the same spots, with a pixel a few sigma above the
background staying unfound in both. A constant threshold, or one that drops
the sigma term, fails one scale or the other.
- the GPU engine has to agree with itself across runs, which is what the ring
sums being order-independent buys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ring sigma is the cancelling difference sum2/n - m^2, and both sums were
float accumulated by atomics whose order is arbitrary. Two costs: the
cancellation left only ~4 digits in the variance, and the ordering moved the
resulting threshold by ~0.05 counts between runs - enough to flip a pixel
sitting on the hard "value >= threshold" test, and with it a connected
component's size. So the GPU engine did not reproduce the CPU one and did not
reproduce itself.
Only the accumulators that span blocks are widened. The per-block staging stays
float, because a block contributes a few dozen similar-magnitude pixels to a
ring and there is nothing to lose there - that also keeps the shared-memory
footprint of the hot loop, and hence its occupancy, exactly as it was: measured
on a 4.5 MP frame, 0.960 vs 0.966 ms/frame (40.9x over the CPU path, unchanged).
finalize_rings now does the cancellation in double and rounds to float last,
which is what AdaptiveSpotFinderCPU::AccumulateRings does.
The device properties are also read from the current device rather than device
0; callers round-robin engines across GPUs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b81c6f00b took the container depth from bit_depth_image in the file, which fixed
32-bit EIGER2 files but got the general case wrong: the reader converts every
image to SIGNED int32 (PixelSigned(true) a few lines up), while bit_depth_image
describes an unsigned container, and GetOverflow() combines the two. So a 16-bit
file still capped at INT16_MAX rather than 65535, and an 8-bit file newly capped
at 127 - flagging counts 127..254 as saturated, which drops the whole reflection
at the integration accept gate.
Declare 32 bits, matching what the reader actually hands out. The saturation cap
then comes from the file's own saturation_value, which is what it is for, and
the error value reported for a read dataset becomes INT32_MIN - the sentinel the
reader really uses. A file whose bit_depth_image is not 8/16/32 also stops
throwing on open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The host sized the grid from 2*max_hkl while the kernel guards against
2*max_hkl+1, so whenever the rounded-up grid landed exactly on 2*max_hkl threads
(max_hkl a multiple of 4, with the 8x8x8 block) the h = +max_hkl plane was never
launched. The CPU loop runs -max_hkl..+max_hkl inclusive, so the GPU predicted a
strict subset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
calc_std uses the cancellation-prone (sum2 - sum^2/n) form on float accumulators
summed over millions of pixels, so a flat ring - true variance near zero - comes
out negative as often as positive and GetStd() returns NaN. Both adaptive
spot-finder ring accumulators already floor this at zero; this one did not, and
1a0774eed made sum2 correct, so the path is now actually exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Making the high limit optional removed the implicit upper bound on the low one
(it used to follow from high <= maxQ and high > low), so --azim-min-q 50 with no
maximum is accepted and ResolveHighQ then calls std::clamp with its lower bound
above its upper bound, which is undefined.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other reader of spot_finding.high_resolution_limit spells "unset" as
value_or(0) and compares, so 0 and nullopt are interchangeable - except in
SpotAnalyze, which passed the 0 straight to ResolutionShells and threw
"Resolution must be above zero" on every image. Reachable over the REST API,
where 0 is the natural way to say "no limit" and the settings check lets it
through; the rugnux CLI already maps 0 to unset before this point.
While here, check that a limit that IS set is finite regardless of its sign -
NaN fails the > 0 test and was skipping validation entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two-pass rotation run reinstates pass-1's space group for pass 2's merge, so
pass 2 skips the search block entirely - and with it the flag that records that
the Laue class was CHOSEN by the search rather than given. The canonical output
therefore printed the plain "no twinning: the Laue class is holohedral, so no
merohedral twin law exists", which is exactly the circular conclusion the flag
was added to replace; only the throwaway _01 output carried the caveat.
The text is written per pass, inside RunPipeline, so the flag has to travel with
prepass_merge_sg_ rather than being patched onto the returned result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two-arm search is meant to use the Lorentz-filtered merge for the point
group only - systematic absences live in the weak reflections a |zeta| cut
removes, and reading them off the filtered arm is what cost four crystals their
screw axes. That is what the code comment and CPU_DATA_ANALYSIS both say, but
the filtered-arm-wins branch kept its whole result, screws and centering
included.
Let a search be pinned to a point group decided elsewhere (fixed_point_group)
and re-run Stage B on the all-observation merge when the filtered arm rescues
the point group. The point group is passed as its symmorphic representative, not
by name: gemmi calls both P321 and P312 "32". Reporting that representative also
lets the ambiguity check see two arms that disagree about which 2-folds are real
- by name they looked identical - and the advice it prints now names a space
group -S can actually be given.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The row-relative cut scales the "too strong to be absent" threshold by the axial
row's own median E^2, floored at the plain value - so it can only raise the bar,
and a row whose control class holds a single strong reflection sets it from that
one reflection. That direction invents screws: a genuine 4_2 whose 00l happen to
be observed only at l=4n reads its l=4n+2 reflections as absent and ranks
4_1/4_3 above the truth. Require three controls before the row may set the
scale; below that the row keeps the plain cut.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A candidate can have several confirmed subgroups of the same order - 422 has
both 4 and 222 - and on a twinned crystal the rival is not a harmless
alternative: a P4 crystal twinned by 2[100] has 222 confirmed too, and 222
CONTAINS the twin laws, so its own merge b is already ballooned. The H test
already answers to every tied parent; the systematic-b veto and rescue took
whichever one the enumeration happened to list first (222 before 4, by space-
group number), which disabled the veto on exactly the case it exists for. Take
the smallest parent b, which is the conservative direction for both tests.
The refusal message also quoted the raw parent b rather than the floored value
the veto actually compared against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cap that refuses to report an impossible ISa was zeroing the asymptotic b
itself, and that same value is the floor passed to SigmaWithSystematicFloor -
where zero means "no floor". So on the degenerate low-multiplicity fit the guard
is written for, instead of capping merged I/sigma at 100 it removed the cap
entirely. Report the asymptote as unmeasured, keep the fitted value for the
floor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6be94f2be stopped subtracting a negative intensity's Poisson term from the
background variance, but only in the host Combine(). The CUDA combine is the
path that actually runs: Run() selects it whenever a device is present and no
observation dump was asked for, so the correction never took effect on a normal
run, and a --dump-observations run merged differently from a normal one - the
two are meant to be identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The document had accumulated development narrative: what was tried and
rejected, which datasets a change rescued or cost, measured percentages
from test batteries. That belongs in commit messages and reports, not in
a reference description of the pipeline - it dates quickly, and a reader
looking up what an algorithm does has to sort it out from how it came to
be.
Removed throughout, keeping the algorithmic content and the design
reasoning that explains a choice on principle:
* 3.2 the whole paragraph justifying the rotation spot-finding limit
from battery measurements, and the CPU-vs-GPU per-frame timings;
* 3.3 "a significance/z-score was considered but is uninformative";
* 7.4 / 7.5 the comparisons to a robust loss and to joint refinement
as approaches that had failed;
* 9.2 the R_meas / CC1/2 outcomes attributed to the trimmed-mean
background;
* 9.3 "per-detector-region and crystal-anisotropy profiles were
evaluated and add nothing";
* 10.2 the stills tilt "succeeds where a freely-fitted width
collapses";
* 10.5 the CC_anom argument, trimmed to why the statistic behaves as
it does;
* 10.6 the survey of per-frame correlation medians across datasets;
* 13 the space-group bullet, restructured into the three gates it
actually applies, dropping the dataset anecdotes;
* 14.2 the free-form per-shell rescale, stated as a design choice
rather than an experiment.
Section 13's space-group text was one 20-line paragraph; it is now a
numbered list of the three tests, which is what the code does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotation kept a 1.5 A high-resolution limit for spot finding on the
strength of one indexing-rate measurement (100.0 -> 96.8% on a strong
set). Measured properly, over the whole 33-crystal rotation battery, it
does not earn its place:
* no space-group decision changes - the same 30/33 agree with XDS, and
the three that differ are the known pre-existing cases, unchanged;
* 29 of 33 crystals are identical to the digit - same indexing rate,
R_meas, CC1/2, ISa. The limit was doing nothing on the large
majority;
* where it does bite, the limit is the worse setting. The one crystal
that loses appreciable indexing rate without it (99.50 -> 94.22%)
comes back with lower R_meas (29.4 -> 28.1), higher high-resolution
CC1/2 (27.9 -> 29.1) and higher ISa (5.77 -> 6.17). Another loses
0.4% of frames and gains 2.8 points of CC1/2_hi. Fewer frames
indexed, better data from them;
* runtime is unchanged (16m46s vs 17m32s over the battery).
So the indexing-rate cost is real but does not carry through to the
merged data, which is what the limit was protecting. Unset now means "as
far as the detector reaches" for rotation as well as stills;
--spot-high-resolution still sets a limit for weak, high-background data
where the extra high-resolution spots are genuinely noise.
This also removes the flag that distinguished "the user asked for no
limit" from "the user said nothing" - with no rotation default left,
both mean the same thing. While rewriting the comment block, corrects
its neighbouring claim that rotation keeps the fixed-threshold finder;
adaptive detection has been the default for both workflows since
6f4917dce.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten places where the document and the implementation had drifted apart.
Each was checked against the source before rewriting:
* 7.5 rotation post-refinement: it is TWO separate cross-validated
steps (cell+axis from the angles, then distance+beam from the
positions with the cell fixed), not one joint fit against the merged
fulls; the held-out split is an hkl hash, not a frame split; the
bounds are +-5% on distance and +-15 px on the beam, not "under
~1%"; and only the distance and beam centre reach pass 2, which
re-indexes de novo.
* 9.2 the trimmed-mean background: it is computed in the shared
background pass, so it DOES apply to --integrator boxsum. Only the
broadband sigma-clip is excluded. The section previously said both,
contradicting itself two paragraphs apart.
* 9.3 per-reflection profile rebuild, sub-pixel centring and radial
elongation are gaussian-only; the empirical profile keeps the fixed
per-shell grid and is accumulated on rounded predicted positions,
not centroids.
* 10.5 the asymptotic ISa and the b_ISa sigma floor are rotation-only;
stills report 1/b and floor with the whole-range b.
* 8.4 centering absences are applied only when the user fixes the
space group - de novo, prediction runs in P so the search can
confirm the centering from the intensities.
* 10.6 per-batch relative-B cross-validates on ASU-group parity, not
the frame parity the other surfaces use.
* 13 the resolution cutoff sits one reported-shell width PAST the
CC1/2 = 0.30 crossing, so data below 0.30 are kept.
* 7.1 the orientation-only prior penalises all three components of the
angle-axis vector.
* 10.1 rlp is the RECIPROCAL Lorentz factor, L = 1/rlp.
* 14.3 the model scaling is fitted over work and free reflections
alike, so R-free is free of refinement, not of the scaling fit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Making azim_int_settings.high_q_recipA and
spot_finding_settings.high_resolution_limit optional changes the wire
format: when unset they are omitted rather than sent with a placeholder,
and a client generated from an older spec does j.at() on them. The
azimuthal-integration limit now defaults to unset, so a stock broker
omits it out of the box - the break needs no operator action to hit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first-pass spot cache constructed an MXAnalysisWithoutFPGA and an
AzimuthalIntegrationProfile inside the per-image lambda, so every cache
miss allocated a CUDA stream, the preprocessing buffer, the spot finder,
the azimuthal integrator and the Bragg engine, used them for one frame,
and freed them again - hundreds of times, serially, on the
--redo-rotation-spots path. Both worker loops already hoist the same
object out of their loop; only this path did not.
Build them once for the whole first pass, and only when spots actually
have to be found (with --reuse-rotation-spots there is nothing to
allocate). Reuse is safe because every azimuthal-integration path -
CPU, GPU and the fused adaptive engine - clears the caller's profile
before adding to it, so each frame's output is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The expected-variance weights decompose an observation's sigma^2 into a
background part and a Poisson signal part, then rebuild the signal part
at the reflection's merged mean. The decomposition subtracted corr*I with
I taken as-is, so a negative I ADDED to the background part: an
observation at I = -1.5 with sigma^2 = 1 came out with a base variance of
2.7 rather than 1.
That inflates the variance of precisely the down-fluctuated observations
the correction exists for. Below about one photon they are then
under-weighted and the merged mean is biased high - the same direction of
error, in the same regime, that weighting by the observation's own sigma
produces. Subtract max(0, I) instead: a negative intensity has no Poisson
signal to remove.
Both users of the decomposition are fixed - the stills merge, where
expected-variance weighting is now the default, and the rotation combine
it was mirrored from, which had it first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Passing 0 reset the limit to "unset" and logged "No high resolution limit
for spot finding: as far as the detector reaches" - and then, 700 lines
later, the rotation default put 1.5 A back, because unset carried two
different requests: the user said nothing, or the user asked for none.
The log said one thing and detection did another, and there was no way to
lift the limit on rotation data at all.
Remember whether the option was given, and apply the rotation default
only when it was not. Documented in the usage message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two-arm check for "same order, different symmetry" ran after the
all-observation arm had already been adopted by value, so it compared
that arm against a copy of itself: the point-group names were always
equal and the branch was dead. In the other direction the orders were
always unequal, so it was dead there too. The AMBIGUOUS warning - written
for the case where the two merges support different symmetries of the
same order, which implies two different molecular-replacement searches -
could never be emitted.
Check before the adoption, while both arms still hold their own result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two-arm search compared its arms by the order of the space group each
had picked, but Stage B leaves best_space_group unset whenever no
candidate is eligible - no candidate had enough observed absences to
trust. That is not rare on the Lorentz-filtered arm, and for a systematic
reason: the filter removes the badly-measured observations, which is
where the weak systematically-absent reflections are.
An arm that confirmed 422 but stopped short of naming a space group
therefore scored order 0 and lost to an arm supporting P2, and the
demotion was logged as "taking the higher symmetry" - the comparison and
the message both wrong, in the one direction the design says cannot
happen.
Carry the point-group order in the result, set from the order Stage A
actually adopted, and compare on that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mapping took its width and height from the CONVERTED geometry
unconditionally, while pixel_to_bin is sized per mode: converted when the
geometry is transformed, raw module layout when it is not. In raw mode
the two disagreed - 2068x2162 reported against a 1024x4096 map on a JF4M.
Only the adaptive spot finders read those dimensions, and they read them
for exactly the thing that breaks: the CPU finder derives npix = w*h and
then indexes the image, pixel_to_bin and the resolution mask with it, so
it walked ~277k pixels past the end of all three; the GPU finder stays in
bounds but decodes the strong-pixel bit index with the wrong row stride
and reports spots at wrong coordinates. Nothing combines raw geometry
with adaptive detection today, so this was latent rather than live.
Take them from GetXPixelsNum()/GetYPixelsNum(), which already follow the
geometry mode. The converted path is unchanged - it is the same number
there - and every internal use is inside SetupConvGeom, which only runs
when the geometry is transformed.
Covered by two tests: the mapping's dimensions must match pixel_to_bin in
both modes, and the CPU adaptive finder must return a spot planted on a
raw-geometry image at that raw pixel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Clear() reallocated sum and count but left sum2 as the constructor had
sized it. Every caller reuses one profile across the images of a dataset
(rugnux keeps one per worker, the viewer one per view), so the squares
kept accumulating while the means restarted at zero: the per-image
standard deviation written to /entry/azint and shown in the plots was
meaningless from the second image on, and grew without bound over a run.
The size mismatch was the sharper edge. Clearing to a mapping with a
different bin count left sum2 shorter than sum, and GetStd() and
operator+= then read past its end - reachable in the viewer by opening a
dataset with a wider q range than the one before it.
Both are covered by tests: the same frame twice with a Clear() in
between has to give the same standard deviation, and a profile cleared
to a wider mapping has to report the right value in a bin that only the
wider mapping has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The search runs in pass 1 of the rotation two-pass; pass 2 only reuses the
group it decided, so the result the CLI renders carried no search at all
and the whole report - operator correlations, the space-group candidate
table, the refused higher symmetry - was silently dropped on every
rotation dataset. Most costly of all, the "or <group> (indistinguishable
from these data)" line never appeared, so an enantiomorphic pair the
intensities genuinely cannot separate was reported as a single answer.
Carry pass 1's search into the returned result.
Also de-duplicate the alternatives when the centred-lattice test swaps the
metric-matching candidate into the answer: the group it displaced was left
out and the chosen one listed twice ("C2 or P21 or C2").
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A reflection the group predicts absent counted as a violation when
I/sigma > 3 AND E^2 = I/<I>(shell) > 0.3. Neither half survives contact
with real data:
* merged sigma is floored at b|I|, so merged I/sigma saturates at ISa
for nearly every reflection - the I/sigma half is an on/off switch
keyed on ISa vs 3, not a per-reflection test. On one crystal the
absent class read <I/s> 4.10 against 3.73 for the present class while
being genuinely extinct;
* <I>(shell) decays with resolution while a systematically-absent
reflection keeps a small NON-decaying residual (background / profile
leakage), so absent reflections drift over an absolute E^2 cut at high
resolution. That cost a tetragonal 42_12 crystal its 4_1: 18 of its 47
absent 00l crossed the cut, all beyond 3.7 A, at absolute intensities
identical to the low-resolution ones correctly judged absent, while
their l=4n row-mates sat 20-60x higher at the same resolution.
A screw extinguishes only the reflections that lie ON its axis, so the
fair yardstick is the rest of that same row. The threshold is now
0.3 * max(1, median E^2 of the reflection's own row), the row being the
gcd-reduced reciprocal-space direction and the control class the same-row
reflections the group predicts present. Floored at 1, so it only ever
relaxes: a screw can be recovered by it, never lost.
Per row, not pooled. A 4_1 along c and a 2_1 along a are separate
conditions with separate controls; pooling let the weak a/b rows (median
E^2 ~0.5) set the threshold for a strong c row (8.4) and the rescue never
fired.
The candidate table now reports the screw evidence (median E^2 of the
absent class and of its rows) - the <I/s> columns are the centering
evidence and say nothing about screws, for the sigma-floor reason above.
Rotation battery, 33 crystals: 31 decisions bit-identical, the 42_12
crystal recovers its 4_1 (0 violations, row E^2 8.4 vs absent 0.12), and
one crystal with a long axis and heavy 00l overlap moves to a 4_1 group at
exactly 10.0% violations - marginal, and its sister crystal of the same
form sits at 13.3% and does not move. Real screws now span 0-9.3%
violations, so max_absent_violation_fraction cannot be tightened below
0.10 without risking a genuine one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--search-min-zeta now defaults to 0.85 for rotation, so the de-novo search runs
on a merge of all the observations AND on a merge of only the well-measured
ones, and keeps whichever found more symmetry. Previously it shipped off and
the second opinion had to be asked for.
Rotation battery, 33 crystals, NO flags beyond the resolution limit:
fixed-threshold finder 30/33 - zero crystals differ from the single search
adaptive finder 30/33 - the same three mismatches
Both arms now agree crystal for crystal, which they have not done before. The
last disagreement was a thaumatin set whose 4-fold sits 88.9 deg from the
spindle: at defaults it now reads P42(1)2 (all-observation merge -> 222,
Lorentz-filtered -> 422, higher taken) where it read P222. The classic arm is a
strict no-op - zero differences against both the explicitly-flagged run and the
run predating the dual search - so the default costs nothing where the geometry
is not the problem, and 47.2 s against 47.8 s on the same crystal back to back.
The default is safe to set because the two searches can only disagree by a LOST
operator: discarding observations starves an operator correlation, it cannot
invent one. That also makes the 0.85 itself uncritical - too aggressive a cut
only means the second opinion contributes nothing and the full merge wins.
--search-min-zeta 0 restores the single search.
Docs: CHANGELOG gains a 1.0.0-rc.161 section covering the branch, and
CPU_DATA_ANALYSIS records the four analysis changes of this work - the
confidence-weighted per-image refinement, the collapsed per-frame scale guard,
the opt-in per-image rejection, and the operator-disagreement criterion with
the two-search rule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--search-min-zeta rescues a point group that the full merge cannot confirm, but
used on its own it is a trade: on the crystal it was built for it recovers the
correct 422, and on four others it costs the space group outright, because
discarding 40-80% of the observations starves operator correlations that were
perfectly healthy. Both ways of applying it - filtering the pairs that enter
the statistic, and filtering the observations that enter the merge - trade the
SAME crystals, so the cut itself is the problem, not where it is applied.
Filip's observation makes it one-way: every disagreement between the two is a
LOST operator, never an invented one. Discarding observations can starve a
correlation; it cannot manufacture symmetry that is not there. So run the
search on both merges and keep whichever found MORE symmetry, and the failure
mode disappears - each arm rescues the other exactly where it fails.
crystal all observations Lorentz-filtered adopted
thaumatin (weak) 222 422 422
tetragonal lysozyme 422 222 422
cubic insulin x3 23 2 / 222 23
The filtered merge is used ONLY to rescue the point group. The screw and
centering determination always comes from the merge with all the observations,
because systematic absences are decided by the WEAK reflections and the filter
throws most of them away. Preferring the filtered arm on a tie is not a
conservative choice, it is a wrong one: it cost four crystals their screw axes
(P2(1) read as P2, P4(1)2(1)2 as P42(1)2) with the point group and every
intensity statistic identical - a regression invisible to CC1/2, R_meas and ISa.
Where the two find the same ORDER but different symmetry, nothing can prefer
one, so the run says so: it names both space groups, states that the data do
not decide, reports which one processing continued in, and gives the flag to
force the other. Two candidates of the same order imply different molecular
replacement searches, and trying both is cheap next to reprocessing - much
cheaper than a confident wrong answer.
Rotation battery, 33 crystals, both spot finders:
fixed-threshold finder 30/33 - ZERO crystals differ from the single search
adaptive finder 30/33 - the same three mismatches, gap CLOSED
The adaptive finder now matches the fixed-threshold one exactly, which it has
not done before: its last remaining loss was the thaumatin set whose 4-fold
sits 88.9 deg from the spindle, and it now reads P42(1)2 (all-observation merge
-> 222, Lorentz-filtered -> 422, higher taken). A merohedral twin stays refused
in BOTH arms at all three frame ranges where it over-promotes, and at one of
them the second opinion is strictly better than shipping behaviour - the full
merge collapses to P1 where the filtered one finds the correct H3.
Cost is the extra scale-combine-merge on already-ingested partials, with no
re-integration: 47.2 s against 47.8 s on the same crystal back to back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zeta is the sine of the angle between a reflection's rocking path and the
spindle. Near 0 the reflection crosses the Ewald sphere almost tangentially,
spends many frames in diffracting position and is measured worst. The de-novo
space-group search asks how EQUAL an operator's paired intensities are, so its
answer is dominated by whichever reflections are measured worst - and when the
spindle lies in a lattice plane, an operator that permutes the two in-plane
axes samples a different mixture of measurement qualities than one that only
flips signs. That is not a fair comparison, and it can make a real symmetry
operator look like a twin law.
Measured on a thaumatin set mounted that way (its 4-fold is 88.9 deg from the
spindle), the added operators' disagreement is 1.74x the parent's over pairs
where both reflections have zeta < 0.85 and 1.003x - i.e. the symmetry is
exact - over pairs where both are above it. The search consequently refuses the
422 promotion and merges the crystal in P222, while the same data forced to the
right group give CC1/2 99.2% at multiplicity 10.7, matching XDS.
With the option the de-novo pass ignores those observations (the final merge
keeps everything - there completeness is the point):
zeta cut observations ignored H ratio adopted
0 (off) - 1.47 P222
0.5 1620648 1.44 P222
0.7 3006013 1.34 P21212
0.85 4536724 promoted P4212 (correct point group)
OFF BY DEFAULT, and it must stay off, because the same cut costs four other
crystals their space group (P41212 -> P212121, I23 -> P2, I23 -> I222 twice):
at 0.85 it discards 40-80% of all observations, which on a crystal whose
geometry is not the problem simply starves the search. Two independent
implementations - filtering the pairs that enter the statistic, and filtering
the observations that enter the merge - trade exactly the same crystals, so
this is a property of the cut and not of where it is applied. Verified
bit-identical to the previous binary when off.
The companion diagnostic is already there: the run now reports how close a
symmetry axis lies to the spindle, which is the geometry that makes this
option worth reaching for.
Implementation note for anyone tempted by the cheaper route: excluding these
observations from the ASU grouping alone does NOT work. The 3D combine selects
partials on corr, not on their group, so their intensity still reaches the
fulls and the merged intensities are unchanged - measured, the statistic did
not move by 0.03 while 67% of observations were nominally excluded. Zeroing
corr is what removes an observation from the combine, the merge and the error
model alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A rotation sweep never records the reflections whose reciprocal vector lies
within the Bragg angle of the spindle - the blind cusp. Symmetry normally
supplies them from an equivalent elsewhere in reciprocal space, so the hole
closes. It cannot when a symmetry axis IS the spindle: the cusp is then mapped
onto itself, every reflection in it is equivalent only to other reflections in
it, and it stays empty however long the sweep runs. The user can fix this at
the microscope - re-mount, or add a sweep on another axis - but only if they
are told, and nothing in the output mentioned it.
Report the smallest angle between any proper rotation axis of the adopted
space group and the goniometer axis, always on rotation data, and warn when it
falls under 15 deg. The axis is found by projecting onto each operator's
invariant direction (the sum of its powers annihilates everything else) and
mapping that fractional direction through the refined lattice into the lab
frame; the angle is invariant under the sweep, so the reference orientation is
enough. Cross-check: this reports 30.6 deg for a crystal whose 4-fold an
independent analysis of the XDS orientation matrix put at 30.5 deg.
Measured on three rotation sets: 13.6 deg (2-fold, warns), 16.2 deg (2-fold,
99.7% complete) and 30.6 deg (4-fold). The 15 deg bound is practical rather
than derived - the blind cone's half-angle is the maximum Bragg angle, ~15 deg
for 2 A data at 1 A wavelength - and the wording says what the diagnostic can
honestly support: the angle is a risk indicator, the loss is confined to the
cone rather than spread over the data, and overall completeness may still look
reasonable while the region near the spindle is empty. It does not promise a
completeness number, because across those three sets the overall figure does
not track the angle (99.7% at 16.2 deg, 92.6% at 30.6 deg).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DetectorSetup hardcodes bit_depth_image = 16 for every DECTRIS detector
(DetectorSetup.cpp:78), and GetByteDepthImage() consults that BEFORE the depth
the reader takes from the file - so a file storing 32-bit images had its
overflow computed as a 16-bit one. With the reader also declaring the images
signed, GetOverflow() returned INT16_MAX and GetSaturationLimit() became
min(file value, 32767).
Every count above 32767 was therefore marked saturated, and because the
integration accept gate requires ALL inner pixels valid, the whole reflection
was discarded. That silently removes the strongest reflections of a strong
crystal - the low-resolution ones that anchor scaling - while the file itself
declares saturation at 105000-133000.
Measured on a lysozyme rotation set (200 frames), before -> after:
saturated pixels per frame 0.815 -> 0.000
brightest accepted pixel 32738 -> 87633
mean per-frame maximum 26218 -> 37390
i.e. the ceiling was exactly INT16_MAX and nothing genuine reached it.
Which datasets this touches depends on how bright they are: measured pixels
above the old ceiling range from 0.0 per frame on some rotation sets to 6.1 on
others, so the fix is a no-op on weak data and only ever adds reflections.
Rotation battery, 33 crystals: no point group changed (30/33 before and after)
and no run failed. Four crystals move on quality, in both directions - ISa
1.90 -> 2.40 and 3.29 -> 4.80 on two, 2.97 -> 1.85 and 20.83 -> 18.47 on two
others; three of the four are the battery's known weak or run-to-run-unstable
crystals. The one strong crystal that moves gains 136 observations out of
1.9 million and loses 2.4 ISa: the reflections restored are by construction the
brightest ones, and they carry the systematic error that the strongest
reflections always carry. That is a real cost, but it is the cost of MEASURING
them rather than discarding them unseen, and a lower asymptotic I/sigma on data
that are now complete is preferable to a flattering one on data that quietly
are not.
Only the offline file reader is affected; the online path builds its detector
setup from configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It was held back because a 33-crystal rotation battery showed it breaking three
crystals deterministically - a lost space group, a halved indexing rate and a
collapsed merge. None of those causes turned out to be in detection.
The extra spots adaptive finds are real. Measured per spot against a
finder-neutral local background: 64% recur at the same position on the adjacent
frame (chance rate 0.5%) with 2-frame rocking curves, and 0.00% would fail a
conventional local SNR >= 4 test, median local SNR 34. What they include is
genuine peaks belonging to no lattice the indexer found, and the damage they did
scaled with their absolute COUNT (80.6 per frame against 36.8 for the fixed
finder), not with their quality - which is why nothing aimed at judging
individual spots ever worked.
The three failures fell to fixes elsewhere:
merge collapsed - a per-frame scale free to collapse toward zero amplified
two junk frames by 546x (704098712, ec7a82613). Not a
detection problem at all: the fixed-threshold finder trips
the same bug on a different frame range.
space group lost - the per-image geometry refinement was dragged 2.3-2.8 deg
off by the weak-spot tail in an unweighted fit; weighting
each spot by how strong it is FOR ITS RESOLUTION fixed it
(ae126c3d5), and gained a point group for the fixed finder
too.
indexing halved - gone with the same two; that crystal is now better under
adaptive (CC1/2 92.6 -> 95.9, high-shell 44.9 -> 56.4).
Battery, 33 crystals, adaptive vs the fixed finder:
exact space group matching XDS 26/33 vs 25/33
point group matching XDS 29/33 vs 30/33
ISa better on 6 crystals
recovers a screw axis the other misses (P321 -> P3121)
The one point group it loses is a tetragonal crystal where adaptive collects
2.4x the observations at better R_meas (21.4% vs 28.3%) and better ISa (4.58 vs
2.91), and forced to the right group gives CC1/2 99.2% at multiplicity 10.7 -
matching XDS. Only the automatic symmetry call fails there, and six candidate
causes have been measured and refuted (mixed indexing hands, off-crystal frames,
a badly integrated minority, radiation damage, pseudo-tetragonality, uncorrected
anisotropy). It is left as the subgroup, which is the recoverable direction: -S
gives XDS-quality data from the same run, whereas the failures this unblocks
were not recoverable.
--no-adaptive-spots reverts to the fixed-threshold finder.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The flag was accepted on rotation data and did nothing - it is read only by the
stills merge (Merge.cpp), and the CLI warned about that rather than fixing it.
Meanwhile RotationScaleMerge already COMPUTES a per-frame correlation against
the merged reference and writes it to the per-image table; nothing acted on it.
Wire the two together. A rejected frame has its partials' corr set to 0, which
is how a frame already leaves the pipeline - every consumer requires corr > 0,
so the combine, the merge and the error model all drop it together. The GPU
path reuses the SmoothCorr kernel with a ratio of 0, so one implementation
covers both. Off by default (0), and verified bit-identical to the previous
binary when off.
What it catches, on the two rotation datasets that have a population to catch:
a two-lattice crystal - two lattices in two physical AREAS of the sample, so
the sweep passes from one to the other and whole blocks of frames measure a
different crystal from the one being merged (frames 500-700 index perfectly
well at a per-frame CC of 0.22 against 0.47-0.56 either side, in 11 contiguous
runs). R_meas 28.6 -> 24.6%, CC1/2 93.6 -> 95.1, high-shell CC 23.4 -> 38.3.
a second dataset with 9.5% of frames below CC 0.30: R_meas 24.3 -> 23.4%,
CC1/2 92.6 -> 93.4.
The criterion is "this frame disagrees with the merged reference", NOT "this
frame is off-crystal". It happens to catch both, because a frame that measures
nothing and a frame that measures a DIFFERENT crystal fail the same test, and it
does not need to know which. For the two-area case that is a workaround, not a
treatment: it recovers one crystal by discarding the other, where processing the
two as separate sweeps would keep both. The frame-block structure is clean
enough that such a split could be detected automatically.
WHY THERE IS NO DEFAULT. The per-frame CC is not comparable between datasets -
it is as much a measure of data quality as of frame validity. Measured medians
across the battery run from 0.30 to 0.81, so one absolute bound removes 13
frames from one dataset and 584 of 1800 from another:
battery at --min-image-cc 30, 33 crystals: no point group changed (30/33),
four crystals clearly better (one +5.4 CC1/2 points, the two-lattice case
above, and ISa gains of 1.3-4.6 on three others) - and one healthy crystal
lost a third of its frames and with them its high-resolution shell
(CC1/2_hi 26.2 -> 2.0).
This is the same trap as an absolute bound on any per-operator or per-frame
agreement statistic, and the same one the per-frame scale guard avoids by
measuring against the run's own median. A principled version would cut on the
SHAPE of the per-frame CC distribution - a dataset with a bad subpopulation is
bimodal, a uniformly weak one is not - rather than on an absolute value. Until
that exists this stays opt-in, and the per-image CC it keys on is already in the
_image.dat table for anyone choosing a value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A merohedral twin mixes EVERY reflection with its twin mate, so it shifts the
whole distribution of |I1-I2|/(I1+I2). A minority of badly measured
reflections shifts only the tail. The mean cannot tell those apart; the median
is blind to the second and just as sensitive to the first.
Measured on real crystals, moving the statistic from the mean to the median
leaves genuine promotions where they are and pushes every twin up:
genuine tetragonal 1.016 -> 1.013
genuine lysozyme 1.051 -> 1.067
genuine tetragonal 1.238 -> 1.231
twin (-e 1050) 1.272 -> 1.447
twin (-e 450) 1.280 -> 1.622
twin (full) 1.441 -> 1.522
twin (-e 600) 1.427 -> 2.010
The margin around the 1.25 bound widens from 2.7% (genuine 1.238 against twin
1.272 - uncomfortably tight for a decision that cannot be undone downstream)
to 17.5% (1.231 against 1.447). The bound itself does not move.
Rotation battery, 33 crystals in both detection modes: no point group changed
in either (30/33 and 29/33, as before), and only one crystal's numbers move at
all - the one already documented as nondeterministic between repeat runs of
the same binary. The synthetic twin-fraction x multiplicity grid passes
unchanged. So this buys margin, not outcomes.
Found while testing a different hypothesis, which the same measurement refuted:
a tetragonal crystal whose 422 promotion is wrongly refused reads 1.484 by the
mean and 1.472 by the median, i.e. its disagreement is distribution-wide and is
NOT a badly-integrated minority. That crystal's cause is elsewhere and is not
addressed here - see the note below.
Its indexing-ambiguity operator (-k,-h,-l) lies INSIDE 422 but OUTSIDE 222,
so the subgroup merge the search is given mixes lattices indexed in the two
alternative hands. That corrupts exactly the 4-fold relationships and leaves
the 2-fold ones intact - measured, the 222 step reads 0.917 and the 422 step
1.484 - and the corruption is indistinguishable from a twin law. Forcing the
tetragonal group merges the two hands as equivalent and the same data give
CC1/2 99.2% at multiplicity 10.7, matching XDS. The failure is worse the
BETTER the frames index (99.9% vs 63.3% for the run that gets it right),
because indexing more frames picks up more of both hands.
So no statistic computed on a subgroup merge can arbitrate a promotion whose
added operators include an indexing-ambiguity operator. Fixing that means
resolving the ambiguity before the search, or detecting the coincidence and
deciding another way; the operators needed to detect it are already computed
(the run warns about them).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`RefineGeometryIfNeeded` hands XtalOptimizer the WHOLE spot list, not the
indexed subset, and the first pass admits anything within 0.3 fractional-Miller
units of an integer - which is 11.3% of RANDOMLY placed spots, since the
admitted volume is (4/3)*pi*t^3. Every one of them then enters an unweighted L2
fit with an arbitrary rounded index. On images with many detections the
refined orientation ends up 2.3-2.8 degrees from the goniometer-consistent one
and explains 14 of its own 250 spots where the undragged orientation explains
68; mosaicity and profile radius inherit the error and integration follows.
Weight every spot by its intensity divided by the median intensity of its own
equal-count resolution shell, applied as w^2 on the squared residual with
w^2 = r/(1+r). The shell normalisation is the point: refinement needs the
high-resolution spots because they carry the cell and distance, and those are
LEGITIMATELY weaker, so a raw intensity weight would suppress exactly the
spots the fit depends on. Measured, the weight is resolution-neutral - median
exactly 0.707 in every shell, and corr(w, 1/d^2) = -0.20 / -0.11 against
-0.32 / -0.34 for the same function of un-normalised intensity.
This is a PRIOR: it is computed from the spot alone and never looks at the
current residual, so unlike a robust loss it cannot mistake a genuine spot for
an outlier while the starting geometry is still far off and leave the fit
unable to move. That failure is not hypothetical - a CauchyLoss on this same
residual, at the scale the multi-frame GeometryRefiner uses, collapsed one
crystal's indexing rate from 99.89% to 19.83% and was rejected.
It does not work by telling good spots from bad, and it does not need to. No
per-spot property separates spots that index from spots that do not: measured
AUC is 0.53 for peak pixel, 0.53 for total intensity, 0.51 for pixel count,
0.45 for peakedness, and a logistic regression on all twelve available
features with pairwise interactions reaches only 0.64. What the weight does is
halve the EFFECTIVE COUNT of every spot (mean w^2 = 0.517), and the damage
scales with the absolute count of unexplained spots in the objective - 80.6
per frame here against 36.8 for the finder that was never damaged. That is
also why an empirical `--max-spots 66` cap works while leaving the list no
purer than before: it reaches the same operating point by discarding spots.
This reaches it without discarding any, and without a tuned constant.
Rotation battery, 33 crystals, both spot finders:
finder A 29/33 -> 30/33 point groups (one crystal P222 -> P4212 = XDS,
its high-shell CC1/2 86.0 -> 98.4)
finder B 28/33 -> 29/33 point groups (one crystal I222 -> I23,
its high-shell CC1/2 14.8 -> 38.0)
No crystal lost its point group in either mode and no run failed. On the
meta-stable multi-lattice dataset the CC1/2 spread over four frame ranges
falls 19.7 -> 13.1 for finder B, and the indexing rate rises in 8 of 8
configurations. The crystal that the rejected robust loss destroyed keeps its
99.89% indexing rate exactly.
The cost, stated plainly: ISa falls by 0.2-1.7 on about five crystals (and
rises on two). Point-group correctness is worth more than that - merging in
the wrong symmetry cannot be undone from the output, whereas ISa is a quality
metric of data that remain correct - but it is a real trade and not a free win.
Off by default. The indexers pass a spot list they have already selected, so
their calls are unchanged; only the per-image refinement, which gets the raw
list, turns it on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
704098712 guarded the per-frame scale on the partials, but it put the check
inside ComputeSmoothGWindow - which only the partials path calls. Step 4
refits the scale from scratch on the COMBINED FULLS (Unity model, so corr is
exactly 1/G), with no smoothing and no floor, and that refit was still free to
collapse toward zero.
It is the same failure and it is worse here, because there is no window
average to dilute it: the collapsed frame's own fulls are multiplied directly.
Measured on a dataset where the previous commit had already fixed the partials
stage, the fulls refit put 1/G = 559x and 175x on two frames carrying 517
observations, and the merged CC1/2 read 26.2% where the intensities ENTERING
that stage were fine - better, in fact, than the comparison run's in all ten
resolution shells (R_meas 34.5% vs 39.5%, CC1/2 89.1% vs 83.0%).
Reject a collapsed scale here as well, on the same measured criterion, and let
those fulls merge unscaled - the state the combine left them in, and the same
fallback the fit already uses for a frame with too few reflections. The check
reads the host fulls after both the CPU loop and the GPU ScaleFulls, so one
implementation covers both paths; the corrected corr is pushed back to the
device exactly as the correction surfaces already do.
THIS IS NOT A DETECTION-MODE PROBLEM. Over 8 configurations (both spot
finders x 4 frame ranges) the separation is exact: every run with a collapsed
scale had CC1/2 <= 58.5%, every run without had CC1/2 >= 74.1%, and nothing
else predicted it. On one frame range it is the DEFAULT finder that collapses
(CC1/2 58.5%) while the other is clean at 93.8%. The instability was never
specific to the finder; it was latent in the scaling stage and either finder
could trip it.
Same 8 configurations, with this commit:
finder A full 93.7 -> 93.7 (untouched)
finder A -s 1 58.5 -> 91.3 (recovered)
finder A -e 899 94.0 -> 94.0 (untouched)
finder A -e 898 87.2 -> 87.2 (untouched)
finder B full 26.2 -> 91.1 (recovered)
finder B -s 1 93.8 -> 93.8 (untouched)
finder B -e 899 8.1 -> 91.7 (recovered)
finder B -e 898 74.1 -> 74.1 (untouched)
Every collapse recovers; every healthy run is unchanged. The CC1/2 spread over
the four frame ranges falls from 35.5 to 6.8 points for one finder and from
85.7 to 19.7 for the other - this dataset was not sampling a deep instability
when its CC1/2 swung between 17 and 94 across frame ranges, it was sampling
whether this bug happened to fire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-frame scale enters every intensity as 1/G, and SolveScaleIRLS floors G
at zero and nothing else. A frame whose fit is not determined by its data can
return G ~ 0.002 against a run median of 0.865, and every observation it
carries is then multiplied by ~500 - sigma by the identical factor, which is
why no sigma-based outlier test can see it and why this looked for a long
time like a partiality problem. (The 1/partiality path is in fact guarded:
min_captured_fraction floors it at 0.7 by default on rotation.)
The window smoothing that should have absorbed such a frame instead made it
permanent. It averages log G over a window, so a scale collapsing toward zero
does not merely corrupt its own frame - its logarithm drags the whole window
down. Worse, where a run has a stretch of frames too sparse to fit at all, the
only FITTED frames in a window can be the collapsed ones, and the geometric
mean then averages the fault with itself. Measured on a multi-lattice dataset:
frames 816 and 818 fitted G = 0.0023 and 0.0014 with every neighbour from 800
to 839 unfitted, so smoothing set G = 0.0018 across the whole neighbourhood -
a 546x amplification. About 500 observations of 152000 (0.66%) then carried
99% of sum(I^2), and the merged CC1/2 read 17.2% where the same data with the
classic finder read 93.7%.
Treat a fitted scale far below the run's median as what it is - an
undetermined scale, exactly like the too-few-reflections case the code already
handles - rather than as a successful fit. Such frames no longer contribute to
the smoothing mean, and a frame whose own scale is not credible takes the
neighbourhood's, or the run's typical scale when the neighbourhood holds
nothing credible either.
The bound is a RATIO to the run's own median because the rotation per-frame G
is not gauge-fixed: G and the group means have an exact global multiplicative
degeneracy, and the fitted median drifts over 0.745-1.358 across the battery.
An absolute floor would reject everything in a run that drifted low.
MIN_CREDIBLE_SCALE_RATIO = 0.02 was chosen from measurement over 12 crystals
in the default configuration, where the smallest legitimate min(G)/median(G)
is 0.070; the failing case sat at 0.0017. It is 3.5x below anything real and
12x above the failure.
Effect on the intensity tail of the failing case: max I 10224 -> 438, and the
top 1000 observations' share of sum(I^2) 0.990 -> 0.421 (the classic-finder
reference is 0.632, so the tail is now cleaner than the run this was compared
against). Rotation battery, 33 crystals in the default configuration: ZERO
crystals differ - no space group, CC1/2, high-shell CC or ISa change anywhere.
The guard fires only on the pathology.
It does NOT rescue that dataset: with the amplification gone its CC1/2 is
26.2% and R_meas 49.2% against the classic finder's 93.7% and 27.8%. Adaptive
detection degrades those intensities for a second, independent reason that is
still open. This commit removes a latent hazard for any run with a sparse
stretch of frames; it is not the fix for that dataset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The refusal message fell through to the chi^2 branch whenever the
systematic-b balloon veto was the binding test, so it reported a chi^2 ratio
that did not justify the refusal at all - on one battery crystal it printed
"merge chi^2 is 1.25x the subgroup's (bound 1.85)", i.e. a number comfortably
inside its own bound, as the reason for processing in the lower symmetry. A
diagnostic that names the wrong cause is worse than none: it sends the reader
after the wrong statistic.
Report the b test when it is what fired, with both b values and the bound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The point-group decision moved with the AMOUNT of data at fixed physics: a
partially twinned trigonal crystal was promoted into the twin's holohedry
whenever the search happened to see a larger first-pass merge, and kept its
true subgroup when it saw a smaller one. Simulation over 6 noise draws with
only the merge multiplicity varying: the twin is promoted 0/6 at multiplicity
2 and 6/6 at 18, while the genuine control is promoted 6/6 throughout. The
cause is that every existing gate is a ratio to the merge error model -
b_parent grows toward the true systematic scatter as sigma shrinks with
1/sqrt(N), while b_cand is already saturated by the twin's disagreement, so
the ratio slides down through a fixed veto. The parent statistic moves with
data amount and the candidate statistic does not.
Gate promotions on the operator disagreement H = <|I1-I2|/(I1+I2)> instead,
as the ratio of the operators a promotion ADDS to the parent group's own
operators on the same reflections. There is no sigma in it, so it cannot
drift with the error model, and the parent normalisation cancels data
quality. Measured over 27 runs, 5 promotion types and 450-1800 images:
genuine symmetry 0.862-1.219, merohedral twins 1.270-2.084. On the synthetic
grid it is flat across a 9x change in multiplicity - genuine pinned at 1.00,
twins 3-12x the bound - which is precisely the property the old gates lacked.
chi^2 and the systematic-b stay as secondary vetoes; they protect against
non-crystallographic pseudo-symmetry, which is where correlation-based
scoring is weak.
Pick the parent carefully: 422 has two maximal subgroups of order 4, and on a
tetragonal crystal twinned by 2[100] the rival (222) is CC-confirmed too and
CONTAINS the twin laws, so normalising against it hides the twin among the
promotion's own real operators (ratio 8.19 against the true parent, 0.78
against the rival). Where several parents tie, judge on the most damning.
Also:
- Report a refused promotion instead of silently processing lower. Merging a
twin in the twin's holohedry averages non-equivalent reflections into each
other and cannot be undone from the output; keeping the subgroup costs only
redundancy. The refusal names the group and the number that caused it.
- Stop the twinning report from arguing in a circle. It ran after adoption and
conditioned on the adopted group, so a promotion into a holohedral Laue
class made it print "no merohedral twin law exists" - the test was
conditioned on the decision it should audit. Twinning is now also measured
on the subgroup merge before adoption, and the post-adoption text says when
its own conclusion is not authoritative.
- Compare PRIMITIVE cell volumes in the first-pass scheme tie-break. A centred
setting's cell is an exact integer multiple of its primitive one (a
rhombohedral lattice in hexagonal axes is exactly 3x), so the
integer-supercell test fired on a pure setting difference and demoted a good
scheme to a threefold-smaller merge - which is what let the twin see the
small merge to begin with.
Rotation battery, 33 crystals: point-group agreement 30/33 -> 29/33, one
crystal moved. That crystal (P422 -> P222) is the one with the known
unresolved integration defect where reflections near the rotation-axis plane
are wildly mis-integrated; its symmetry mates genuinely disagree, and its
lower-symmetry merge is measurably better (ISa 2.72 -> 3.63, high-shell CC
75.4 -> 86.0). The threshold was not moved to accommodate it: 1.25 sits inside
the measured gap and widening it would admit real twins. Separately the
tie-break improved one crystal's CC1/2 from 77.7 to 84.0.
Tests: a synthetic twin-fraction x multiplicity grid, which is what the search
had never had - the existing tests are noise-free and exercise only Stage B
absences.
A NOTE ON WHAT WAS TRIED AND REJECTED, so it is not rebuilt: the obvious
"physics-anchored" statistic is the disattenuated cross-validated correlation
rho = corr(I_half0(h), I_half1(Rh)) / corr(I_half0, I_half1), which is 1 for
real symmetry at any data quality and 2a(1-a)/((1-a)^2+a^2) for a twin. It
passes the synthetic grid perfectly and FAILS ON REAL DATA IN BOTH
DIRECTIONS - five false refusals of genuine symmetry on the battery, and it
waves through a twin (rho 0.998) that H refuses. The reason is that cc_half
correlates the two halves of the SAME reflection and so measures only random
error, while cc_cross compares DIFFERENT reflections carrying different
systematic error; dividing by cc_half removes the noise and leaves a
systematic floor that varies by crystal AND by operator. Genuine rho measures
0.9987 on strong data and 0.73 on weak. A synthetic generator validates a
statistic's arithmetic, never its premise, and this premise - that the only
departure from exact symmetry is noise - is false for every real crystal.
Any per-operator agreement statistic needs a same-crystal reference; an
absolute threshold on one cannot be made to work by tuning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--min-image-cc is consumed only by the stills merge (MergeOnTheFly); RotationScaleMerge
never reads it. On rotation data it was accepted and then silently did nothing, so a run
that looked filtered was not. It now says so.
FitProfileRadius_MAD had zero callers - a robust twin sitting uncalled next to the
non-robust estimator that is actually used is a trap, so it goes.
Neither changes any result: verified on a rotation dataset (indexing rate, cell, space
group and merge statistics identical, warning emitted).
Context for anyone tempted to wire that estimator in: I tested exactly that today and it
is NOT justified. The population it would clip is truncated by construction - a spot is
only marked `indexed` when its fractional-Miller norm is inside the indexing tolerance -
and is measurably shorter-tailed than Gaussian (kurtosis 2.85). Across four serial-stills
datasets a MAD-clipped variant only narrowed the prediction window (-17% integrated
reflections everywhere), which was neutral on strong data and destroyed real signal on
weak data (one set lost completeness 96.0 -> 93.9%), with R-free 0.3753 -> 0.3767.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One changeset, developed together in response to a review of this branch, so the
files carry several of the changes at once. Full test suite passes (733 cases).
Spot finding
- Split ImageSpotFinder into Detect() (flag strong pixels - the expensive
per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with
Run() = both. The per-image min-pix escalation now detects ONCE and repeats
only the cheap extraction, instead of re-running the whole finder four times
per frame as it did on the default path. It also keeps the winning attempt's
spot list rather than re-extracting it, so the frame that is integrated is
exactly the frame that was scored - which a GPU re-extract could not guarantee
(float atomic ordering).
- spot_finding_time_s no longer swallows indexing time, and indexing_time_s now
sums every escalation call instead of reporting only the last.
Detection limits follow the detector
- The azimuthal-integration upper q and the spot-finding high-resolution limit
are now std::optional, in the C++ structs AND in the OpenAPI schema, and
resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_
recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a
pixel outside that q range could never be strong - the integration range
silently bounded what detection could see, regardless of the requested
resolution limit. Regenerated the C++ and TypeScript clients; the viewer and
the web frontend each gained a "to detector edge" switch.
Detection defaults are now per workflow (measured, not assumed)
- Stills: adaptive detection, min-pix chosen per image, no resolution clipping.
- Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit.
On a 33-crystal rotation battery, adaptive detection helped four hard crystals
but deterministically broke three (a lost space group, a halved indexing rate,
a collapsed merge), and the detector-edge limit cost indexing on a strong
rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and
--no-adaptive-spots is new.
Indexer seed escalation
- Stop escalating once a seed's lattice explains >= 90% of the seed spots.
Previously any frame with >= 80 spots always paid three indexer calls, online
broker included.
Merge-consistency filter
- --min-image-cc gated on a per-image CC computed BEFORE the stills partiality
post-refinement and never refreshed; the refiner now recomputes it, so the
reported CC describes the data that are actually merged.
- Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the
merge, the error model and MergeStats can no longer disagree about which
images are in (the --scale path merged unfiltered while its statistics were
filtered).
Per-image B-factor refinement (-B) removed
- Measured on four serial-stills datasets: it is a no-op where the per-image fit
is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas
+23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on
14-25% of images). It had also been silently DISCARDED since the partiality
post-refinement landed - reported but not applied. Rather than fix and keep a
knob with no demonstrated benefit, the flag and the whole image_scale_b_factor
chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and
read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs.
ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS.
(The Wilson per-image b_factor is a different quantity and stays.)
Stills partiality width now fits both of its components
- sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d*
with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2
on d*^2. The angular-only width is fitted over a d*^2-dense population, so it
was pinned by the high-resolution edge and collapsed at low d*: median
partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55%
of them under the merge's partiality floor, and the survivors divided by those
values - which inflated the merged low-resolution intensity scale 3.6x
(~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x,
no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and
R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining
R-free are all blind to that ramp, which is why it survived earlier validation;
the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged).
Removed dead code from add-then-remove churn
- Prediction-time "still partiality" (unreachable: no setter), the phantom
IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the
constant it always was), StillsPartialityRefine's caller-less Settings
constructor and its reference to a long-gone env var, ProcessImage's unread
bool return, an unused include, and a dead viewer overlay hook.
Also
- Viewer: the magnifier compared a QImage with itself, so its scene rect was set
once ever and it could not pan into a larger dataset; the hover tail timer
could fire after leaveEvent and resurrect the resolution readout outside the
image.
- update_version.sh regenerated the frontend lock file BEFORE bumping the
version (every release shipped an off-by-one lock), and did git rm/git add on
a path that has not existed since the client moved to src/client - with no
set -e, both failed silently.
- fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a
redirect, not a test, so dkms add never ran.
- Unit tests for the adaptive-threshold host functions, which had none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit 681e80e46.
Dropped for now: QTextBrowser::setMarkdown leaves the inline LaTeX in
CPU_DATA_ANALYSIS.md as raw "$...$" source, and the document's own HTML/math
rendering wants sorting out first. Reverted rather than rewritten so the
implementation stays available in history to bring back afterwards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds Help > Data Analysis Algorithms, showing docs/CPU_DATA_ANALYSIS.md in a
window. The document is baked into the binary through the Qt resource system
(aliased to :/cpu_data_analysis.md), so it needs no docs/ directory at runtime
and cannot drift from the build it shipped with. Same shape as the existing
third-party licences window, created once and raised thereafter.
Limitation worth knowing: QTextBrowser::setMarkdown renders the headings, lists,
emphasis and inline code well, but it has no math support, so the inline LaTeX in
the more quantitative sections appears as raw "$...$" source. The descriptive
material - which is most of the 744 lines - reads fine. Fixing that properly
means either pre-rendering the document to HTML with a math filter at build time,
or sending the user to the Read The Docs copy instead; neither seemed worth doing
without knowing which you would prefer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drawing an ROI reported Sum 0 and Max 0 until the next frame was loaded, with
only the pixel count looking right because that comes from the ROI map rather
than from the data.
SetROIDefinition_i called RunROIOnly, which integrates whatever the preprocessor
buffer already holds. LoadImage_i only preprocesses an image when a ROI is
already defined, so the very first ROI is drawn on an image that was never
preprocessed: the buffer is empty and every sum integrates to zero. From the next
frame on a ROI exists, LoadImage_i preprocesses, and the numbers look correct --
which is what made this look like a refresh problem rather than a wrong call.
Use AnalyzeROIOnly, which preprocesses the image before integrating. It costs a
pass over the image per ROI edit, of the same order as one recolour, and ROI
edits already keep at most one recompute in flight (live_pending_ in
JFJochDiffractionImage), so the editing rate is bounded. I did not measure the
drag rate specifically.
Verified with a single frame loaded and no frame step: the first ROI drawn over
the beam centre reports Sum 66065, Max 2761, Mean 0.473, centre of mass
(787.3, 844.5). The Max equals the image's own reported maximum of 2761, as it
must for a box containing the brightest pixel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The statistics shown for a drawn ROI do not come from the view at all. Drawing
one promotes it to a named ROI, roiGeometryEdited goes to the reading worker, and
the worker's per-image results arrive in ImageData().roi, which is what the
Inspector's ROI section displays. So accumulateROI/CalcROI/roiCalculated were a
second implementation of the same thing whose output nothing read -- and the
worker's version is the better one: it handles the mask and it persists per
image.
Remove them. The view now owns only the ROI's geometry and gestures, which is
all the worker needs from it.
This corrects the previous commit's claim that nothing surfaces ROI statistics:
the Inspector does, via the worker. Verified by drawing a box over the beam
centre: Sum 65453, Max 1634, Mean 0.468, centre of mass (786.3, 843.4) against a
beam centre of (764, 850). Note the numbers appear from the next analysed frame
onward, since the worker attaches them at analysis time and the displayed frame
was analysed before the ROI existed -- that behaviour is unchanged here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drawing an ROI only means something where there are detector counts to
accumulate. Gate the gesture on a virtual AllowROI(), true only for
JFJochDiffractionImage: shift-drag, the resize handles, the hover cursor and the
"Clear ROI" context entry now do nothing in the azimuthal, grid-scan and
calibration views, which cannot report anything about a box anyway.
The statistics move out of the base class into the diffraction view and read the
int32 image directly, so no float copy of the detector image is built for them
either. With the labels already converted, image_fp is now untouched by the
diffraction view, and the lazy EnsurePixelValues machinery it needed is gone.
image_fp stays as the base's representation for the views whose data really is
float: the azimuthal profile, the grid-scan 1/sigma^2 map, and the calibration
viewer's eight source types.
Removed with it: the ROI readouts in the calibration and 2D azimuthal windows,
which were the only two consumers of roiCalculated -- the diffraction view
emitted it and nothing listened. Nothing surfaces ROI statistics now; the
pixel-mask case wants rectangles counting excluded pixels and deserves its own
design. JFJochViewerROIResult is still used by the side-panel ROI list, so the
widget stays.
Verified in the GUI: shift-drag in the diffraction view still draws the box,
turns it into a named ROI and runs the statistics; fit-view panel remains
pixel-identical to the pre-series baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two changes to the per-pixel value labels, which appear above 30x zoom.
They were up to 5000 QGraphicsSimpleTextItems created and destroyed on every
overlay rebuild - so on every pan step while zoomed in. Paint them in
drawForeground() instead: no item churn, no scene invalidation, and the text is
laid out in viewport pixels so it is a constant readable size rather than a
scene-space font scaled by 0.2. Same approach as the magnifier's labels.
The value text becomes a virtual, PixelLabel(). The base still formats from
image_fp, which is what the genuinely float-valued views hold (azimuthal
profile, grid-scan 1/sigma^2, the calibration viewer's eight source types).
JFJochDiffractionImage overrides it to read the int32 image directly: counts are
exact integers, so routing them through float32 is a detour that also cannot
represent summed values above 2^24 exactly.
Verified at 38 wheel clicks over a module edge: identical values and
gap/contrast handling to the previous float path, now centred in each pixel.
Fit-view panel still pixel-identical to the pre-series baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Users expect a magnifier to tell them the counts, which the follower view could
not do: it has the rendered pixels but not the numbers behind them.
Take them from the detector's int32 buffer directly, the same source the main
view colours from, so no float copy of the image is needed - the magnifier
still holds nothing full-size of its own, only a shared_ptr to the frame and
one to the reader image.
The labels are painted in drawForeground() rather than as scene items. The main
view creates up to 5000 QGraphicsSimpleTextItems per overlay rebuild for this;
here they are just drawn, so there is no item churn and no scene invalidation.
Text is laid out in viewport pixels so it stays a constant readable size, and
black/white is chosen from the luminance of the rendered pixel underneath, as
the main view does.
Threshold is the same 30x as the main view, so the default 12x magnification
shows no labels until the user wheels in; a cap keeps pathological window sizes
from drawing thousands of them.
Verified in the GUI at 32x: counts drawn per pixel with white text over the
dark centre of a Bragg peak and black elsewhere, and "Gap" across a module gap.
Main image panel still pixel-identical to the pre-series baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The magnifier and the main view are two views of the same image at different
position and zoom, but the magnifier ran the whole pipeline again on its own
copy: it wrapped the same int32 buffer in a SimpleImage, converted it to float,
coloured every pixel and kept its own full-size QImage. That is a second
conversion and two extra full-detector buffers (20 MB at 2.8 Mpx, 138 MB at
18 Mpx) to feed a 320x320 window.
Separate producing a frame from displaying one:
- JFJochImage keeps the rendered frame in a shared_ptr<QImage> (the pointer is
stable for the widget's lifetime; only the contents change, so the existing
buffer reuse is unaffected), publishes it via Frame() and announces new
pixels with frameRendered().
- JFJochImageItem holds that shared_ptr instead of a reference to a member of
its owner, which also removes a lifetime coupling.
- JFJochFollowerImage is a small read-only view of such a frame with its own
zoom and centre. It shows only the image: overlays, ROI tools and per-pixel
labels belong to the view that owns the data.
- The magnifier becomes one of those, fed from frameRendered().
Consequences beyond the saving: the magnifier now agrees with the main view on
colour map, contrast and HDR mode, which it never did -- it was wired to
neither, so it always drew with its own defaults. And the visibility guard
added in 6d1af4921 is gone: there is no longer any per-frame work to skip, so
nothing needs guarding. That guard was a workaround for this design.
Stepping 30 frames with the magnifier open: 5550 -> 4810 ms CPU, which is what
it costs with the magnifier closed (4770 ms) -- it is now free either way.
Verified: main image panel and a drag-pan stay pixel-identical to the
pre-refactor binary (AE=0); the magnifier follows the cursor, updates on a new
frame, and now tracks a colour-map change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The window is a placeholder for future functionality and is closed almost all
of the time, but it extracted the frame's spots and rebuilt and uploaded its
vertex arrays on every image, whether or not anything was on screen.
Guard it in rebuildGL() rather than at each of the eight call sites, so any
future caller inherits the behaviour: while hidden it only records that a
rebuild is owed, and showEvent() pays it. imageLoaded() additionally skips
extracting the frame's spots, which is the other half of the per-frame work.
The OpenGL code path is untouched and still built and exercised the moment the
window is opened.
Note: I could not show a CPU saving for this on the headless test machine --
there, ~74% of the process CPU is Mesa llvmpipe software rasterisation that I
was unable to attribute to any per-frame code path, and it swamps the effect.
The work being skipped is nonetheless unambiguously unnecessary.
Verified in the GUI: after stepping frames with the window closed, opening it
shows the current frame's spots, and it keeps updating while open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
centerAt() checked isVisible(), but imageLoaded() did not, so every frame built
a SimpleImage over the whole detector image and ran it through the full
JFJochSimpleImage path -- convert to float, colour every pixel, redraw -- to
feed a 320x320 window that is closed by default and stays closed most of the
time.
Remember the frame instead and do the work in showEvent(). Holding the
shared_ptr also keeps alive the buffer that the SimpleImage's CompressedImage
points into, which it did not own.
Stepping 30 frames with the magnifier closed: 5545 -> 4770 ms CPU (-14%), on a
2.8 Mpx detector; the saving is per-pixel, so it grows with detector size. With
the magnifier open the cost is unchanged (5500 ms), which is what was being
paid unconditionally before.
Verified in the GUI: opening the magnifier still populates it, and it still
refreshes when the frame changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FullViewportUpdate redraws the whole viewport on any change. The attached
comment ("keep overlays in pixel units independent of zoom") does not describe
what the setting does, and nothing here needs it: SmartViewportUpdate repaints
the changed rectangles and falls back to a full repaint by itself once there
are too many to be worth tracking.
This is the view used by the calibration window and the magnifier, and the
magnifier is driven from every hover, so on a remote session it repainted
its whole viewport per pointer motion.
Note: not exercised visually -- both windows are opened from menus, which the
headless harness does not drive. The change is a repaint-mode switch with no
effect on what is drawn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hovered "d = ... A" readout was a QGraphicsTextItem flagged
ItemIgnoresTransformations, repositioned on every mouse motion. Qt cannot
compute a tight dirty rect for an item that ignores the view transform, so it
marks the entire viewport dirty whenever such an item moves or changes text --
and this one moved constantly.
Paint it in drawForeground() in viewport pixels instead, and repaint only the
union of its old and new rectangles. That also removes the item lifetime
special-casing: it was deliberately kept out of overlay_items_, had to be
nulled by hand after scene()->clear(), and carried comments in three places
warning about the dangling pointer.
This does not reduce raw X11 traffic -- there every repaint uploads the whole
window whatever the damage -- but it cuts the work per hover, and it does
matter under a compressing remote protocol (VNC/NX/xpra), which encodes only
the region that actually changed.
Verified against the previous build: same text, colour and position.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The status bar, the resolution readout and the magnifier were all regenerated
on every single mouse motion event. Each regeneration repaints, and on a remote
X session a repaint uploads the whole window regardless of how little changed,
so the pointer merely crossing the image saturates the link: measured with a
counting relay in front of the X server, 50 motions over the image cost 273 MB,
and a build with the hover work removed cost 18 KB.
Rate-limit it. Two details matter:
- The limit is applied inline, not from a timer. Running the update inside the
mouse event keeps its damage in the same repaint as anything else that event
triggers (a pan). A first attempt deferred the work to a timer instead, which
split one repaint into two and made panning measurably worse.
- The catch-up that reports the final position is debounced, not queued per
skipped motion, so it fires once after the pointer stops rather than
repeatedly mid-gesture.
mouseHover() now takes the scene position and modifiers instead of the event,
which also removes the identical mapToScene() from all four implementations.
Hover traffic over 3 repeats: 173 MB mean -> 140 MB, and the run-to-run spread
drops from +-14% to +-2%. The harness tops out near 30 motions/s, barely above
the 15 Hz limit; a real mouse reports far faster, where the cap does more.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
image_fp is the base class's one pixel representation, and it earns that for
three of the four image widgets: the azimuthal image is already float, the grid
scan holds computed 1/sigma^2 floats, and the calibration viewer accepts eight
source types from uint8 to float64. The diffraction image is the odd one out --
its source is a large int32 buffer -- and it is the one paying: a full
int32 -> float pass plus a second resident copy of the image, on every frame.
Split the mapping from the source. PixelColorMap holds the precomputed LUT
constants and does value -> colour; a virtual ColorRow() picks the pixels out of
whatever buffer the subclass has. Both paths now go through the same Apply(), so
only the gap/bad/saturated dispatch differs, and it lines up exactly with the
encoding LoadImageInternal used:
GAP_PXL_VALUE -> NAN -> gap
ERROR_PXL_VALUE -> -INF -> bad
SATURATED_PXL_VALUE -> +INF -> saturated
The base class still needs real pixel values for ROI statistics and per-pixel
labels, so image_fp is filled on demand instead of per frame -- and only when
something reads it: a non-empty scratch ROI, or labels above 30x zoom. Neither
happens while simply looking at frames, and nothing else routinely sets roiBox
(the named ROIs are computed in the reading worker, not here).
18.1 Mpx: 10.5 -> 5.9 ms per frame and 72 MB less resident. 4.5 Mpx: 1.8 -> 1.0 ms
and 18 MB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DrawResolutionRings traced every ring point by point on each overlay rebuild:
361 ResPhiToPxl calls per ring, so about 4000 geometry evaluations per rebuild
with the 11 ice rings shown -- and a rebuild happens on every pan step.
The contours depend only on the ring list and the geometry, neither of which
changes while the view moves, so keep them. The cache is keyed on the ring list
(which RingMode::Auto recomputes from the visible area, so it still re-traces
when it should) and cleared in loadImage for a possibly-new geometry. Labels
are still placed per rebuild: they depend on the visible rect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Panning called updateOverlay() three times per mouse move: once for each
scrollbar's valueChanged -> onScroll(), then once explicitly. Zooming was the
same. Every one of those tore down and rebuilt every overlay item.
Suppress onScroll() for the duration of the gesture instead, and let the
gesture do its single rebuild at the end. Note this cannot be done by blocking
the scrollbars' signals: QAbstractScrollArea drives the actual scrolling off
valueChanged, so blocking it would stop the view moving at all.
updateOverlay() also refreshed the image item unconditionally, which marks the
whole item dirty and forces a full-viewport repaint even though pan and zoom
never change the pixels. Track whether RenderImage has run since the last
refresh and skip it otherwise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"auto img = image->Image()" deduced std::vector<int32_t> by value, so every
frame copied the whole detector image before converting it -- 72 MB on a 16 Mpx
detector. Bind a const reference instead.
The sentinel-to-float conversion also ran single-threaded on the GUI thread;
spread it over rows the same way RenderImage does. 18.1 Mpx: 11.8 -> ~1 ms,
plus the copy that is now gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ctrl+wheel, Shift+wheel and the foreground slider each recoloured the whole
image synchronously, once per input event. On a large detector the recolour is
slower than the events arrive, so they queued up and the view lagged behind the
cursor for as long as the user kept scrolling.
Defer the recolour to a zero-delay single shot and drop the intermediate
values: at most one recolour is in flight, and it always uses the newest
foreground/background.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every recolour ended with QPixmap::fromImage(), which allocates a second
full-size buffer and converts the whole image into the screen format. That
conversion was the largest single cost left in the colouring path.
Replace QGraphicsPixmapItem with a small item that paints qimg_buffer_ with
QPainter::drawImage. The buffer is already what the raster engine wants, so
nothing is converted or copied. The item declares its opaque area, as the
pixmap item did, so the view still skips the background fill underneath it,
and it turns SmoothPixmapTransform off before drawing to keep the
nearest-neighbour sampling QGraphicsPixmapItem gave us by default -- zoomed-in
detector pixels stay sharp squares.
GeneratePixmap is renamed RenderImage: it no longer makes a pixmap.
18.1 Mpx recolour: 22 -> 5.6 ms (28.0 ms before this series).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GeneratePixmap wrote every pixel twice: once into the QImage and once into
image_rgb. The only reader was writePixelLabels, which needs a colour for at
most 5000 pixels and only above 30x zoom, so the mirror cost a W*H*3 buffer
and a second store per pixel to serve a fraction of a percent of them.
Read the colour back from the rendered image instead. 18.1 Mpx colouring loop:
9.5 -> 6.2 ms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The qimg_buffer_ member was added to avoid reallocating the full-size image
every recolour, but GeneratePixmap still built a local QImage and the member
was never referenced. Wire it up: the buffer is reallocated only when the
image dimensions change.
The data pointer is taken once, before the parallel loop. scanLine() is
non-const and would otherwise have every worker detach the buffer at the same
time, which is a data race as soon as the buffer is shared with the pixmap.
18.1 Mpx recolour: 28.0 -> 22 ms (measured on the colouring path alone).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For stills indexing the minimum-pixels-per-spot filter is now chosen per image
instead of being fixed: the frame is indexed at min-pix 3/2/1 and the setting that
maximises indexed-spot count weighted by indexed fraction (n_indexed^2 / n_total)
is kept, then integrated once at that min-pix. The fraction factor keeps a smaller
min-pix's extra spots only when the lattice actually explains them, so strong frames
retain their real weak spots (extending resolution) while noise-flooded frames stay
strict.
The mode is selected by the presence of --min-pix-per-spot, now optional
(SpotFindingSettings::min_pix_per_spot is std::optional<int64_t>): omit it for the
adaptive per-image path, give a value to force a fixed min-pix. It applies only to
the stills indexing path -- rotation indexing builds one global lattice and keeps a
fixed min-pix, and the online receiver and the FPGA host path always carry a concrete
value, so neither changes. IndexAndRefine::ProcessImage now returns whether the frame
indexed, to drive the per-image selection.
Exposed in the jfjoch_viewer spot-finding settings (adaptive-threshold and
adaptive-min-pix checkboxes, each greying out the control it overrides); the broker
uses neither.
Validated on the full rotation regression battery (no regression) and the whole
serial-stills target battery at full image count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AdaptiveSpotFinderGPU does the per-resolution-ring reduction once on the GPU and
drives both products from it: the azimuthal-integration profile (corrected space)
and the self-calibrating adaptive spot-detection threshold (raw counts). This
replaces the separate GPU azint pass and the host-side adaptive spot finder that
runs on the GPU path today. On a ~4.5 MP detector it does both jobs in ~1 ms/frame
versus ~40 ms for the CPU adaptive finder (~42x), with an identical spot list and
azimuthal profile.
The per-ring threshold math (Poisson tail + read-floored Gaussian, operating point
from the false-pixels-per-frame knob) is factored into AdaptiveThreshold.h so the
CPU and GPU finders share one source of truth and cannot drift.
Wired opt-in via a MXAnalysisWithoutFPGA constructor flag, default on for the rugnux
offline path and the interactive viewer, off for the online receiver (so the broker
path is unchanged). When on, Analyze() skips the separate azint pass and lifts the
profile from the fused engine. The viewer gains an "Adaptive threshold" checkbox that
greys out the signal/noise and photon-count sliders (the adaptive finder uses neither).
Dedicated tests exercise both products (spot-finding parity vs the CPU finder,
azimuthal profile vs a standalone GPU azint) plus a speed benchmark. Validated
end-to-end on lysozyme serial stills: fused == CPU-adaptive index rate and merge stats.
Docs: new section 3.2 in docs/CPU_DATA_ANALYSIS.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trims three opt-in stills parameters that did not improve data quality on the
external-reference (PDB R-free) battery and only added code:
- --partiality-uncertainty: the (1-p)/p merge-sigma term was null on all four
serial-stills datasets of the battery vs their reference structures (and
neutral-to-harmful at higher coefficients); removed the flag, setting and
CorrectedSigma term.
- --stills-modulation: the detector-plane flat-field surface was net-negative
on flooded data; removed the flag, setting and MergeOnTheFly::RefineModulation
(the rotation modulation in RotationScaleMerge is unaffected).
- --min-indexed-fraction: every value other than the 0.20 default collapsed
CC1/2; removed the override flag/setter, keeping the fixed 0.20 acceptance
floor.
Default behaviour is unchanged (all three were off / at their default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both were opt-in adaptive-spot refinements that did not help. Soft per-spot
weighting was index-rate neutral across the battery (re-ranking only bites when
spots exceed the max-spot cap, which weak serial data does not reach). The
local-SNR gate was neutral on index rate and degraded merged CC1/2 on flooded
XFEL data. Drops the flags, ApplyWeights/FilterByLocalSNR, the per-spot weight
field, and the by-weight FilterSpotsByCount branch (now strongest-first only).
--adaptive-spots itself is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drops the --reject-delta-cchalf flag and MergeOnTheFly::DeltaCChalfReject.
The CLI value was parsed but never consumed (the method had no call site), so
the flag was already a no-op. Wiring it up and testing against an external
reference structure showed it is confirmation bias: on a spurious-crystal flood it
raised internal CC1/2 while CCref (correlation to the true structure) fell, and
it never improved R_meas. The merge weights are already correct; per-crystal
merge-side rejection has no genuine lever here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drops --persistence-spots and AdaptiveSpotFinderCPU::RunPersistence (the 0-D
topological-persistence detector added in 5a33b0743). It was a research variant
that never beat the hard-threshold adaptive detector on a CC1/2 basis and is a
GPU dead-end (global candidate sort + union-find), so it is not a production
path. The hard-threshold --adaptive-spots detector is unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two opt-in tools for weak serial-stills tuning; both default-off, so the
default pipeline is bit-identical (verified: a serial-stills reference run
reproduces HEAD's 7.85% indexing rate exactly).
--local-snr <sigma> (AdaptiveSpotFinderCPU::FilterByLocalSNR): after the loose
per-ring adaptive threshold builds connected-component spots, drop any spot that
does not stand this many sigmas above its OWN LOCAL background (robust median/MAD
of a square annulus), not just the azimuthal ring mean. On structured-background
(XFEL) frames the ring mean underestimates the local diffuse level in some
sectors, so the ring threshold floods; a real Bragg peak still stands many local
sigmas proud. Validated on XFEL stills to separate real peaks from flood at the
pixel level (real median local-SNR ~70 vs flood ~2.6; SNR>=5 keeps ~99.8% of
real peaks, ~14% of flood). GPU-portable (a per-spot local reduction). NOTE: on
the current serial-stills battery it is index-rate/CC1/2 neutral -- the flood that
survives as CC clusters overlaps weak-real spots, and only lattice-fit separates
those -- but it is the correct tool for genuinely floody data (ice/jet/loosened
detector) and the right substrate for the online FPGA path.
--min-indexed-fraction <f>: exposes the previously hardcoded 0.20 minimum
indexed-spot fraction (AnalyzeIndexing) as a per-run setting. Lowering it admits
weaker/sparser crystals; on flooded XFEL data the extra lattices are spurious
(pair with --min-image-cc to gate them), on clean synchrotron data there are no
marginal frames so it is a no-op -- useful as a gating-experiment primitive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-image CC-to-reference filter (--min-image-cc) was only honoured on
the rotation merge; the stills merge added every crystal unconditionally.
Extend it to stills so the flag is meaningful there too: on flooded frames
that produce many spurious lattices (large-cell serial data), the crystals
whose per-image CC to the reference falls below the limit are dropped,
keeping only the coherent ones in the merge.
Opt-in and default-off (limit 0 -> the loop passes cc_filter=false and the
merge is bit-identical to before), so no existing behaviour changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On flooded or noisy still frames (weakly-diffracting detectors, XFEL background,
ice) the full spot list derails the known-cell indexer: its many spurious peaks
compete with the true reflections for the search, so genuinely diffracting frames
fail to index.
Seed the indexer with a few spot-count subsets (30 / 80 / all) and keep the lattice
that explains the largest FRACTION of its own seed -- a lean, clean seed that a good
lattice indexes almost fully beats a flooded seed it fits only in small part. This
auto-selects a lean seed on noisy frames and the full seed where the extra spots are
real signal, with no per-dataset setting. Geometry refinement and integration still
use the full spot list (the orientation refiner filters spots by lattice match, so
the flood is ignored while high-resolution spots are kept), so resolution is
preserved. Costs at most ~3 indexer calls per frame, only on frames that do not
index on the first, lean seed.
Lifts the indexed-crystal yield on mildly-flooded synchrotron serial data with no
regression elsewhere. Stills only; the rotation indexing path is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add --soft-weight (implies --adaptive-spots): give every detected spot a
continuous quality weight in (0,1] and keep the highest-weight spots rather than
the brightest, so a deliberately loose detector self-cleans -- bright ice / salt
/ jet blobs and single-pixel noise no longer evict faint clean Bragg spots from
the max-spots cut.
The weight is a product of dimensionless gates (AdaptiveSpotFinderCPU::ApplyWeights,
computed against the per-ring background the adaptive finder already builds): a
logistic ramp in the spot's SNR and a soft size band (rises from one pixel,
plateaus, falls for oversized ice/salt/streak blobs). It carries on
DiffractionSpot -> SpotToSave and is consumed by FilterSpotsByCount, which ranks
by {non-ice, weight, intensity} when requested and by intensity otherwise, so the
classic and FPGA paths are unchanged.
Honest result: on the serial-stills battery this is index-rate-NEUTRAL. The
weighted ranking only changes the outcome when the spot count exceeds the
max-spots cap and the weight disagrees with intensity in a way that affects
indexing; the adaptive detectors already produce clean spot lists and the weak
sets sit under the cap, so re-ranking is a wash there (and a wash, not a
regression, on the one set that floods). Its intended benefit -- robustness to
ice/jet-contaminated frames and to a loosened detector -- is not exercised by
this battery; kept opt-in as the substrate for that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add --persistence-spots, a second parameter-free detector alongside --adaptive-spots.
Instead of a hard per-ring threshold it builds the noise-normalised image
z = (I - ring_mean) / sqrt(ring_sigma^2 + read^2) (same per-ring background as the
hard variant) and scores every intensity maximum by its 0-D topological persistence:
sweeping the height from high to low, each maximum is born and, when its basin meets
a taller one at a saddle, dies with persistence = birth - saddle, in sigma. A lone
noise spike merges into the background almost immediately (persistence ~1 sigma); a
real peak stands many sigma proud. Emitting maxima whose persistence clears the same
z(E) significance bar needs no photon threshold and no min-pix, and it deblends
touching peaks (each keeps its own maximum). Implemented with the same union-find
idiom as the connected-component labeller.
On serial stills this auto-adapts with no per-dataset tuning like --adaptive-spots,
finding fewer but cleaner (deblended) spots; the hard-threshold variant remains more
sensitive on the very weakest data. Both share the per-ring background and read-noise
floor. comp_of is allocated lazily so the default and hard-adaptive paths pay nothing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The offline CPU spot finder marks a pixel strong when it clears a fixed photon
count AND a local-window SNR. The fixed photon floor forces per-dataset tuning:
its sweet spot tracks the background level (weak sets want a low threshold,
strong or high-background sets a high one) and the usable window is narrow, so
users hand-tune --spot-threshold/--spot-sigma per dataset.
Add an opt-in --adaptive-spots mode (AdaptiveSpotFinderCPU) that replaces the
fixed floor with a per-resolution-ring threshold derived from each image's own
noise. Per ring it computes a peak-excluded background mean and sigma (one plain
pass + two sigma-clip passes over the assembled photon image, binned by the
azimuthal-integration ring index) and sets
thr = max( PoissonTail(mean, p), mean + z * sqrt(sigma^2 + read^2) )
with p = false_pixels_per_frame / n_pixels the single portable knob (default
100) and z = Phi^-1(1 - p). The Poisson arm is the correct significance where
the background is countable (it carries the sqrt(mean) shot noise, so a bright
low-resolution ring gets a high threshold); the read-noise-floored Gaussian arm
keeps the threshold physical where the background vanishes (empty high-resolution
rings), without which those rings flood. read is a detector-level constant, not
a per-dataset knob. Both arms are needed: Poisson alone floods near-zero
background, Gaussian alone drops the shot-noise term and under-thresholds bright
rings.
One --adaptive-spots setting then adapts across a wide range of serial datasets
with no per-dataset threshold, matching or beating hand-tuned thresholds and the
peakfinder8/xgandalf reference on both weak large-cell and strong serial data,
with equal merged R-free.
The finder runs on the CPU (offline/viewer path) and reads the host image, which
the GPU pipeline already keeps in sync, so it works in either build. The default
(non-adaptive) path and the online/FPGA path are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the frozen scalar-sigma stills partiality with a physical, refined model.
Per crystal, refine an orientation tilt (dpsi_x, dpsi_y) against the running merge
and recompute each reflection's partiality analytically from the refined geometry
(angular Ewald-proximity model, sigma(d*) = gamma_e*d*), with the per-crystal scale
G profiled out by the existing robust IRLS - no re-integration. A soft Gaussian
prior on dpsi tames weak-data overfit while staying inert on strong data. The
merge <-> refine loop iterates a few times.
This is now the stills default via ScalingSettings::stills_partiality_refine (on).
A single opt-out flag `--simple-stills` reverts to treating every reflection as a
full (p=1, single pass). Retires the experimental `--still-partiality` flag. The
viewer gains a "Partiality post-refinement (stills)" checkbox in Scaling settings.
Validated (integrate-once / --scale): CC1/2 and R_meas both improve on three
monochromatic serial-stills datasets (+2.8 / -10, +5.6 / -3.4, +2.1 / -4);
neutral on a pink-beam DMM set (already-full reflections); R-free/R-work down vs
a fixed model; competitive with CrystFEL partialator on matched frames.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The serial-stills merge (MergeOnTheFly::CorrectedSigma) weighted each
observation by 1/sigma^2 using the observation's OWN sigma. Below ~1
photon the Poisson signal part of that sigma correlates with the
observation's up/down fluctuation, so the inverse-variance mean is
biased low: an up-fluctuated observation acquires a larger sigma and is
over-downweighted. The rotation combine (RotationScaleMerge::
process_rawrun) already avoids this by rebuilding the signal variance at
the pooled estimate; the stills path did not.
Decompose each observation's variance into a background/read part (kept
per-observation) and a Poisson signal part, and rebuild the signal part
at the reflection's expected <I>. Bit-identical when an observation sits
at its reflection mean; only weak-shell weights move. Now default on, so
the stills path matches the rotation path;
--no-expected-variance-merge restores the old observed-sigma weighting.
Validated by paired refinement (phenix, 5 free-set seeds, byte-identical
free flags across arms): R-free-neutral on strong lysozyme and lower
R-free on weak serial-stills data checked against an independent
deposited model (6/6 seeds). The CC1/2 dip on strong data reflects
precision, not accuracy. Applies to both offline rugnux and the online
broker stills merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On very-low-multiplicity data (e.g. EP_cs_01-24, mult ~1.4) the merge has too
few symmetry equivalents to measure the asymptotic I/sigma: both the (a, b)
error-model fit and the per-group strong-reflection scatter collapse toward
zero, so 1/error_model_b_asymptotic either explodes to an impossibly high ISa
(tiny positive b) or is left as 0. Real macromolecular data does not exceed
ISa ~50, so clamp the reported asymptote at a generous cap (ISa 100) and treat
anything past it as unmeasured (result.isa undetermined) rather than emitting a
spurious extreme. No-op for all well-measured data (b_asy well above the cap).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GenerateSpotPlot iterated msg.spots, but SpotAnalyze called it before
assigning output.spots. In the online path the DataMessage is fresh per
frame, so the plot was built from an empty list and spot_plot_intensity
/ spot_plot_count came out all zeros. Pass the finished spots vector
explicitly instead of relying on the field being set: the live path
passes the full pre-truncation list, the HDF5 read-back path passes
message.spots.
GetResolution scaled the 5th-percentile index by spots.size() (which
includes ice-ring spots) while indexing the ice-filtered resolutions
vector, biasing the estimate and reading out of bounds on ice-heavy
frames. Index by resolutions.size() instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation.
* jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window.
* rugnux: Accept an absolute `-o` output prefix in offline processing.
* Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed).
* Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`.
* rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling.
* rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics.
* rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1.
* rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group.
* rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix.
* jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation.
* jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning.
* jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint.
* jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed).
* jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types.
* jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails.
* jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak.
* Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation.
* CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D)
* jfjoch_viewer: Some graphical improvements.
* jfjoch_viewer: Simplify und unify data analysis settings.
* jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition.
Reviewed-on: #65
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability)
Reviewed-on: #64
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
* jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname
Reviewed-on: #61
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
* jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin.
* jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins.
Reviewed-on: #60
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_viewer: Add reciprocal space viewer
* jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset
* jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress)
Reviewed-on: #57
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset.
jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling.
jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file.
jfjoch_scale: Dedicated tool for rescaling/merging existing data.
jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel.
WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being.
Reviewed-on: #56
This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144.
* **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future.
* XDS plugin: Improve performance of VDS reading.
* jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD.
* jfjoch_writer: Clean-up of HDF5 routines to better handle issues.
Reviewed-on: #55
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Improve performance of preview JPEG image generator at receiver startup (saving about 150 ms on measurement start for 16M)
Reviewed-on: #54
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
jfjoch_broker: Avoid copying gain calibration together with DiffractionExperiment
Reviewed-on: #53
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8
* Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent
Reviewed-on: #52
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times
* frontend: Fix selection of FFTW in indexing settings
Reviewed-on: #51
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection
* jfjoch_broker: Minor bug fixes for rare conditions
Reviewed-on: #50
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start`
* jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start'
* jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time
* jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received
* jfjoch_broker: Use nanoseconds internally for frame/image/readout time
* jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished)
* jfojch_broker: Improve profiling of different data analysis steps
* jfjoch_broker: Record integration reflection count
* jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds)
* jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked
* jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets
**Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time.
Reviewed-on: #49
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
jfjoch_broker: Cleanup DECTRIS start-up code to enable a shorter start time
jfjoch_broker: Allow for asynchronous start to allow overlapping detector configuration with other beamline preparations
jfjoch_broker: Goniometer axis name is converted to lowercase
jfjoch_broker: Fix bug, where wrong HTTP error codes were returned
jfjoch_broker: Improve sigma estimation during merging (K. Takaba)
---------
Co-authored-by: takaba_k <kiyofumi.takaba@psi.ch>
Reviewed-on: #48
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Better track time for each operation in the processing stack
* jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress)
* jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress)
* jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0)
Reviewed-on: #46
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress).
Reviewed-on: #45
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long
* jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata
* XDS plugin: Add generation of Jungfraujoch plugin for XDS
* CI: Add tests with XDS and DIALS (`xia2.ssx`)
Reviewed-on: #43
Updates to CI pipeline
* New docker image for Ubuntu 22.04 with CMake 3.26
* New docker image for Rocky 9 with DIALS 3.27
* New automated test to check for DIALS processing with xia2.ssx
Reviewed-on: #42
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Use httplib for HTTP server instead of Pistache
* jfjoch_broker: Drop OpenSSL support
* jfjoch_broker: Base work for multi-lattice support in the future
* Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib)
Reviewed-on: #41
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Fix bug in saving JUNGFRAU calibration (pedestal/pedestalRMS)
* jfjoch_viewer: Fix calibration (pedestal) images being open flipped
* jfjoch_process: Add space group detection (EXPERIMENTAL)
Reviewed-on: #39
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Rotation indexer has two retries if failes
* jfjoch_broker: Rotation indexer handles small number of rotation images (like test shot)
* jfjoch_broker: Integration calculates background mask based on R2 radius
* jfjoch_process: HDF5 files are not saved by default
Reviewed-on: #37
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files)
* jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction
* jfjoch_broker: Image pusher statistics are accessible via the REST interface
* jfjoch_writer: Supports TCP image socket and for these auto-forking option
Reviewed-on: #36
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Handle properly reuse of image buffer locations
* jfjoch_broker: Fix bug in counting idle slots
* jfjoch_broker: Force obtuse angle for monoclinic cells
* jfjoch_process: Change scaling refinement tolerance
Reviewed-on: #35
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Default EIGER readout time is 20 microseconds
* jfjoch_broker: Multiple improvements regarding performance
* jfjoch_broker: Image buffer allows to track frames in preparation and sending
* jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer
* jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets
* jfjoch_writer: Fixes regarding properly closing files in long data collections
* jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code
Reviewed-on: #34
This is an UNSTABLE release. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Fix bug for monoclinic space groups being wrongly refined when beta is much different from 90 deg.
Reviewed-on: #33
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis.
If things go wrong with analysis, it is better to revert to 1.0.0-rc.124.
* jfjoch_broker: Improve logic on switching on/off spot finding
* jfjoch_broker: Increase maximum spot count for FFBIDX to 65536
* jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD)
* jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis
* jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection
* jfjoch_writer: Save more information about each reflection
Reviewed-on: #32
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them,
especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123.
* jfjoch_broker: Improve refection position prediction and Bragg integration code.
* jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation.
* jfjoch_writer: Fix saving mosaicity properly in HDF5 file.
* jfjoch_viewer: Introduce high-dynamic range mode for images
* jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%)
* jfjoch_viewer: Zoom-in numbers have better readability
Reviewed-on: #31
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Use newer version of Google Ceres for (potential) CUDA 13 compatibility
* jfjoch_broker: Improve performance of generating preview images, especially for large detectors (9M-16M)
* jfjoch_viewer: Improve performance of displaying images, especially for large detectors (9M-16M)
* jfjoch_viewer: Add more color schemes for better image readability
* HDF5: Common mutex for reading and writing HDF5 if both operations were to happen in the same executable
* HDF5: suppress warning if path (upstream group) doesn't exists when checking if leaf exists
Reviewed-on: #30
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release.
* jfjoch_broker: Add thresholding to prefer shorter vectors after FFT
* jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress)
* jfjoch_viewer: Display file opening errors
* jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available
Reviewed-on: #29
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
2025-12-16 15:27:40 +01:00
1318 changed files with 153592 additions and 68826 deletions
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cmake --build build
- name:Build installer (NSIS)
shell:cmd
run:|
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cd build
cpack
- name:Upload installer to release
if:github.ref_type == 'tag'
shell:powershell
env:
TOKEN:${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run:|
# NSIS installer named jfjoch-viewer-<version>-win64-{cuda<major>|cpu}.exe (see CMakeLists.txt).
| [Ceres Solver](https://github.com/ceres-solver/ceres-solver) | (pinned) | Google Inc. and contributors | BSD-3-Clause | [ceres-solver.txt](licenses/ceres-solver.txt) |
| [wingetopt](https://github.com/alex85k/wingetopt) | `tools/wingetopt/` | Todd C. Miller; The NetBSD Foundation | ISC AND BSD-2-Clause | [wingetopt.txt](licenses/wingetopt.txt) |
## Runtime libraries and SDKs (shipped in binaries, not in the source tree)
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
/// Command will inform FPGA network card to stop pedestal or data collection at the current stage. Any frame that is currently being processed by CPU will be finished and sent to writer. Given the command is making sure to gracefully stop data acquisition and detector, it might take some time to switch back after command finished to `Idle` state. If data collection is not running, the command has no effect.
/// Set configuration for dark data collection to calculate mask
/// </summary>
/// <remarks>
/// This is only possible when operating DECTRIS detectors at the moment; it will be also available for PSI EIGER at some point. This can only be done when detector is `Idle`, `Error` or `Inactive` states.
/// Detector settings are ones that have effect on calibration, i.e., pedestal has to be collected again after changing these settings. This can only be done when detector is `Idle`, `Error` or `Inactive` states. If detector is in `Idle` state , pedestal procedure will be executed automatically - there must be no X-rays on the detector during the operation. If detector is in `Inactive` or `Error` states, new settings will be saved, but no calibration will be executed.
/// Load image for internal FPGA generator. This can only happen in Idle state of the detector. Requires binary blob with 16-bit integer numbers of size of detector in raw/converted coordinates (depending on detector settings).
/// Load image for internal FPGA generator. This can only happen in Idle state of the detector. Requires TIFF with 16-bit integer numbers of size of detector in raw/converted coordinates (depending on detector settings).
/// Jungfraujoch allows to control multiple detectors and/or region-of-interests. The command allows to choose one detector from the list (ID has to be consistent with one provided by GET response). Changing detector will set detector to `Inactive` state and will require reinitialization.
/// Should be in `Idle` state. Upload user mask of the detector - this is for example to account for beam stop shadow or misbehaving regions. If detector is conversion mode the mask can be both in raw (1024x512; stacked modules) or converted coordinates. In the latter case - module gaps are ignored and don't need to be assigned value. Mask is expected as binary array (4-byte; unsigned). 0 - good pixel, other value - masked User mask is stored in NXmx pixel mask (bit 8), as well as used in spot finding and azimuthal integration.
/// Should be in `Idle` state. Upload user mask of the detector - this is for example to account for beam stop shadow or misbehaving regions. If detector is conversion mode the mask can be both in raw (1024x512; stacked modules) or converted coordinates. In the latter case - module gaps are ignored and don't need to be assigned value. Mask is expected as TIFF (4-byte; unsigned). 0 - good pixel, other value - masked User mask is stored in NXmx pixel mask (bit 8), as well as used in spot finding and azimuthal integration. User mask is not automatically applied - i.e. pixels with user mask will have a valid pixel value in the images.
/// Jungfraujoch can generate metadata message stream on ZeroMQ PUB socket. This stream covers all images. Here settings of the socket can be adjusted. While the data structure contains also socket_address, this cannot be changed via HTTP and is ignore in PUT request.
/// Jungfraujoch can generate preview message stream on ZeroMQ SUB socket. Here settings of the socket can be adjusted. While the data structure contains also socket_address, this cannot be changed via HTTP and is ignore in PUT request. Options set with this PUT request have no effect on HTTP based preview.
/// Should be in `Idle` or `Error` state. Command deactivates data acquisition and turns off detector high voltage and ASIC. Should be used always before turning off power from the detector.
/// Contains full image data and metadata. The image must come from the latest data collection.
/// </remarks>
/// <param name="id">Image ID in the image buffer. Special values: -1 - last image in the buffer, -2: last indexed image in the buffer (optional, default to -1L)</param>
/// Get preview image in JPEG format using custom settings
/// </summary>
/// <remarks>
///
/// </remarks>
/// <param name="id">Image ID in the image buffer. Special values: -1 - last image in the buffer, -2: last indexed image in the buffer (optional, default to -1L)</param>
/// <param name="showUserMask">Show user mask (optional, default to false)</param>
/// <param name="showRoi">Show ROI areas on the image (optional, default to false)</param>
/// <param name="showSpots">Show spot finding results on the image (optional, default to true)</param>
/// <param name="showBeamCenter">Show beam center on the image (optional, default to true)</param>
/// <param name="saturation">Saturation value to set contrast in the preview image; if not provided, then autocontrast procedure is used (optional, default to 0.0f)</param>
/// <param name="jpegQuality">Quality of JPEG image (100 - highest; 0 - lowest) (optional, default to 100L)</param>
/// <param name="showResRing">Show resolution ring, provided in Angstrom (optional, default to 0.1f)</param>
/// <param name="id">Image ID in the image buffer. Special values: -1 - last image in the buffer, -2: last indexed image in the buffer (optional, default to -1L)</param>
/// Can be run at any stage of Jungfraujoch operation, including during data collection. The status of the image buffer is volatile during data collection - if data collection goes for more images than available buffer slots, then image might be replaced in the buffer between calling /images and /image.cbor.
/// Should be used in two cases: - Detector is in `Inactive` state - Detector is in `Error` state X-ray shutter must be closed. This operation will reconfigure network interface of the detector. During operation of the detector it is recommended to use the `POST /pedestal` operation instead. If storage cells are used, the execution time might be few minutes. This is async function - one needs to use `POST /wait_till_done` to ensure operation is done.
/// Updates calibration of the JUNGFRAU detector. Must be in `Idle` state. X-ray shutter must be closed. Recommended to run once per hour for long integration times (> 100 us). This is async function - one needs to use `POST /wait_till_done` to ensure operation is done.
/// Generate 1D plot from Jungfraujoch and send in raw binary format. Data are provided as (32-bit) float binary array. This format doesn't transmit information about X-axis, only values, so it is of limited use for azimuthal integration.
/// </summary>
/// <remarks>
///
/// </remarks>
/// <param name="type">Type of requested plot</param>
/// <param name="roi">Name of ROI for which plot is requested (optional, default to "")</param>
/// <param name="type">Type of requested plot</param>
/// <param name="binning">Binning of frames for the plot (0 = default binning) (optional, default to 1)</param>
/// <param name="compression">Enable DEFLATE compression of output data. (optional, default to false)</param>
/// <param name="fill">Fill value for elements that were missed during data collection (optional, default to 0.0f)</param>
/// <param name="experimentalCoord">If measurement has goniometer axis defined, plot X-axis will represent rotation angle If measurement has grid scan defined, plot X-axis and Y-axis will represent grid position, Z will be used as the final value For still measurement the number is ignored (optional, default to false)</param>
/// <param name="azintUnit">Unit used for azim int. (optional, default to "Q_recipA")</param>
/// Start data acquisition. Detector must be in `Idle` state. Doesn't run calibration procedure. When the function returns, detector is ready to accept soft/TTL triggers.
/// Block execution of external script till initialization, data collection or pedestal is finished. Running this command does not affect (cancel) running data collection, it is only to ensure synchronous execution of other software. To not block web server for a indefinite period of time, the procedure is provided with a timeout. Extending timeout is possible, but requires to ensure safety that client will not close the connection and retry the connection.
/// </remarks>
/// <param name="timeout">Timeout in seconds (0 == immediate response) (optional, default to 60)</param>
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
@@ -68,10 +68,12 @@ public:
boolisSolidAngleCorr()const;
voidsetSolidAngleCorr(boolconstvalue);
/// <summary>
///
/// Upper q limit of the azimuthal integration [1/Angstrom]. Optional: if omitted, the integration (and the adaptive spot detection that shares these q bins) extends to the highest q the detector reaches.
/// </summary>
floatgetHighQRecipA()const;
voidsetHighQRecipA(floatconstvalue);
boolhighQRecipAIsSet()const;
voidunsetHigh_q_recipA();
/// <summary>
///
/// </summary>
@@ -89,6 +91,13 @@ public:
voidsetAzimuthalBins(int64_tconstvalue);
boolazimuthalBinsIsSet()const;
voidunsetAzimuthal_bins();
/// <summary>
/// Force CPU processing of azimuthal integration in the FPGA data acquisition workflow. This allows to extend number of azimuthal integration bins, as well as to calculate standard deviation of the azimuthal integration results.
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* https://openapi-generator.tech
* Do not edit the class manually.
*/
/*
* Bragg_integration_settings.h
*
* Settings for Bragg spot integration
*/
#ifndef Bragg_integration_settings_H_
#define Bragg_integration_settings_H_
#include"Integration_model.h"
#include<nlohmann/json.hpp>
namespaceorg::openapitools::server::model
{
/// <summary>
/// Settings for Bragg spot integration
/// </summary>
classBragg_integration_settings
{
public:
Bragg_integration_settings();
virtual~Bragg_integration_settings()=default;
/// <summary>
/// Validate the current data in the model. Throws a ValidationException on failure.
/// </summary>
voidvalidate()const;
/// <summary>
/// Validate the current data in the model. Returns false on error and writes an error
/// message into the given stringstream.
/// </summary>
boolvalidate(std::stringstream&msg)const;
/// <summary>
/// Helper overload for validate. Used when one model stores another model and calls it's validate.
/// How far the Bragg predictor walks the lattice: reflections with |h|,|k|,|l| above this are never predicted. An axis is truncated once a/d_min exceeds it, so a long axis - or a short one taken to high resolution - loses its outermost reflections. The cost grows as the cube (2n+1)^3 of candidates per image, which is why online keeps a fixed, predictable value instead of taking it from whichever crystal is mounted. Omitting the field selects the default above; it is never interpreted as \"choose per crystal\". The offline tools (rugnux, viewer) do derive it from the refined cell when it is left unset there, but that is their own default and is not reachable through this API.
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
@@ -21,6 +21,7 @@
#include"Grid_scan.h"
#include<nlohmann/json.hpp>
#include"Dataset_settings_smargon.h"
#include"Rotation_axis.h"
#include"Unit_cell.h"
#include<string>
@@ -168,7 +169,7 @@ public:
boolgridScanIsSet()const;
voidunsetGrid_scan();
/// <summary>
/// Header appendix, added as user_data/user to start ZeroMQ message (can be any valid JSON) In general, it is not saved in HDF5 file. However, if values are placed in \"hdf5\" object, `jfjoch_writer` will write them in /entry/data of the HDF5 file. This applies solely to string and number (double floating-point). No arrays/sub-objects is allowed. For example {\"hdf5\": {\"val1\":1, \"val2\":\"xyz\"}}, will write /entry/user/val1 and /entry/user/val2.
/// Header appendix, added as user_data/user to start ZeroMQ message (can be any valid JSON) In general, it is not saved in HDF5 file. However, if values are placed in \"hdf5\" object, `jfjoch_writer` will write them in /entry/user of the HDF5 file. This applies solely to string and number (double floating-point). No arrays/sub-objects is allowed. For example {\"hdf5\": {\"val1\":1, \"val2\":\"xyz\"}}, will write /entry/user/val1 and /entry/user/val2.
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
/// Geometry of Smargon goniometer at SLS 2.0 / MX beamlines. Assuming that Smargon is used as static positioner and not moving during the scan, allowing to reconstruct geometry.
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
@@ -111,26 +111,26 @@ public:
boolsensorThicknessUmIsSet()const;
voidunsetSensor_thickness_um();
/// <summary>
/// Minimum difference between frame time and count time in microseconds Defaults are 3 us for EIGER and 20 us for JUNGFRAU
/// Minimum difference between frame time and count time in microseconds Defaults are 3'000 ns for EIGER and 20'000 ns for JUNGFRAU
/// </summary>
int64_tgetReadoutTimeUs()const;
voidsetReadoutTimeUs(int64_tconstvalue);
boolreadoutTimeUsIsSet()const;
voidunsetReadout_time_us();
int64_tgetReadoutTimeNs()const;
voidsetReadoutTimeNs(int64_tconstvalue);
boolreadoutTimeNsIsSet()const;
voidunsetReadout_time_ns();
/// <summary>
/// Minimum count time available for the detector.
/// </summary>
int64_tgetMinimumCountTimeUs()const;
voidsetMinimumCountTimeUs(int64_tconstvalue);
boolminimumCountTimeUsIsSet()const;
voidunsetMinimum_count_time_us();
int64_tgetMinCountTimeNs()const;
voidsetMinCountTimeNs(int64_tconstvalue);
boolminCountTimeNsIsSet()const;
voidunsetMin_count_time_ns();
/// <summary>
/// Minimum frame time available for the detector.
/// </summary>
int64_tgetMinimumFrameTimeUs()const;
voidsetMinimumFrameTimeUs(int64_tconstvalue);
boolminimumFrameTimeUsIsSet()const;
voidunsetMinimum_frame_time_us();
int64_tgetMinFrameTimeNs()const;
voidsetMinFrameTimeNs(int64_tconstvalue);
boolminFrameTimeNsIsSet()const;
voidunsetMin_frame_time_ns();
/// <summary>
/// Can be empty for all detectors - default calibration used. For JUNGFRAU: list of gain files, one entry per module. For EIGER: one directory (with detector settings) or list of trim bit files, one entry per half-module.
/// </summary>
@@ -139,6 +139,13 @@ public:
boolcalibrationFileIsSet()const;
voidunsetCalibration_file();
/// <summary>
/// List of energies at which trimming calibration is provided. Only needed for PSI EIGER and compulsory in this case
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
* API to control Jungfraujoch developed by the Paul Scherrer Institute (Switzerland). Jungfraujoch is a data acquisition and analysis system for pixel array detectors, primarly PSI JUNGFRAU. Jungfraujoch uses FPGA boards to acquire data at high data rates. # License Clarification While this API definition is licensed under GPL-3.0, **the GPL copyleft provisions do not apply** when this file is used solely to generate OpenAPI clients or when implementing applications that interact with the API. Generated client code and applications using this API definition are not subject to the GPL license requirements and may be distributed under terms of your choosing. This exception is similar in spirit to the Linux Kernel's approach to userspace API headers and the GCC Runtime Library Exception. The Linux Kernel developers have explicitly stated that user programs that merely use the kernel interfaces (syscalls, ioctl definitions, etc.) are not derivative works of the kernel and are not subject to the terms of the GPL. This exception is intended to allow wider use of this API specification without imposing GPL requirements on applications that merely interact with the API, regardless of whether they communicate through network calls or other mechanisms.
*
* The version of the OpenAPI document: 1.0.0-rc.121
* The version of the OpenAPI document: 1.0.0-rc.161
* Contact: filip.leonarski@psi.ch
*
* NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
/// None - no master file written NXmxLegacy - legacy format with soft links to data files in the master file; necessary for DECTRIS Albula 4.0 and DECTRIS Neggia NXmxVDS - newer format with virtual dataset linking data files in the master file, also includes better metadata handling
/// NoFileWritten - no files are written at all NXmxOnlyData - only data files are written, no master file NXmxLegacy - legacy format with soft links to data files in the master file; necessary for DECTRIS Albula 4.0 and DECTRIS Neggia NXmxVDS - newer format with virtual dataset linking data files in the master file, also includes better metadata handling NXmxIntegrated - single HDF5 per dataset CBF - DEPRECATED, no longer supported; kept for back compatibility only. Requests using this value are rejected. Only HDF5 formats are written. TIFF - DEPRECATED, no longer supported; kept for back compatibility only. Requests using this value are rejected. Only HDF5 formats are written.
/// Selection of an post-indexing detector geometry refinement algorithm used by Jungfraujoch. This option is using non-linear least squares optimization to find unit cell and beam center.
/// Selection of an post-indexing least-square diffraction geometry refinement algorithm used by Jungfraujoch. BeamCenter - This option is refining both beam center and lattice (restricted to a chosen/detected Bravais lattice). OrientationOnly - This option is refining only orientation of the lattice. Flex - Tries all per-image refinements and keeps whichever indexes the most spots, letting the pipeline decide (the rugnux flex mode).
/// Number of slots in the image buffer that are currently in preparation for sending.
/// </summary>
int64_tgetInPreparationSlots()const;
voidsetInPreparationSlots(int64_tconstvalue);
/// <summary>
/// Number of slots in the image buffer that are currently sending/writing data.
/// </summary>
int64_tgetInSendingSlots()const;
voidsetInSendingSlots(int64_tconstvalue);
/// <summary>
/// Counter of changes in the image buffer - either new start message or new image added. For optimization one can only load new images/datasets from the HTTP if this value changes. Counter is optional as it was not implemented in older versions to avoid breaking change
/// List of addresses of image pushers. For HDF5 socket - no addresses will be returned. For ZeroMQ - list of addresses for each socket will be provided. For TCP/IP - single address to connect all writers will be provided.
/// </summary>
std::vector<std::string>getAddr()const;
voidsetAddr(std::vector<std::string>const&value);
/// <summary>
/// Number of connected writers For ZeroMQ image socket: number is constant For TCP/IP image socket: number is updated live during operation
/// </summary>
int64_tgetConnectedWriters()const;
voidsetConnectedWriters(int64_tconstvalue);
/// <summary>
/// Number of images written to the image socket. This number is updated live during operation for TCP/IP image socket and direct HDF5 writer. It is updated at the end of experiment for ZeroMQ image socket.
/// </summary>
int64_tgetImagesWritten()const;
voidsetImagesWritten(int64_tconstvalue);
boolimagesWrittenIsSet()const;
voidunsetImages_written();
/// <summary>
/// Number of images that could not be written to the image socket. This number is updated live during operation for TCP/IP image socket. No other socket use it.
/// </summary>
int64_tgetImagesWriteError()const;
voidsetImagesWriteError(int64_tconstvalue);
boolimagesWriteErrorIsSet()const;
voidunsetImages_write_error();
/// <summary>
/// Utilization of internal writer FIFO. This number is updated live during operation for TCP/IP image socket. No other socket use it.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
