ffbf38d2abd8ec6d93bc3c016129457b5b25618a
1200
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  leonarski_fandClaude Opus 5
|
ffbf38d2ab |
Rotation scaling: guard the per-frame scales whatever else is switched on
The protection against a per-frame scale collapsing toward zero lived inside ComputeSmoothGWindow, so it only existed when smooth-G did: --smooth-g=0, a dataset whose oscillation width is unknown, and any caller that never sets a smoothing range - the viewer among them - merged with no guard at all. A collapsed G multiplies that frame's intensities by 1/G and its sigmas by the same factor, so nothing downstream can see it; the merge's n-sigma cut scales with the number that is wrong. Pull it out into ReplaceCollapsedScales, called unconditionally right after the partial scaling loop, and let the smooth-G window assume what it now guarantees instead of computing its own median and floor. The fulls guard built its median from every frame including those never fitted - those sit at the combine's corr = 1, so a run with many unfitted frames dragged the median toward 1 and the floor with it. It also reported the absolute amplification where the message says "below the run median". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
a770f0857b |
CHANGELOG: record the rc.161 fixes
Also corrects the reader bit-depth entry: taking the depth from the file was the wrong fix for the 32-bit EIGER2 case and broke 8-bit files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
ae2b11088d |
docs: drop generated pages for schemas the API no longer has
Eight pages under docs/python_client/docs describe schemas that appear nowhere in jfjoch_api.yaml and are linked from no index - left behind because the regeneration step never cleared python-client/, which these are copied from. That is fixed in update_version.sh; this removes what accumulated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
a95aca382c |
Remove dead code left behind by recent changes
None of this has a reader:
- ScalingSettings::scaling_regularize and its setter/getter
- ScaleOnTheFlyResult::succesful (never set) and ::time_s (set, never read),
with the timing that only fed the latter
- JFJochImage::last_fit_viewport_ (written twice, read nowhere) and the
comment claiming the retry uses it - the retry keys off initial_fit_done_
- JFJochDiffractionImage::ice_ring_width_Q_recipA, and a QtConcurrent include
in a file that uses none
- an unused gemmi::Op accumulator in the spindle-angle helper
- <random> in Merge.{h,cpp}, from before the half-set split became a hash
- an orphaned comment describing the Ceres B-factor residual deleted in
|
||
|
leonarski_fandClaude Opus 5
|
2ccd1a7e4b |
update_version.sh: clear the python client before regenerating it
The script now removes the generated C++ model, the frontend client and the published python docs before regenerating, but not python-client/ itself - and docs/python_client/docs is filled by copying that directory. So a schema dropped from the API kept its generated model in the PyPI package and its .md page in the published docs, linked from no index. Eight such pages are in the tree today, JfjochSettingsSsl among them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
3b9e1424ea |
docs: RUGNUX.md on adaptive detection and --search-min-zeta
Adaptive spot detection became the default for rotation data as well in 6f4917dce; RUGNUX.md still said rotation kept the fixed-threshold finder, which is also the opposite of what the usage message and CPU_DATA_ANALYSIS say. --search-min-zeta had no entry in the option tables at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
30252722e4 |
Viewer: the magnifier re-opens on the current cursor position
centerAt returns early while the window is hidden, and nothing replays the last position when it comes back, so re-opening the magnifier showed whatever region the cursor was over when it was closed - with current pixels, which makes it look like a live view of the wrong place. Remember the position while hidden and apply it on show. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
379feac2e9 |
Viewer: lay out pixel labels only for the area being repainted
drawPixelLabels took its range from the whole viewport rather than from the exposed rect it was given, so a 200x40 px hover repaint still walked up to 5000 cells doing mapFromScene + QImage::pixel + drawText for each, only to have the result clipped away. With hover feedback now rate-limited to 15 Hz that ran ~75 times a second at high zoom, against once per overlay rebuild before the rendering rework. Intersect with the exposed rect; same in the magnifier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
dc5721ad8a |
Viewer: repaint the resolution readout when the view scrolls
The "d = ... A" readout moved from a scene item flagged ItemIgnoresTransformations to a fixed viewport position painted in drawForeground, which is what made a hover update dirty a small rect instead of the whole viewport. But QGraphicsView pans by blitting the viewport: the painted text is shifted along with the image and left there, and the pending update for its old position is translated away too, so dragging the image smears ghost copies of the readout across the corner. Dirty the old and the new rect when the view scrolls - still a couple of hundred pixels, not the viewport. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
f9c8ab4552 |
rugnux: a run that indexes nothing still writes a process file
With merging on, the _process.h5 is skipped because the merged reflections are the wanted output and that file is large (113 MB for 200 images here). But if nothing indexes there are no merged reflections either, so the run finished successfully having written no file at all - the one case where the user most needs something to look at. Write it in that case. The per-image messages have already gone past unwritten, so this carries the dataset metadata, the mask, the azimuthal profile and the summary scalars rather than the full per-image tables - and it is small for the same reason it is needed (124 kB on a zero-index run). A run that does index is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
504393d10e |
rugnux: report bad input instead of aborting on it
main had no enclosing try/catch, but plenty of ordinary input reaches a setter that throws: --polarization 2, --detector-distance 0, -q 0, --azim-max-q 20, --scale combined with a reference MTZ, and every failure inside the pipeline itself. All of them ended as "terminate called after throwing an instance of 'JFJochException'" and exit 134, with the message nowhere to be seen. Move the body into RunRugnux and let main report what was thrown, exit 1. Two options also still bypassed the numeric parser that exists to prevent this: --scaling-high-resolution used atof, which turns a typo into 0 and then throws from the setter, and --integration-radius used std::stof, which throws on non-numeric input. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
f90969ea21 |
Stills partiality: do not adopt a tilt from a failed solve
The Ceres summary was discarded, so a solve that diverged or aborted left its last iterate in psi and that tilt was written onto the partiality of every reflection of the crystal. Restore the tilt the crystal came in with and stop refining it; the scale fit alone is still a usable model, which is what the other three early returns in this function fall back to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
3e8a994d2e |
Stills scaling: leave an image unscaled when its scale collapses
SolveScaleIRLS returns whatever it converged to and both writers accept any G > 0, so a fit that collapsed to ~1e-3 multiplies that image's intensities by a thousand. Nothing downstream notices, because the sigmas are multiplied by the same factor and the merge's n-sigma outlier test is therefore blind to it - only a total collapse self-heals, by overflowing corr to inf. The rotation path refuses a per-frame scale this far below its neighbours; the stills path had no guard. Judge each image against the median of the images that did scale, and put a collapsed one back to G = 1 - the same state as an image with too few reflections to fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
af97ad61e3 |
Rotation scaling: the search-only filters must not outlive their pass
Two faults in the same block, both of which let a search pass corrupt the production merge that follows it. The device's corr was only copied back to the host for the diagnostic dump, but the |zeta| filter runs on the host and then uploaded the whole host array - so on a CUDA build it wrote the values ingested BEFORE scaling over the scaled and smoothed corr the device had just computed. With the rotation default --search-min-zeta 0.85 that means the space-group search was deciding the symmetry from an unscaled merge. Copy corr back first, and upload once after both filters instead. Zeroing corr also has no owner: it is how an observation leaves the merge, but the only thing that ever rewrites it is the scaling loop, which skips frames it cannot fit. A frame left with too few well-measured reflections therefore kept its dropped observations at zero for the rest of the object's life - and the final production merge re-uses the same object without re-ingesting. Snapshot corr before the filters and restore it at the start of the next pass, so each pass decides for itself and the final merge keeps everything, as documented. The frame rejection (--min-image-cc) is now applied on the host for both paths; its separate device path did nothing whenever the CPU combine was in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
aed1a7a6d6 |
Adaptive spot finder: pin the threshold to the image, and the GPU to itself
The existing cases plant blobs at 200 on a background of 8..12, so any threshold between 12 and 200 passes them - replacing RingThreshold with a constant leaves them all green. Two cases that do not: - the CPU threshold has to track the background: a frame and the same frame scaled ten times must give the same spots, with a pixel a few sigma above the background staying unfound in both. A constant threshold, or one that drops the sigma term, fails one scale or the other. - the GPU engine has to agree with itself across runs, which is what the ring sums being order-independent buys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
04450eb618 |
Adaptive spot finder: sum the rings across blocks in double
The ring sigma is the cancelling difference sum2/n - m^2, and both sums were float accumulated by atomics whose order is arbitrary. Two costs: the cancellation left only ~4 digits in the variance, and the ordering moved the resulting threshold by ~0.05 counts between runs - enough to flip a pixel sitting on the hard "value >= threshold" test, and with it a connected component's size. So the GPU engine did not reproduce the CPU one and did not reproduce itself. Only the accumulators that span blocks are widened. The per-block staging stays float, because a block contributes a few dozen similar-magnitude pixels to a ring and there is nothing to lose there - that also keeps the shared-memory footprint of the hot loop, and hence its occupancy, exactly as it was: measured on a 4.5 MP frame, 0.960 vs 0.966 ms/frame (40.9x over the CPU path, unchanged). finalize_rings now does the cancellation in double and rounds to float last, which is what AdaptiveSpotFinderCPU::AccumulateRings does. The device properties are also read from the current device rather than device 0; callers round-robin engines across GPUs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
8dc7254624 |
reader: the stored images are int32, so say so
|
||
|
leonarski_fandClaude Opus 5
|
0076b76566 |
Bragg prediction: launch the whole hkl range on the GPU
The host sized the grid from 2*max_hkl while the kernel guards against 2*max_hkl+1, so whenever the rounded-up grid landed exactly on 2*max_hkl threads (max_hkl a multiple of 4, with the 8x8x8 block) the h = +max_hkl plane was never launched. The CPU loop runs -max_hkl..+max_hkl inclusive, so the GPU predicted a strict subset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
9b55e44acd |
Azimuthal integration: clamp a negative variance before the square root
calc_std uses the cancellation-prone (sum2 - sum^2/n) form on float accumulators
summed over millions of pixels, so a flat ring - true variance near zero - comes
out negative as often as positive and GetStd() returns NaN. Both adaptive
spot-finder ring accumulators already floor this at zero; this one did not, and
|
||
|
leonarski_fandClaude Opus 5
|
2ef8841983 |
Azimuthal integration: keep the low Q limit below the maximum
Making the high limit optional removed the implicit upper bound on the low one (it used to follow from high <= maxQ and high > low), so --azim-min-q 50 with no maximum is accepted and ResolveHighQ then calls std::clamp with its lower bound above its upper bound, which is undefined. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
009555bc49 |
Spot finding: a zero high-resolution limit means no limit here too
Every other reader of spot_finding.high_resolution_limit spells "unset" as value_or(0) and compares, so 0 and nullopt are interchangeable - except in SpotAnalyze, which passed the 0 straight to ResolutionShells and threw "Resolution must be above zero" on every image. Reachable over the REST API, where 0 is the natural way to say "no limit" and the settings check lets it through; the rugnux CLI already maps 0 to unset before this point. While here, check that a limit that IS set is finite regardless of its sign - NaN fails the > 0 test and was skipping validation entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
e123755c7d |
rugnux: carry the point-group promotion into the second pass
The two-pass rotation run reinstates pass-1's space group for pass 2's merge, so pass 2 skips the search block entirely - and with it the flag that records that the Laue class was CHOSEN by the search rather than given. The canonical output therefore printed the plain "no twinning: the Laue class is holohedral, so no merohedral twin law exists", which is exactly the circular conclusion the flag was added to replace; only the throwaway _01 output carried the caveat. The text is written per pass, inside RunPipeline, so the flag has to travel with prepass_merge_sg_ rather than being patched onto the returned result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
64c8117a96 |
Space-group search: absences always come from the all-observation merge
The two-arm search is meant to use the Lorentz-filtered merge for the point group only - systematic absences live in the weak reflections a |zeta| cut removes, and reading them off the filtered arm is what cost four crystals their screw axes. That is what the code comment and CPU_DATA_ANALYSIS both say, but the filtered-arm-wins branch kept its whole result, screws and centering included. Let a search be pinned to a point group decided elsewhere (fixed_point_group) and re-run Stage B on the all-observation merge when the filtered arm rescues the point group. The point group is passed as its symmorphic representative, not by name: gemmi calls both P321 and P312 "32". Reporting that representative also lets the ambiguity check see two arms that disagree about which 2-folds are real - by name they looked identical - and the advice it prints now names a space group -S can actually be given. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
1355d6b2aa |
Space-group search: a screw threshold needs a row with controls behind it
The row-relative cut scales the "too strong to be absent" threshold by the axial row's own median E^2, floored at the plain value - so it can only raise the bar, and a row whose control class holds a single strong reflection sets it from that one reflection. That direction invents screws: a genuine 4_2 whose 00l happen to be observed only at l=4n reads its l=4n+2 reflections as absent and ranks 4_1/4_3 above the truth. Require three controls before the row may set the scale; below that the row keeps the plain cut. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
72cd91ccc3 |
Space-group search: judge a promotion against the worst of its tied parents
A candidate can have several confirmed subgroups of the same order - 422 has both 4 and 222 - and on a twinned crystal the rival is not a harmless alternative: a P4 crystal twinned by 2[100] has 222 confirmed too, and 222 CONTAINS the twin laws, so its own merge b is already ballooned. The H test already answers to every tied parent; the systematic-b veto and rescue took whichever one the enumeration happened to list first (222 before 4, by space- group number), which disabled the veto on exactly the case it exists for. Take the smallest parent b, which is the conservative direction for both tests. The refusal message also quoted the raw parent b rather than the floored value the veto actually compared against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
609650b061 |
Rotation merging: keep the systematic sigma floor when ISa is unmeasurable
The cap that refuses to report an impossible ISa was zeroing the asymptotic b itself, and that same value is the floor passed to SigmaWithSystematicFloor - where zero means "no floor". So on the degenerate low-multiplicity fit the guard is written for, instead of capping merged I/sigma at 100 it removed the cap entirely. Report the asymptote as unmeasured, keep the fitted value for the floor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
386f10ad07 |
Merging: mirror the negative-intensity Poisson guard on the GPU
|
||
|
leonarski_fandClaude Opus 5
|
a3feb1271c |
docs: CPU_DATA_ANALYSIS describes the algorithms, not their history
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m40s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m28s
Build Packages / build:rpm (rocky9) (push) Successful in 13m31s
Build Packages / DIALS test (push) Successful in 14m18s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m57s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m47s
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m53s
Build Packages / build:viewer-tgz:cuda (push) Successful in 9m23s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m54s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m49s
Build Packages / Unit tests (push) Successful in 1h1m51s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m9s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m16s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m17s
Build Packages / build:rpm (rocky8) (push) Successful in 11m23s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m17s
Build Packages / XDS test (durin plugin) (push) Successful in 7m36s
Build Packages / Generate python client (push) Successful in 26s
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
The document had accumulated development narrative: what was tried and
rejected, which datasets a change rescued or cost, measured percentages
from test batteries. That belongs in commit messages and reports, not in
a reference description of the pipeline - it dates quickly, and a reader
looking up what an algorithm does has to sort it out from how it came to
be.
Removed throughout, keeping the algorithmic content and the design
reasoning that explains a choice on principle:
* 3.2 the whole paragraph justifying the rotation spot-finding limit
from battery measurements, and the CPU-vs-GPU per-frame timings;
* 3.3 "a significance/z-score was considered but is uninformative";
* 7.4 / 7.5 the comparisons to a robust loss and to joint refinement
as approaches that had failed;
* 9.2 the R_meas / CC1/2 outcomes attributed to the trimmed-mean
background;
* 9.3 "per-detector-region and crystal-anisotropy profiles were
evaluated and add nothing";
* 10.2 the stills tilt "succeeds where a freely-fitted width
collapses";
* 10.5 the CC_anom argument, trimmed to why the statistic behaves as
it does;
* 10.6 the survey of per-frame correlation medians across datasets;
* 13 the space-group bullet, restructured into the three gates it
actually applies, dropping the dataset anecdotes;
* 14.2 the free-form per-shell rescale, stated as a design choice
rather than an experiment.
Section 13's space-group text was one 20-line paragraph; it is now a
numbered list of the three tests, which is what the code does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
leonarski_fandClaude Opus 5
|
0f1851cf82 |
rugnux: drop the 1.5 A spot-finding limit on rotation data
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
Build Packages / build:viewer-tgz:cpu (push) Successful in 8m16s
Build Packages / build:viewer-tgz:cuda (push) Successful in 9m21s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m55s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m9s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m19s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m26s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m31s
Build Packages / build:rpm (rocky8) (push) Successful in 11m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m17s
Build Packages / XDS test (durin plugin) (push) Successful in 7m37s
Build Packages / Generate python client (push) Successful in 29s
Build Packages / Build documentation (push) Successful in 1m9s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m33s
Build Packages / build:rpm (rocky9) (push) Successful in 13m37s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m40s
Build Packages / DIALS test (push) Successful in 14m11s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m59s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m39s
Build Packages / Unit tests (push) Successful in 1h1m40s
Rotation kept a 1.5 A high-resolution limit for spot finding on the
strength of one indexing-rate measurement (100.0 -> 96.8% on a strong
set). Measured properly, over the whole 33-crystal rotation battery, it
does not earn its place:
* no space-group decision changes - the same 30/33 agree with XDS, and
the three that differ are the known pre-existing cases, unchanged;
* 29 of 33 crystals are identical to the digit - same indexing rate,
R_meas, CC1/2, ISa. The limit was doing nothing on the large
majority;
* where it does bite, the limit is the worse setting. The one crystal
that loses appreciable indexing rate without it (99.50 -> 94.22%)
comes back with lower R_meas (29.4 -> 28.1), higher high-resolution
CC1/2 (27.9 -> 29.1) and higher ISa (5.77 -> 6.17). Another loses
0.4% of frames and gains 2.8 points of CC1/2_hi. Fewer frames
indexed, better data from them;
* runtime is unchanged (16m46s vs 17m32s over the battery).
So the indexing-rate cost is real but does not carry through to the
merged data, which is what the limit was protecting. Unset now means "as
far as the detector reaches" for rotation as well as stills;
--spot-high-resolution still sets a limit for weak, high-background data
where the extra high-resolution spots are genuinely noise.
This also removes the flag that distinguished "the user asked for no
limit" from "the user said nothing" - with no rotation default left,
both mean the same thing. While rewriting the comment block, corrects
its neighbouring claim that rotation keeps the fixed-threshold finder;
adaptive detection has been the default for both workflows since
|
||
|
leonarski_fandClaude Opus 5
|
323c76abde |
docs: correct CPU_DATA_ANALYSIS against the code it describes
Ten places where the document and the implementation had drifted apart.
Each was checked against the source before rewriting:
* 7.5 rotation post-refinement: it is TWO separate cross-validated
steps (cell+axis from the angles, then distance+beam from the
positions with the cell fixed), not one joint fit against the merged
fulls; the held-out split is an hkl hash, not a frame split; the
bounds are +-5% on distance and +-15 px on the beam, not "under
~1%"; and only the distance and beam centre reach pass 2, which
re-indexes de novo.
* 9.2 the trimmed-mean background: it is computed in the shared
background pass, so it DOES apply to --integrator boxsum. Only the
broadband sigma-clip is excluded. The section previously said both,
contradicting itself two paragraphs apart.
* 9.3 per-reflection profile rebuild, sub-pixel centring and radial
elongation are gaussian-only; the empirical profile keeps the fixed
per-shell grid and is accumulated on rounded predicted positions,
not centroids.
* 10.5 the asymptotic ISa and the b_ISa sigma floor are rotation-only;
stills report 1/b and floor with the whole-range b.
* 8.4 centering absences are applied only when the user fixes the
space group - de novo, prediction runs in P so the search can
confirm the centering from the intensities.
* 10.6 per-batch relative-B cross-validates on ASU-group parity, not
the frame parity the other surfaces use.
* 13 the resolution cutoff sits one reported-shell width PAST the
CC1/2 = 0.30 crossing, so data below 0.30 are kept.
* 7.1 the orientation-only prior penalises all three components of the
angle-axis vector.
* 10.1 rlp is the RECIPROCAL Lorentz factor, L = 1/rlp.
* 14.3 the model scaling is fitted over work and free reflections
alike, so R-free is free of refinement, not of the scaling fit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
leonarski_fandClaude Opus 5
|
648e8c62c8 |
CHANGELOG: record the two settings fields that are no longer required
Making azim_int_settings.high_q_recipA and spot_finding_settings.high_resolution_limit optional changes the wire format: when unset they are omitted rather than sent with a placeholder, and a client generated from an older spec does j.at() on them. The azimuthal-integration limit now defaults to unset, so a stock broker omits it out of the box - the break needs no operator action to hit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
c10d7ced17 |
rugnux: build the first-pass analysis engine once, not per image
The first-pass spot cache constructed an MXAnalysisWithoutFPGA and an AzimuthalIntegrationProfile inside the per-image lambda, so every cache miss allocated a CUDA stream, the preprocessing buffer, the spot finder, the azimuthal integrator and the Bragg engine, used them for one frame, and freed them again - hundreds of times, serially, on the --redo-rotation-spots path. Both worker loops already hoist the same object out of their loop; only this path did not. Build them once for the whole first pass, and only when spots actually have to be found (with --reuse-rotation-spots there is nothing to allocate). Reuse is safe because every azimuthal-integration path - CPU, GPU and the fused adaptive engine - clears the caller's profile before adding to it, so each frame's output is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
6be94f2be0 |
Merging: do not subtract a negative intensity's Poisson term
The expected-variance weights decompose an observation's sigma^2 into a background part and a Poisson signal part, then rebuild the signal part at the reflection's merged mean. The decomposition subtracted corr*I with I taken as-is, so a negative I ADDED to the background part: an observation at I = -1.5 with sigma^2 = 1 came out with a base variance of 2.7 rather than 1. That inflates the variance of precisely the down-fluctuated observations the correction exists for. Below about one photon they are then under-weighted and the merged mean is biased high - the same direction of error, in the same regime, that weighting by the observation's own sigma produces. Subtract max(0, I) instead: a negative intensity has no Poisson signal to remove. Both users of the decomposition are fixed - the stills merge, where expected-variance weighting is now the default, and the rotation combine it was mirrored from, which had it first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
59977ae910 |
rugnux: --spot-high-resolution 0 really means no limit on rotation data
Passing 0 reset the limit to "unset" and logged "No high resolution limit for spot finding: as far as the detector reaches" - and then, 700 lines later, the rotation default put 1.5 A back, because unset carried two different requests: the user said nothing, or the user asked for none. The log said one thing and detection did another, and there was no way to lift the limit on rotation data at all. Remember whether the option was given, and apply the rotation default only when it was not. Documented in the usage message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
c40e09d458 |
rugnux: report an ambiguous space group instead of silently picking one
The two-arm check for "same order, different symmetry" ran after the all-observation arm had already been adopted by value, so it compared that arm against a copy of itself: the point-group names were always equal and the branch was dead. In the other direction the orders were always unequal, so it was dead there too. The AMBIGUOUS warning - written for the case where the two merges support different symmetries of the same order, which implies two different molecular-replacement searches - could never be emitted. Check before the adoption, while both arms still hold their own result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
157698d31e |
Space-group search: report the order of the point group it chose
The two-arm search compared its arms by the order of the space group each had picked, but Stage B leaves best_space_group unset whenever no candidate is eligible - no candidate had enough observed absences to trust. That is not rare on the Lorentz-filtered arm, and for a systematic reason: the filter removes the badly-measured observations, which is where the weak systematically-absent reflections are. An arm that confirmed 422 but stopped short of naming a space group therefore scored order 0 and lost to an arm supporting P2, and the demotion was logged as "taking the higher symmetry" - the comparison and the message both wrong, in the one direction the design says cannot happen. Carry the point-group order in the result, set from the order Stage A actually adopted, and compare on that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
0162afd324 |
Azimuthal integration: report the dimensions of the image the mapping covers
The mapping took its width and height from the CONVERTED geometry unconditionally, while pixel_to_bin is sized per mode: converted when the geometry is transformed, raw module layout when it is not. In raw mode the two disagreed - 2068x2162 reported against a 1024x4096 map on a JF4M. Only the adaptive spot finders read those dimensions, and they read them for exactly the thing that breaks: the CPU finder derives npix = w*h and then indexes the image, pixel_to_bin and the resolution mask with it, so it walked ~277k pixels past the end of all three; the GPU finder stays in bounds but decodes the strong-pixel bit index with the wrong row stride and reports spots at wrong coordinates. Nothing combines raw geometry with adaptive detection today, so this was latent rather than live. Take them from GetXPixelsNum()/GetYPixelsNum(), which already follow the geometry mode. The converted path is unchanged - it is the same number there - and every internal use is inside SetupConvGeom, which only runs when the geometry is transformed. Covered by two tests: the mapping's dimensions must match pixel_to_bin in both modes, and the CPU adaptive finder must return a spot planted on a raw-geometry image at that raw pixel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
1a0774eed0 |
Azimuthal integration: clear sum2 along with sum and count
Clear() reallocated sum and count but left sum2 as the constructor had sized it. Every caller reuses one profile across the images of a dataset (rugnux keeps one per worker, the viewer one per view), so the squares kept accumulating while the means restarted at zero: the per-image standard deviation written to /entry/azint and shown in the plots was meaningless from the second image on, and grew without bound over a run. The size mismatch was the sharper edge. Clearing to a mapping with a different bin count left sum2 shorter than sum, and GetStd() and operator+= then read past its end - reachable in the viewer by opening a dataset with a wider q range than the one before it. Both are covered by tests: the same frame twice with a Clear() in between has to give the same standard deviation, and a profile cleared to a wider mapping has to report the right value in a bin that only the wider mapping has. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
8cb1cacadf |
rugnux: show the space-group search on rotation data
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m12s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m47s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m27s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m54s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m58s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m16s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m20s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m53s
Build Packages / build:rpm (rocky8) (push) Successful in 11m27s
Build Packages / XDS test (durin plugin) (push) Successful in 7m49s
Build Packages / Generate python client (push) Successful in 34s
Build Packages / Build documentation (push) Successful in 1m6s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m49s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m23s
Build Packages / build:rpm (rocky9) (push) Successful in 13m51s
Build Packages / DIALS test (push) Successful in 14m5s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m12s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m7s
Build Packages / Unit tests (push) Successful in 1h0m25s
The search runs in pass 1 of the rotation two-pass; pass 2 only reuses the
group it decided, so the result the CLI renders carried no search at all
and the whole report - operator correlations, the space-group candidate
table, the refused higher symmetry - was silently dropped on every
rotation dataset. Most costly of all, the "or <group> (indistinguishable
from these data)" line never appeared, so an enantiomorphic pair the
intensities genuinely cannot separate was reported as a single answer.
Carry pass 1's search into the returned result.
Also de-duplicate the alternatives when the centred-lattice test swaps the
metric-matching candidate into the answer: the group it displaced was left
out and the chosen one listed twice ("C2 or P21 or C2").
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
leonarski_fandClaude Opus 5
|
d28db19ab1 |
Space-group search: judge a screw axis against its own axial row
A reflection the group predicts absent counted as a violation when
I/sigma > 3 AND E^2 = I/<I>(shell) > 0.3. Neither half survives contact
with real data:
* merged sigma is floored at b|I|, so merged I/sigma saturates at ISa
for nearly every reflection - the I/sigma half is an on/off switch
keyed on ISa vs 3, not a per-reflection test. On one crystal the
absent class read <I/s> 4.10 against 3.73 for the present class while
being genuinely extinct;
* <I>(shell) decays with resolution while a systematically-absent
reflection keeps a small NON-decaying residual (background / profile
leakage), so absent reflections drift over an absolute E^2 cut at high
resolution. That cost a tetragonal 42_12 crystal its 4_1: 18 of its 47
absent 00l crossed the cut, all beyond 3.7 A, at absolute intensities
identical to the low-resolution ones correctly judged absent, while
their l=4n row-mates sat 20-60x higher at the same resolution.
A screw extinguishes only the reflections that lie ON its axis, so the
fair yardstick is the rest of that same row. The threshold is now
0.3 * max(1, median E^2 of the reflection's own row), the row being the
gcd-reduced reciprocal-space direction and the control class the same-row
reflections the group predicts present. Floored at 1, so it only ever
relaxes: a screw can be recovered by it, never lost.
Per row, not pooled. A 4_1 along c and a 2_1 along a are separate
conditions with separate controls; pooling let the weak a/b rows (median
E^2 ~0.5) set the threshold for a strong c row (8.4) and the rescue never
fired.
The candidate table now reports the screw evidence (median E^2 of the
absent class and of its rows) - the <I/s> columns are the centering
evidence and say nothing about screws, for the sigma-floor reason above.
Rotation battery, 33 crystals: 31 decisions bit-identical, the 42_12
crystal recovers its 4_1 (0 violations, row E^2 8.4 vs absent 0.12), and
one crystal with a long axis and heavy 00l overlap moves to a 4_1 group at
exactly 10.0% violations - marginal, and its sister crystal of the same
form sits at 13.3% and does not move. Real screws now span 0-9.3%
violations, so max_absent_violation_fraction cannot be tightened below
0.10 without risking a genuine one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
leonarski_f
|
43e9de9573 |
VERSION: 1.0.0-rc.161
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m50s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m41s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m17s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 12m23s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 12m50s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 12m52s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m21s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m53s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m34s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m3s
Build Packages / build:rpm (rocky8) (push) Successful in 12m47s
Build Packages / build:rpm (rocky9) (push) Successful in 13m1s
Build Packages / Generate python client (push) Successful in 18s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 52s
Build Packages / XDS test (durin plugin) (push) Successful in 7m39s
Build Packages / DIALS test (push) Successful in 12m1s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m21s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 6m52s
Build Packages / Unit tests (push) Successful in 1h2m5s
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
|
||
|
leonarski_fandClaude Opus 5
|
40acf1ffc4 |
rugnux: search the space group twice by default on rotation data
--search-min-zeta now defaults to 0.85 for rotation, so the de-novo search runs on a merge of all the observations AND on a merge of only the well-measured ones, and keeps whichever found more symmetry. Previously it shipped off and the second opinion had to be asked for. Rotation battery, 33 crystals, NO flags beyond the resolution limit: fixed-threshold finder 30/33 - zero crystals differ from the single search adaptive finder 30/33 - the same three mismatches Both arms now agree crystal for crystal, which they have not done before. The last disagreement was a thaumatin set whose 4-fold sits 88.9 deg from the spindle: at defaults it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken) where it read P222. The classic arm is a strict no-op - zero differences against both the explicitly-flagged run and the run predating the dual search - so the default costs nothing where the geometry is not the problem, and 47.2 s against 47.8 s on the same crystal back to back. The default is safe to set because the two searches can only disagree by a LOST operator: discarding observations starves an operator correlation, it cannot invent one. That also makes the 0.85 itself uncritical - too aggressive a cut only means the second opinion contributes nothing and the full merge wins. --search-min-zeta 0 restores the single search. Docs: CHANGELOG gains a 1.0.0-rc.161 section covering the branch, and CPU_DATA_ANALYSIS records the four analysis changes of this work - the confidence-weighted per-image refinement, the collapsed per-frame scale guard, the opt-in per-image rejection, and the operator-disagreement criterion with the two-search rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
25458265d3 |
Space-group search: ask twice - all observations, and only the well-measured ones
Build Packages / build:viewer-tgz:cpu (push) Successful in 8m20s
Build Packages / build:viewer-tgz:cuda (push) Successful in 9m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m32s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m58s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m0s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m9s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m22s
Build Packages / build:rpm (rocky8) (push) Successful in 11m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m47s
Build Packages / XDS test (durin plugin) (push) Successful in 8m57s
Build Packages / Generate python client (push) Successful in 39s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky9) (push) Successful in 12m46s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m32s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m31s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m52s
Build Packages / DIALS test (push) Successful in 14m42s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m32s
Build Packages / Unit tests (push) Successful in 1h15m42s
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
--search-min-zeta rescues a point group that the full merge cannot confirm, but used on its own it is a trade: on the crystal it was built for it recovers the correct 422, and on four others it costs the space group outright, because discarding 40-80% of the observations starves operator correlations that were perfectly healthy. Both ways of applying it - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade the SAME crystals, so the cut itself is the problem, not where it is applied. Filip's observation makes it one-way: every disagreement between the two is a LOST operator, never an invented one. Discarding observations can starve a correlation; it cannot manufacture symmetry that is not there. So run the search on both merges and keep whichever found MORE symmetry, and the failure mode disappears - each arm rescues the other exactly where it fails. crystal all observations Lorentz-filtered adopted thaumatin (weak) 222 422 422 tetragonal lysozyme 422 222 422 cubic insulin x3 23 2 / 222 23 The filtered merge is used ONLY to rescue the point group. The screw and centering determination always comes from the merge with all the observations, because systematic absences are decided by the WEAK reflections and the filter throws most of them away. Preferring the filtered arm on a tie is not a conservative choice, it is a wrong one: it cost four crystals their screw axes (P2(1) read as P2, P4(1)2(1)2 as P42(1)2) with the point group and every intensity statistic identical - a regression invisible to CC1/2, R_meas and ISa. Where the two find the same ORDER but different symmetry, nothing can prefer one, so the run says so: it names both space groups, states that the data do not decide, reports which one processing continued in, and gives the flag to force the other. Two candidates of the same order imply different molecular replacement searches, and trying both is cheap next to reprocessing - much cheaper than a confident wrong answer. Rotation battery, 33 crystals, both spot finders: fixed-threshold finder 30/33 - ZERO crystals differ from the single search adaptive finder 30/33 - the same three mismatches, gap CLOSED The adaptive finder now matches the fixed-threshold one exactly, which it has not done before: its last remaining loss was the thaumatin set whose 4-fold sits 88.9 deg from the spindle, and it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken). A merohedral twin stays refused in BOTH arms at all three frame ranges where it over-promotes, and at one of them the second opinion is strictly better than shipping behaviour - the full merge collapses to P1 where the filtered one finds the correct H3. Cost is the extra scale-combine-merge on already-ingested partials, with no re-integration: 47.2 s against 47.8 s on the same crystal back to back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
f2b92e3f4d |
rugnux: --search-min-zeta drops badly-measured observations from the symmetry search
Build Packages / build:viewer-tgz:cpu (push) Successful in 8m30s
Build Packages / build:viewer-tgz:cuda (push) Successful in 9m3s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m52s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m4s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m23s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m27s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m59s
Build Packages / build:rpm (rocky8) (push) Successful in 11m45s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m59s
Build Packages / XDS test (durin plugin) (push) Successful in 7m14s
Build Packages / Generate python client (push) Successful in 29s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky9) (push) Successful in 13m21s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 13m3s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m38s
Build Packages / DIALS test (push) Successful in 14m4s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m4s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m49s
Build Packages / Unit tests (push) Successful in 1h1m28s
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
zeta is the sine of the angle between a reflection's rocking path and the spindle. Near 0 the reflection crosses the Ewald sphere almost tangentially, spends many frames in diffracting position and is measured worst. The de-novo space-group search asks how EQUAL an operator's paired intensities are, so its answer is dominated by whichever reflections are measured worst - and when the spindle lies in a lattice plane, an operator that permutes the two in-plane axes samples a different mixture of measurement qualities than one that only flips signs. That is not a fair comparison, and it can make a real symmetry operator look like a twin law. Measured on a thaumatin set mounted that way (its 4-fold is 88.9 deg from the spindle), the added operators' disagreement is 1.74x the parent's over pairs where both reflections have zeta < 0.85 and 1.003x - i.e. the symmetry is exact - over pairs where both are above it. The search consequently refuses the 422 promotion and merges the crystal in P222, while the same data forced to the right group give CC1/2 99.2% at multiplicity 10.7, matching XDS. With the option the de-novo pass ignores those observations (the final merge keeps everything - there completeness is the point): zeta cut observations ignored H ratio adopted 0 (off) - 1.47 P222 0.5 1620648 1.44 P222 0.7 3006013 1.34 P21212 0.85 4536724 promoted P4212 (correct point group) OFF BY DEFAULT, and it must stay off, because the same cut costs four other crystals their space group (P41212 -> P212121, I23 -> P2, I23 -> I222 twice): at 0.85 it discards 40-80% of all observations, which on a crystal whose geometry is not the problem simply starves the search. Two independent implementations - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade exactly the same crystals, so this is a property of the cut and not of where it is applied. Verified bit-identical to the previous binary when off. The companion diagnostic is already there: the run now reports how close a symmetry axis lies to the spindle, which is the geometry that makes this option worth reaching for. Implementation note for anyone tempted by the cheaper route: excluding these observations from the ASU grouping alone does NOT work. The 3D combine selects partials on corr, not on their group, so their intensity still reaches the fulls and the merged intensities are unchanged - measured, the statistic did not move by 0.03 while 67% of observations were nominally excluded. Zeroing corr is what removes an observation from the combine, the merge and the error model alike. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
eb70684fa9 |
rugnux: report how close a symmetry axis lies to the spindle
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m48s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m57s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m20s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m22s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m31s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m36s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m59s
Build Packages / build:rpm (rocky8) (push) Successful in 11m47s
Build Packages / XDS test (durin plugin) (push) Successful in 7m42s
Build Packages / Generate python client (push) Successful in 27s
Build Packages / Build documentation (push) Successful in 1m8s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 13m6s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m18s
Build Packages / build:rpm (rocky9) (push) Successful in 13m58s
Build Packages / DIALS test (push) Successful in 14m9s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m11s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m6s
Build Packages / Unit tests (push) Successful in 1h2m16s
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
A rotation sweep never records the reflections whose reciprocal vector lies within the Bragg angle of the spindle - the blind cusp. Symmetry normally supplies them from an equivalent elsewhere in reciprocal space, so the hole closes. It cannot when a symmetry axis IS the spindle: the cusp is then mapped onto itself, every reflection in it is equivalent only to other reflections in it, and it stays empty however long the sweep runs. The user can fix this at the microscope - re-mount, or add a sweep on another axis - but only if they are told, and nothing in the output mentioned it. Report the smallest angle between any proper rotation axis of the adopted space group and the goniometer axis, always on rotation data, and warn when it falls under 15 deg. The axis is found by projecting onto each operator's invariant direction (the sum of its powers annihilates everything else) and mapping that fractional direction through the refined lattice into the lab frame; the angle is invariant under the sweep, so the reference orientation is enough. Cross-check: this reports 30.6 deg for a crystal whose 4-fold an independent analysis of the XDS orientation matrix put at 30.5 deg. Measured on three rotation sets: 13.6 deg (2-fold, warns), 16.2 deg (2-fold, 99.7% complete) and 30.6 deg (4-fold). The 15 deg bound is practical rather than derived - the blind cone's half-angle is the maximum Bragg angle, ~15 deg for 2 A data at 1 A wavelength - and the wording says what the diagnostic can honestly support: the angle is a risk indicator, the loss is confined to the cone rather than spread over the data, and overall completeness may still look reasonable while the region near the spindle is empty. It does not promise a completeness number, because across those three sets the overall figure does not track the angle (99.7% at 16.2 deg, 92.6% at 30.6 deg). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
b81c6f00b7 |
reader: take the stored image bit depth from the file, not the DECTRIS default
DetectorSetup hardcodes bit_depth_image = 16 for every DECTRIS detector (DetectorSetup.cpp:78), and GetByteDepthImage() consults that BEFORE the depth the reader takes from the file - so a file storing 32-bit images had its overflow computed as a 16-bit one. With the reader also declaring the images signed, GetOverflow() returned INT16_MAX and GetSaturationLimit() became min(file value, 32767). Every count above 32767 was therefore marked saturated, and because the integration accept gate requires ALL inner pixels valid, the whole reflection was discarded. That silently removes the strongest reflections of a strong crystal - the low-resolution ones that anchor scaling - while the file itself declares saturation at 105000-133000. Measured on a lysozyme rotation set (200 frames), before -> after: saturated pixels per frame 0.815 -> 0.000 brightest accepted pixel 32738 -> 87633 mean per-frame maximum 26218 -> 37390 i.e. the ceiling was exactly INT16_MAX and nothing genuine reached it. Which datasets this touches depends on how bright they are: measured pixels above the old ceiling range from 0.0 per frame on some rotation sets to 6.1 on others, so the fix is a no-op on weak data and only ever adds reflections. Rotation battery, 33 crystals: no point group changed (30/33 before and after) and no run failed. Four crystals move on quality, in both directions - ISa 1.90 -> 2.40 and 3.29 -> 4.80 on two, 2.97 -> 1.85 and 20.83 -> 18.47 on two others; three of the four are the battery's known weak or run-to-run-unstable crystals. The one strong crystal that moves gains 136 observations out of 1.9 million and loses 2.4 ISa: the reflections restored are by construction the brightest ones, and they carry the systematic error that the strongest reflections always carry. That is a real cost, but it is the cost of MEASURING them rather than discarding them unseen, and a lower asymptotic I/sigma on data that are now complete is preferable to a flattering one on data that quietly are not. Only the offline file reader is affected; the online path builds its detector setup from configuration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
6f4917dcee |
rugnux: adaptive spot detection is the default for rotation data too
Build Packages / build:viewer-tgz:cpu (push) Successful in 8m0s
Build Packages / build:viewer-tgz:cuda (push) Successful in 9m13s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m40s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m48s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m42s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m27s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m43s
Build Packages / build:rpm (rocky8) (push) Successful in 11m32s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m9s
Build Packages / XDS test (durin plugin) (push) Successful in 7m22s
Build Packages / Generate python client (push) Successful in 27s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m57s
Build Packages / build:rpm (rocky9) (push) Successful in 13m25s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m50s
Build Packages / DIALS test (push) Successful in 13m52s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m0s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 8m55s
Build Packages / Unit tests (push) Successful in 1h2m59s
Build Packages / build:windows:nocuda (push) Canceled after 0s
Build Packages / build:windows:cuda (push) Canceled after 0s
It was held back because a 33-crystal rotation battery showed it breaking three
crystals deterministically - a lost space group, a halved indexing rate and a
collapsed merge. None of those causes turned out to be in detection.
The extra spots adaptive finds are real. Measured per spot against a
finder-neutral local background: 64% recur at the same position on the adjacent
frame (chance rate 0.5%) with 2-frame rocking curves, and 0.00% would fail a
conventional local SNR >= 4 test, median local SNR 34. What they include is
genuine peaks belonging to no lattice the indexer found, and the damage they did
scaled with their absolute COUNT (80.6 per frame against 36.8 for the fixed
finder), not with their quality - which is why nothing aimed at judging
individual spots ever worked.
The three failures fell to fixes elsewhere:
merge collapsed - a per-frame scale free to collapse toward zero amplified
two junk frames by 546x (
|
||
|
leonarski_fandClaude Opus 5
|
4895dc1018 |
Rotation: let --min-image-cc drop frames that disagree with the merged reference
The flag was accepted on rotation data and did nothing - it is read only by the stills merge (Merge.cpp), and the CLI warned about that rather than fixing it. Meanwhile RotationScaleMerge already COMPUTES a per-frame correlation against the merged reference and writes it to the per-image table; nothing acted on it. Wire the two together. A rejected frame has its partials' corr set to 0, which is how a frame already leaves the pipeline - every consumer requires corr > 0, so the combine, the merge and the error model all drop it together. The GPU path reuses the SmoothCorr kernel with a ratio of 0, so one implementation covers both. Off by default (0), and verified bit-identical to the previous binary when off. What it catches, on the two rotation datasets that have a population to catch: a two-lattice crystal - two lattices in two physical AREAS of the sample, so the sweep passes from one to the other and whole blocks of frames measure a different crystal from the one being merged (frames 500-700 index perfectly well at a per-frame CC of 0.22 against 0.47-0.56 either side, in 11 contiguous runs). R_meas 28.6 -> 24.6%, CC1/2 93.6 -> 95.1, high-shell CC 23.4 -> 38.3. a second dataset with 9.5% of frames below CC 0.30: R_meas 24.3 -> 23.4%, CC1/2 92.6 -> 93.4. The criterion is "this frame disagrees with the merged reference", NOT "this frame is off-crystal". It happens to catch both, because a frame that measures nothing and a frame that measures a DIFFERENT crystal fail the same test, and it does not need to know which. For the two-area case that is a workaround, not a treatment: it recovers one crystal by discarding the other, where processing the two as separate sweeps would keep both. The frame-block structure is clean enough that such a split could be detected automatically. WHY THERE IS NO DEFAULT. The per-frame CC is not comparable between datasets - it is as much a measure of data quality as of frame validity. Measured medians across the battery run from 0.30 to 0.81, so one absolute bound removes 13 frames from one dataset and 584 of 1800 from another: battery at --min-image-cc 30, 33 crystals: no point group changed (30/33), four crystals clearly better (one +5.4 CC1/2 points, the two-lattice case above, and ISa gains of 1.3-4.6 on three others) - and one healthy crystal lost a third of its frames and with them its high-resolution shell (CC1/2_hi 26.2 -> 2.0). This is the same trap as an absolute bound on any per-operator or per-frame agreement statistic, and the same one the per-frame scale guard avoids by measuring against the run's own median. A principled version would cut on the SHAPE of the per-frame CC distribution - a dataset with a bad subpopulation is bimodal, a uniformly weak one is not - rather than on an absolute value. Until that exists this stays opt-in, and the per-image CC it keys on is already in the _image.dat table for anyone choosing a value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
11c7cab2e5 |
Space-group search: take the operator disagreement as a median, not a mean
A merohedral twin mixes EVERY reflection with its twin mate, so it shifts the whole distribution of |I1-I2|/(I1+I2). A minority of badly measured reflections shifts only the tail. The mean cannot tell those apart; the median is blind to the second and just as sensitive to the first. Measured on real crystals, moving the statistic from the mean to the median leaves genuine promotions where they are and pushes every twin up: genuine tetragonal 1.016 -> 1.013 genuine lysozyme 1.051 -> 1.067 genuine tetragonal 1.238 -> 1.231 twin (-e 1050) 1.272 -> 1.447 twin (-e 450) 1.280 -> 1.622 twin (full) 1.441 -> 1.522 twin (-e 600) 1.427 -> 2.010 The margin around the 1.25 bound widens from 2.7% (genuine 1.238 against twin 1.272 - uncomfortably tight for a decision that cannot be undone downstream) to 17.5% (1.231 against 1.447). The bound itself does not move. Rotation battery, 33 crystals in both detection modes: no point group changed in either (30/33 and 29/33, as before), and only one crystal's numbers move at all - the one already documented as nondeterministic between repeat runs of the same binary. The synthetic twin-fraction x multiplicity grid passes unchanged. So this buys margin, not outcomes. Found while testing a different hypothesis, which the same measurement refuted: a tetragonal crystal whose 422 promotion is wrongly refused reads 1.484 by the mean and 1.472 by the median, i.e. its disagreement is distribution-wide and is NOT a badly-integrated minority. That crystal's cause is elsewhere and is not addressed here - see the note below. Its indexing-ambiguity operator (-k,-h,-l) lies INSIDE 422 but OUTSIDE 222, so the subgroup merge the search is given mixes lattices indexed in the two alternative hands. That corrupts exactly the 4-fold relationships and leaves the 2-fold ones intact - measured, the 222 step reads 0.917 and the 422 step 1.484 - and the corruption is indistinguishable from a twin law. Forcing the tetragonal group merges the two hands as equivalent and the same data give CC1/2 99.2% at multiplicity 10.7, matching XDS. The failure is worse the BETTER the frames index (99.9% vs 63.3% for the run that gets it right), because indexing more frames picks up more of both hands. So no statistic computed on a subgroup merge can arbitrate a promotion whose added operators include an indexing-ambiguity operator. Fixing that means resolving the ambiguity before the search, or detecting the coincidence and deciding another way; the operators needed to detect it are already computed (the run warns about them). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
ae126c3d5b |
Per-image refinement: weight each spot by how strong it is for its resolution
`RefineGeometryIfNeeded` hands XtalOptimizer the WHOLE spot list, not the
indexed subset, and the first pass admits anything within 0.3 fractional-Miller
units of an integer - which is 11.3% of RANDOMLY placed spots, since the
admitted volume is (4/3)*pi*t^3. Every one of them then enters an unweighted L2
fit with an arbitrary rounded index. On images with many detections the
refined orientation ends up 2.3-2.8 degrees from the goniometer-consistent one
and explains 14 of its own 250 spots where the undragged orientation explains
68; mosaicity and profile radius inherit the error and integration follows.
Weight every spot by its intensity divided by the median intensity of its own
equal-count resolution shell, applied as w^2 on the squared residual with
w^2 = r/(1+r). The shell normalisation is the point: refinement needs the
high-resolution spots because they carry the cell and distance, and those are
LEGITIMATELY weaker, so a raw intensity weight would suppress exactly the
spots the fit depends on. Measured, the weight is resolution-neutral - median
exactly 0.707 in every shell, and corr(w, 1/d^2) = -0.20 / -0.11 against
-0.32 / -0.34 for the same function of un-normalised intensity.
This is a PRIOR: it is computed from the spot alone and never looks at the
current residual, so unlike a robust loss it cannot mistake a genuine spot for
an outlier while the starting geometry is still far off and leave the fit
unable to move. That failure is not hypothetical - a CauchyLoss on this same
residual, at the scale the multi-frame GeometryRefiner uses, collapsed one
crystal's indexing rate from 99.89% to 19.83% and was rejected.
It does not work by telling good spots from bad, and it does not need to. No
per-spot property separates spots that index from spots that do not: measured
AUC is 0.53 for peak pixel, 0.53 for total intensity, 0.51 for pixel count,
0.45 for peakedness, and a logistic regression on all twelve available
features with pairwise interactions reaches only 0.64. What the weight does is
halve the EFFECTIVE COUNT of every spot (mean w^2 = 0.517), and the damage
scales with the absolute count of unexplained spots in the objective - 80.6
per frame here against 36.8 for the finder that was never damaged. That is
also why an empirical `--max-spots 66` cap works while leaving the list no
purer than before: it reaches the same operating point by discarding spots.
This reaches it without discarding any, and without a tuned constant.
Rotation battery, 33 crystals, both spot finders:
finder A 29/33 -> 30/33 point groups (one crystal P222 -> P4212 = XDS,
its high-shell CC1/2 86.0 -> 98.4)
finder B 28/33 -> 29/33 point groups (one crystal I222 -> I23,
its high-shell CC1/2 14.8 -> 38.0)
No crystal lost its point group in either mode and no run failed. On the
meta-stable multi-lattice dataset the CC1/2 spread over four frame ranges
falls 19.7 -> 13.1 for finder B, and the indexing rate rises in 8 of 8
configurations. The crystal that the rejected robust loss destroyed keeps its
99.89% indexing rate exactly.
The cost, stated plainly: ISa falls by 0.2-1.7 on about five crystals (and
rises on two). Point-group correctness is worth more than that - merging in
the wrong symmetry cannot be undone from the output, whereas ISa is a quality
metric of data that remain correct - but it is a real trade and not a free win.
Off by default. The indexers pass a spot list they have already selected, so
their calls are unchanged; only the per-image refinement, which gets the raw
list, turns it on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|