40 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	2c51e00aae	Rotation indexing: do not keep a metric symmetry that indexes almost nothing ... The Bravais class is decided from the UNREFINED FFT candidate against a fixed 3 degree angular tolerance (LatticeSearch). A lattice that is pseudo-symmetric to a few tenths of a degree is therefore promoted a class too far, and the constraint then snaps a real angle to the ideal one - which throws nearly every reflection of every frame out of tolerance. Measured on a monoclinic crystal that is pseudo-C-orthorhombic to 0.42 degrees: the promoted cell indexes 2 of 60 validation frames and the run dies, where its own primitive cell indexes 39. It is the same lattice in a different setting, b_oC = -(a + 2c), volume exactly 2.00x. The perverse part is that BETTER SPOTS MAKE IT WORSE. LatticeSearch applied to the true cell returns the promoted class deterministically; runs that succeed escape only because the raw FFT candidate is inaccurate enough to miss the promotion window. So it is bistable and non-monotone in every knob - 190 spots per image gives 44/60, 195 gives 12/60, 200 gives 2/60 - and it will bite harder as spot finding improves. The indexer already refines a free triclinic cell alongside each constrained candidate, but decides between them on the fraction of the accumulated first-pass cloud that indexes, where the two differ by less than a factor 2 (measured 0.243 vs 0.135, missing both of that guard's bars). The caller has a far sharper statistic: it already counts how many of 60 validation frames a candidate indexes, and there the same pair differs by more than 20x. So keep the triclinic cell instead of dropping it, and let the first pass settle it. The bar is a clear majority, not a margin, and that is the part that took a battery to get right: the unconstrained refinement holds NO cell parameter fixed, so it can only index at least as many frames as the constrained one, and on genuine symmetry it does index a few more. A 10 % margin - the bar a later scheme needs to displace an earlier one - demoted a real I-centred orthorhombic crystal to P1 (47 -> 54 frames) and perturbed an F-cubic one (49 -> 58). Only a constrained cell that fails outright while its unconstrained cell works is evidence of a false promotion, so demand exactly that. It is the same "fails to index half the frames" test the long-axis rescue below already uses. Battery over 37 rotation crystals: 33/37 space groups matching XDS with one hard failure becomes 34/37 with none, and the other 36 crystals are identical in every printed statistic (checked against a repeat run of the previous binary, which itself differs on one crystal by one observation). The extra validation pass runs only where the constrained cell already failed - 71 ms in a 15 s run - and not at all on the 34 crystals whose constrained cell indexes a majority. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 22:11:11 +02:00
leonarski_fandClaude Opus 5	3f7b4b4686	rugnux: say which lattice the first pass rejected ... "Two-pass rotation indexing found only a lattice that indexes 2/60 validation frames" tells a user that something went wrong and nothing about what. The commonest cause is a metric symmetry promoted one class too far: the constrained cell then misses every reflection by the small angle the constraint snapped away, and the run dies with no way to see that a centred supercell setting was chosen over the primitive one it should have kept. Print the centring, crystal system and cell that was rejected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 21:17:10 +02:00
leonarski_fandClaude Opus 5	b5f5879a1d	rugnux: measure the ice in the first pass, and always find its own spots ... Ice handling was gated on a measurement the run only made AFTER the images had been processed, so the per-image pass could not use it. The flagging therefore ran unconditionally: ice-band spots were ordered last in the --max-spots budget and held out of the indexer seed and the geometry refinement on every crystal, iced or not. The eleven bands are fixed geometry holding 16-26 % of the unique reflections whether or not there is ice, so on a clean crystal that discards a fifth of the spots - the strongest first - for nothing. Measured on a crystal whose gate never fires, that moved the merged data by a mean of 0.85 sigma against a run-to-run floor of 9.3e-5. Measure it in the first pass instead. That pass already looks at ~100 images spread over the sweep, and it already stops at the spot finder, so it sees the azimuthal profile for the smooth channel and the unfiltered connected components for the spot channel. Both counts SpotAnalyze takes are pre-filter, so pooling them there is the run's own verdict, reached before anything has been discarded and in time for the pass that acts on it. Where the sample sees no ice, the run indexes on the ice-band spots too. It has to be the whole sample: the spot channel is a ratio pooled over images, because one frame carries a handful of control spots. A per-image gate is not an alternative - two of the crystals whose indexing this rescues fire on that channel alone, at profile scores of 1.12 and 1.22, so gating per image on the profile score would drop exactly the cases that matter. This also removes the first-pass spot reuse, and with it --redo-rotation-spots and the reuse path. Finding the ~100 first-pass spots costs little, and reusing was actively wrong here: the stored spots were found online at the acquisition's threshold and have already had their ice-band entries ordered last and dropped by its spot budget, so counting ice from them under-reads it by construction, and the lattice search never saw the spot-finding settings at all. It also removes the need for the machinery that re-found spots whenever a spot-finding option was named, which made those options impossible to A/B. IndexAndRefine cached index_ice_rings at construction, which happens before the first pass; it holds a reference to the experiment, so it now reads the setting where it uses it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 20:48:22 +02:00
leonarski_fandClaude Opus 5	61a7c91b90	Ice: detect it on two channels, and only handle it when it is there ... The per-image ice score was read off the PLAIN azimuthal profile. That profile is a per-ring mean, so a few strong Bragg reflections landing in a ring's q bin lift it exactly as ice would. Measured over 37 rotation crystals, that did not merely add noise - it INVERTED the metric: the two highest-scoring crystals had no ice at all (4.23 and 4.06), while a clean control read 1.57. A decoy null - the identical statistic evaluated at q positions where hexagonal ice cannot be - reaches 1.51 at its 99th percentile and 2.70 at its maximum, so that metric cannot support any absolute threshold whatsoever. The adaptive spot finder already computes the right input for its own threshold: a sigma-clipped per-resolution-ring background, in the same bins. A powder ring is azimuthally smooth and survives the clip; Bragg peaks do not. On the clipped profile the clean population tightens to 1.00-1.22 and the crystals with confirmed ice sit at 2.08-2.37, against a decoy null that never exceeds 1.29. That channel is blind to one thing: ice in large crystallites diffracts as DISCRETE spots and leaves the radial profile flat. So a second channel counts found spots on the rings against the same q width of ice-free flanks beside them. The two barely overlap - the smooth-ice crystals read 2.1-2.4 / ~1.0 and the textured ones ~1.1 / 3.8-17.6, while a clean crystal reads 1.04 on both. Both are then used as a GATE (--ice-min-score 1.5, --ice-min-spot-ratio 2.0, both calibrated on the battery, 0 disables): the eleven fixed hexagonal bands cover 16-26 % of the unique reflections at typical resolutions whether or not the crystal has ice, so flagging, the exclusion from the scale fit and the merge-time CC1/2 ring mask are now all skipped when neither channel sees any. The gate is applied in the full pipeline and in --scale, which reads the stored per-image values back out of the _process.h5. Also fixes the merge-time mask's control: the shoulder now excludes reflections that are themselves on an ice ring. The rings are not evenly spaced - 1.947/1.916/1.882 A sit 0.05-0.06 apart in q - so for those three the [w,3w) shoulder landed squarely on the neighbours and the test compared ice against ice. Measured, that is the only thing this changes: it removes firings on those three rings and leaves every other firing's CC pair identical to three decimals. And the online ice half-width, which was 0.02 in the API against 0.03 offline, so the same data got a narrower band online than the measured ~0.06 ring FWHM justifies. Battery (37 rotation crystals, against the previous behaviour): space groups 34/37 in both and NO crystal's space group changes; 6 crystals gain unique reflections, 1 loses. Best of them gains 7082 unique reflections with R_meas 16.0 -> 14.3, CC1/2 95.9 -> 97.3 and ISa 13.7 -> 19.0; another goes R_meas 54.9 -> 42.9, CC1/2 84.0 -> 90.4, ISa 3.9 -> 5.5; a third reaches CC1/2 99.4 from 95.7 at an unchanged reflection count. The one crystal that loses reflections improves on both R_meas and CC1/2. Not done here: the ScanResult/API/plot-type/frontend/viewer layers for the new spot_count_ice_control (they need the OpenAPI regeneration). Message, CBOR, HDF5 write/read and the receiver plots are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 16:17:23 +02:00
leonarski_fandClaude Opus 5	bb7df09086	rugnux: separate the merge-time ice-ring mask from ice detection ... --detect-ice-rings did two unrelated jobs at once: flagging ice spots so indexing de-prioritises them and keeping ice reflections out of the scale fit, AND gating the merge-time mask that drops a decorrelated ice ring and re-merges. Turning it off to de-confound a merge-stage experiment therefore also changed how the data were indexed - measured, that breaks indexing outright on two of the 37 rotation battery crystals - while leaving it on lets the mask land differently between two arms of an experiment and contaminate the comparison (measured on up to 19 of 37 crystals in response to a small intensity change). Add --ice-ring-mask[=on|off], default on, gating only the merge-time mask. Verified with =off: ice-spot flagging and the scaling exclusion still log and still apply, no mask line, no second merge, and the first error model is bit-identical to the =on arm. The full pipeline and the offline --scale path reach the same verdict on the same data, as they must. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 19:00:10 +02:00
leonarski_fandClaude Opus 5	6b713cf3da	rugnux: size the written datasets by the ordinals reached, not by the successes ... Each image is written at its own ordinal, so a frame that fails to load or analyse leaves a HOLE - the frames after it keep their positions rather than shifting up. The end message nevertheless reported the number of frames that SUCCEEDED as the image count, and the writer sizes /entry/data/data and every per-image array from that. So one failed frame in the middle of a run made the declared extent one short, and the image it dropped was the LAST one written, not the one that failed. Two failures dropped two, and so on: the file quietly ends before the data does, with the per-image metadata still carrying rows for images the VDS no longer maps. It also under-counted the data files when the run was split. Track the highest ordinal actually written and use that. A hole then reads as the fill value, which is what a frame that was never written should look like, and the counts of collected and written images stay counts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 16:33:24 +02:00
leonarski_fandClaude Opus 5	212fbf9bab	rugnux: make the geometry-refinement sample deterministic, and spread it over the run ... The stills first pass drew frames from a shared cursor and stopped when a shared counter reached its target, which got two things wrong at once. The cursor walked the equally spaced sample in ascending order, so stopping early read only its leading PREFIX - the beam centre, distance and cell were fitted to the beginning of the run, not across it, and the comment claiming otherwise was wrong. And where the stop landed depended on how the workers happened to interleave, so the set of frames varied run to run: on the same data at -N 32 and -N 8 the pass examined 483 and 457 frames and refined the detector distance to 168.0481 and 168.0530 mm. The sample is now cut into a fixed number of interleaved stripes, each stopping once it has contributed its share. Every stripe spans the whole run, so an early stop no longer biases the fit, and a stripe is processed identically whichever worker claims it - so what gets examined depends only on the data, not on timing and not on -N. The same three runs now give 451 frames examined and 168.0452 mm, identically. The bundle selection was order-dependent too: frames are collected in worker-completion order and sorted by spot count with a non-stable sort, so equally strong frames swapped places between runs. They carry their image ordinal now and it breaks the tie. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 15:19:11 +02:00
leonarski_fandClaude Opus 5	42eb5bbe83	rugnux: clamp the refinement thread count to what the setting accepts ... RefineThreads rejects anything above 64, and the two-pass rotation path - the default - passed nthreads/2 straight from the machine. On a host with 130 or more hardware threads rugnux therefore exited 1 with the bare message "Candidate-cell refinement thread count" before reading a single image, and the only workaround was to discover -N and pass a smaller value. Dual 64-core parts are squarely in range. The limit is now a named constant the setter and the callers share, so the two cannot drift apart again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 14:14:09 +02:00
leonarski_fandClaude Opus 5	7e47afe47f	rugnux: parallelise candidate-cell refinement, and stop repeating work in the tail ... Three independent changes to the CPU-bound parts of an offline rotation run, none of which alters a result. Candidate-cell refinement now splits across threads. RefineCandidateCells already took a (block, nblocks) partition, but the only call site passed nblocks=1, so the whole first pass of a two-pass rotation run sat on one thread per scheme - two threads, unchanged at every -N, for a third of the run. A block touches only its own scores(j) and cells rows and holds its own scratch, so the split is exact. The budget is a new IndexingSettings::RefineThreads, left at 1 by default and set only where few indexer threads exist: raising it unconditionally would oversubscribe the paths that already run one indexer per image across all workers. The mmCIF writer built a std::ostringstream per formatted number, twelve per reflection. snprintf gives the same digits for 0.535 -> 0.220 s per file. The space-group search built the same orbit mapping twice per candidate point group - once for the merge chi^2 and once for the systematic-error b, an apply_to_hkl and Canonicalize per observation per operator each time. Build it once and hand it to both. 18 Mpx rotation set 24.6 -> 18.7 s, 2.5 Mpx 13.0 -> 10.7 s, and the 37-crystal battery 13m55s -> 10m47s with no failures, the same 34/37 space groups, and statistics unchanged on 30 of 37 (the rest drift within the run-to-run spread the binary already had, which a control build with the split disabled reproduces). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 07:30:33 +02:00
leonarski_fandClaude Opus 5	ea667cb306	rugnux: handle ice rings in --scale as the full pipeline does ... --scale did none of the ice handling the run that wrote the _process.h5 had done, so re-scaling a stored dataset silently produced a different - and flatteringly more complete - answer than the pipeline it was meant to reproduce. Three separate gaps: * --detect-ice-rings was accepted and ignored. The --scale block returns before the line that applies it. * Reflections were never flagged as sitting on an ice ring, so the per-image scale fit included them. The flag is not stored per reflection, so it has to be recomputed from the resolution. * RotationScaleMerge was constructed with the ice half-width hardcoded to zero. That is what turns a resolution into a ring index, so every ice test inside the merge was a no-op whatever was passed to it. The CC1/2 ring test that decides which rings to drop moves into FindDecorrelatedIceRings, shared with the full pipeline so both reach the same verdict on the same data, and --scale now re-merges with the mask the way the pipeline does. The stills branch re-runs only the merge: the scaling has already been applied to the reflections and repeating it would compound it. Measured on a rotation dataset with three decorrelated rings, --scale went from 8765 unique / 36.3% completeness / R-meas 18.5% / <I/sig> 1.1 to 7638 / 31.6% / 18.0% / 1.3, against the full pipeline's 7692 / 31.8% / 17.9% / 1.3 - the reported completeness had been inflated by reflections the pipeline drops. The full pipeline is bit-identical across the refactor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 19:49:55 +02:00
leonarski_fandClaude Opus 5	cd16053c2b	rugnux: use the project PI constant, not M_PI ... MSVC does not define M_PI without _USE_MATH_DEFINES, and rugnux is part of the portable subset that JFJOCH_VIEWER_ONLY builds. JFJochMath.h already carries PI for exactly this reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 18:53:06 +02:00
leonarski_fandClaude Opus 5	639fbb3fbc	indexing: select predicted reflections by partiality, build indexers where it pays ... When more reflections are predicted for a frame than the output can hold, the surplus was dropped by keeping those closest to the Ewald sphere. On the rotation path that quantity is identically zero by construction - the rocking coordinate is chosen so the scattering vector lands exactly on the sphere - so the comparison fell through to h, k and l and the survivors were whichever came first in lexicographic order. Measured on a large cell: every value within one float ulp of zero, and the kept set had a MEAN PARTIALITY BELOW that of the full set, i.e. worse than choosing at random. Rank by partiality instead, which the predictor already computes and which is what the header always claimed was being kept. On the one regression crystal large enough to cross the cap this lifts completeness from 84.8% to 90.2% on the same observations; multiplicity and R_meas move the way they must when the same measurements cover more of reciprocal space. The online path asked for a cap of ten thousand but the truncation was hardcoded to the offline limit, so the broker predicted and integrated up to six times what it could transport and discarded the rest after paying for it. Honour the caller's limit, which also makes the post-integration re-truncation dead code. Indexer pool construction becomes a policy. The online service needs every indexer resident before data arrives, because a cuFFT plan built on the first frame is planning time inside the measurement; spending memory to be ready is the intended trade there and stays the default. Offline there is no such deadline, and a stills run with a known cell was holding a fully allocated FFT indexer per worker that the algorithm resolution can never dispatch - 2.8 GB where 0.4 GB is needed. rugnux and the viewer opt into building on first use; the broker, the receiver and the tests are untouched. This also removes a dangling reference that was latent: the worker held the settings by reference although the pool is routinely constructed from a temporary, which only survived because eager construction finished inside the constructor call. Finally, refuse a first-pass lattice that indexes fewer than a sixth of the validation frames. It fires on nothing in the regression set - the weakest real crystal sits at 22 of 60, more than twice the floor - so it is a backstop, but the failure it prevents is one the set does contain: a dataset with no crystal at all adopts a lattice from its powder rings, integrates every image against it, and dies much later inside the merge complaining about resolution. It now stops in the first pass and says what to try. Regression set: 36 of 37 crystals byte-identical, the exception being the completeness gain above; 34 of 37 space groups, no failures. Full unit suite passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 09:12:27 +02:00
leonarski_fandClaude Opus 5	bfb8cb813c	rugnux: report per-image cost honestly instead of per-worker blocked time ... Each stage timer measures wall time inside one worker, so it counts whatever that worker spent blocked on a contended resource - above all the single GPU - as well as its own work. Those waits overlap across workers, so the mean was printed as if it were the per-image cost when it is roughly the per-image cost times the worker count. At the default thread count on a large detector the reported total came out more than twenty times the truth, and single stages were printed as several times the entire per-image budget of the run. That is the one output anyone tuning performance reads, and it sent this investigation at the wrong stage for a while. Divide by the worker count. It is a lower bound - a worker idle rather than blocked is not counted - so rather than hide the remainder, report the image loop's own wall time next to it, and with it the time spent OUTSIDE the loop. Nothing measured the latter before, yet on a rotation run the first-pass indexing and the scaling and merging can be more of the run than the per-image work is: on a large-detector run here it is 5.1 s against 3.0 s. Both figures are for the last pass, and a two-pass rotation run does all of it twice. Also stop printing nan. The per-image indexing and scaling timers are never fed on the two-pass rotation path, because the lattice is forced rather than searched per image and the merge happens outside the loop, so every default rotation run reported "indexing nan scaling nan". A stage that did not run is now simply absent. Measured against the loop's own wall clock on a 18 Mpx dataset: 5% at one worker, 11% at eight, 29% at thirty-two, versus 23x too high before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-01 14:49:45 +02:00
leonarski_fandClaude Opus 5	0ae1a307bc	indexing: complete a rank-deficient direction set, and keep the higher-symmetry setting ... The FFT shortlist could be rank-deficient, and then no cell could be formed at all. FilterFFTResults takes the strongest max_vectors RAW directions and only then prunes ones within 5 degrees of each other, but a single lattice row is sampled by many neighbouring directions of the 16k half-sphere, so thirty raw peaks routinely prune down to four or five distinct directions - the strongest, hence shortest, rows. When a crystal's densest rows share a plane, every surviving direction is coplanar, every triple the reduction forms is degenerate, and the indexer returns nothing. On such a crystal the weak third axis was the eighth distinct direction, at raw rank 78. Keep walking the same magnitude order for up to four more directions that are 5 degrees clear of everything kept, appended after the length sort so the earlier entries hold their positions and the reduction still forms every triple it formed before - the shortlist only gains candidates at its end. That exposed two ways a change of SETTING was mistaken for a different lattice. A centred conventional cell is an exact integer multiple of its primitive one, so the same lattice described two ways differs by that factor: comparing conventional volumes reads a setting change as a sub-cell or a supercell. Both the candidate selection in the rotation indexer and the pass-2 comparison in the driver did exactly that, and between them they discarded a correctly-classified cubic F cell in favour of the body-centred tetragonal description of the very same lattice. Compare primitive volumes in both, as the scheme comparison already did. Fixing the volumes alone was not enough, because the indexed fraction is also biased across crystal systems: a subgroup setting holds fewer cell parameters fixed than its supergroup, so it can never index fewer spots and will always look better by that measure. Where a candidate has a lower lattice point-group order at the same primitive volume - the signature of the same lattice in less symmetry - require it to index markedly better, not merely better, before it displaces the incumbent. A general metric-symmetry promotion was implemented and rejected on evidence: it raised a correct body-centred orthorhombic cell to triclinic and a monoclinic one to C-centred orthorhombic, and no threshold separates the cases, because a false pseudo-orthorhombic degeneracy measured tighter than a true cubic one on obliquity and on alternative-basis axis excess alike. Metric alone cannot decide this; only the intensities can, which is what the space-group search is for. Measured over the 37-crystal regression set: one crystal goes from failing outright to 91% indexed with 91% completeness and a better R_meas than the reference, one keeps the cubic setting it had before, and every other crystal is byte-identical. Full unit suite passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-01 12:59:34 +02:00
leonarski_fandClaude Opus 5	2238290d6d	rugnux: make spot settings reach rotation indexing, and stop over-claiming ... Three small honesty and cost fixes on the two-pass rotation path. Spot-finding settings did not reach the step that determines the unit cell. The two-pass first pass reuses the spots stored in the file whenever it has them, and reuse is the default, so both sampling schemes and the validation loop ran on acquisition-time spots while only the per-image pass saw the command line. Every --spot-* option was therefore a no-op for the lattice search on any file written by this software, silently, and the lattice was cross-validated against one spot set and applied to another. Giving spot settings now implies re-finding them for the first pass as well, and plain reuse says so in the log. The summary printed a space group and unit cell even when nothing indexed. With a zero indexing rate the cell is whatever the lattice search happened to return, no reflection was ever measured on it, and no output file is written - so stating it as the run's answer claims a result the data do not support. Say that no lattice was determined instead. The second pass re-indexes de novo so the cell comes out self-consistent with the post-refined geometry, and its result was already checked against the first pass - once by the supercell test and once by the centring test - but only after every image had been integrated with it, so a disagreement cost a whole extra pass on a dataset that ended up on the first pass's lattice regardless. Compare them at the point the lattice is adopted instead, using the same two tests and the same fallback. A triclinic de-novo cell is left alone, being the demotion the merge reindexes. Measured over the 37-crystal regression set: merge statistics are unchanged on every crystal (the two that move are the known rotation-indexing non-determinism - one observation in 2.9 million, and a zero-score lattice landing on no partials instead of a few). Crystals whose data were already cached in the reference run are unchanged in wall time. The one dataset that was burning a discarded pass went from three passes to two, 509 s to 198 s, against 0.81x for the same-detector dataset that was already running two passes - so about 214 s of the saving is the removed pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 23:13:11 +02:00
leonarski_fandClaude Opus 5	7786fc1af3	Stills geometry refinement: stop sampling once there are enough strong frames ... The first pass sampled min(n, max(refine_frames * 50, 8000)) images to keep the 200 strongest, so any serial run of 10000 frames or fewer indexed every frame TWICE - and 99.3% of the pass was that sampling, the bundle adjust itself taking 0.24 s. The budget is sized for a low-hit-rate dataset; on data that indexes well almost all of it was wasted. Stop once four times the bundle size has been found, which still leaves the "strongest N" selection a real pool and still spans the run, because the sample is equally spaced. On a lysozyme jet dataset that is 1500 frames examined instead of 4000, the same 200 bundled, and the same refined geometry - beam and distance to the pixel, cell to 0.01 A. Warm cache: the pass drops 33 s -> 9.1 s and the whole run 65 s -> 35 s, with CC1/2 and R-meas unchanged inside replicate noise. Stills only - rotation has its own two-pass and returns from this function early. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 09:30:11 +02:00
leonarski_fandClaude Opus 5	111cb2f9a6	Stills: re-check the per-image scale after the partiality refine ... ScaleOnTheFly's collapsed-scale guard ran, and then StillsPartialityRefine re-fitted every crystal's scale with no floor and adopted it unconditionally whenever the image had no prior CC - which is exactly the state the guard leaves behind. So the guard was protecting almost nothing. Measured on a lysozyme jet dataset: of 367 images it left unscaled, only 8 were still unscaled in the output, and 53 reached the merge at or below a fiftieth of the run median, the worst at a 4525th; on a second run of the same sample, 297 images, worst at a 75000th. Those intensities are what the merge saw - up to 94x too high in the written file. Run the guard again on the refined scales. It now reports 367 then 51 on that dataset, and the merge improves: R-meas 117.4 -> 110.8%, CC1/2 96.3 -> 96.5%. Measurements confirm the rest of the guard is right as it stands: 0.02 is ~5x below the lowest scale ever seen on an image that correlates with the merge (no image at CC >= 0.4 falls below a tenth of the median), and leaving the image at G = 1 beats both dropping it and replacing its scale with the median - on a run where 30% of images are affected, dropping costs 1.8 CC1/2 and 29% multiplicity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 23:05:11 +02:00
leonarski_fandClaude Opus 5	a739b156f3	Two-pass: do not adopt a pass whose lattice the reused space group cannot describe ... The second pass re-indexes de novo and can land in a different setting from the first - most often on the PRIMITIVE sub-cell of a centred lattice. The reindex that exists to undo that declines when the metric does not match, and the code then went on to stamp pass 1's group onto the cell regardless. That is not a small error. A C-centred group on an already-primitive cell means the centring absence rule removes half the reflections that genuinely exist, so the merge holds more unique reflections than its own cell can - measured here as "117% complete" with CC1/2 0.62, against the first pass's 92.6% and 0.98, on a cell of exactly half the C-centred volume. Detect the conflict where it happens and feed it to the pass-2 credibility guard rather than acting on it locally: letting the pass re-search its own group instead produced a P1 answer on a crystal XDS and the first pass both call C2, which is a worse outcome than simply not trusting the pass. The completeness and CC1/2 tests stay as the symptom-side net. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 19:52:13 +02:00
leonarski_fandClaude Opus 5	ade61eea60	Rotation: predict without truncating, and keep the better of the two passes ... Digging into the selection logic showed the caps were not deciding the science - the two-pass geometry post-refinement was, and the caps only fed it randomness. Caps. The prediction buffer now grows to whatever a frame predicts instead of keeping an arbitrary subset of it, and the per-image reflection limit is raised to 65536, with the image-buffer transport headroom derived from the same constant so the two cannot drift. Measured: bit-identical output on five battery crystals, because a normal cell never approached the old limits - only a large cell (~2.8e6 A^3, ~30000-44000 predictions per frame) ever did. Pass-2 guard. The refined pass is normally the better answer, which is why it is the canonical output, but it was adopted whatever it produced. On that same crystal it merged more unique reflections than its own cell can hold - completeness "117%", which is arithmetically impossible - while the header- geometry pass sat at 92.6% and CC1/2 0.98. Compare the two and, when the refined pass is not credible, go back to the header geometry and re-run so the canonical files are the ones that are kept. Both bounds are set where only a failure reaches them. Together on that crystal: 111639 unique against XDS's 118730 (was 88000-99000 and different every run), CC1/2 98.0% (was 96.9-97.7%), ISa 8.54, and two runs now agree bit for bit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 19:20:51 +02:00
leonarski_fandClaude Opus 5	8493095912	Viewer: ISa diagnostic plot, and the B-factors in the merge hero row ... The error model says sigma -> b*I for strong reflections, so merged I/sigma flattens off at 1/b - the number reported as ISa. Plotting I/sigma against I with that asymptote drawn on it is what shows whether the reported ISa describes the data or comes from a degenerate fit, which nothing in the window could show before. A third page next to the per-shell plot and table. The merge carries a few thousand strided (I, sigma) pairs to the viewer for it - a shape, not a reflection list; the reflections themselves are in the .mtz/.cif. The hero row gains the Wilson B and the radiation-damage Delta-B. Both were already computed and already in MergeStatistics, so they only needed showing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 17:56:55 +02:00
leonarski_fandClaude Opus 5	a95aca382c	Remove dead code left behind by recent changes ... None of this has a reader: - ScalingSettings::scaling_regularize and its setter/getter - ScaleOnTheFlyResult::succesful (never set) and ::time_s (set, never read), with the timing that only fed the latter - JFJochImage::last_fit_viewport_ (written twice, read nowhere) and the comment claiming the retry uses it - the retry keys off initial_fit_done_ - JFJochDiffractionImage::ice_ring_width_Q_recipA, and a QtConcurrent include in a file that uses none - an unused gemmi::Op accumulator in the spindle-angle helper - <random> in Merge.{h,cpp}, from before the half-set split became a hash - an orphaned comment describing the Ceres B-factor residual deleted in 014e43a4c, and two trailing comments that had collided on one line Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:29:42 +02:00
leonarski_fandClaude Opus 5	f9c8ab4552	rugnux: a run that indexes nothing still writes a process file ... With merging on, the _process.h5 is skipped because the merged reflections are the wanted output and that file is large (113 MB for 200 images here). But if nothing indexes there are no merged reflections either, so the run finished successfully having written no file at all - the one case where the user most needs something to look at. Write it in that case. The per-image messages have already gone past unwritten, so this carries the dataset metadata, the mask, the azimuthal profile and the summary scalars rather than the full per-image tables - and it is small for the same reason it is needed (124 kB on a zero-index run). A run that does index is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:20:32 +02:00
leonarski_fandClaude Opus 5	e123755c7d	rugnux: carry the point-group promotion into the second pass ... The two-pass rotation run reinstates pass-1's space group for pass 2's merge, so pass 2 skips the search block entirely - and with it the flag that records that the Laue class was CHOSEN by the search rather than given. The canonical output therefore printed the plain "no twinning: the Laue class is holohedral, so no merohedral twin law exists", which is exactly the circular conclusion the flag was added to replace; only the throwaway _01 output carried the caveat. The text is written per pass, inside RunPipeline, so the flag has to travel with prepass_merge_sg_ rather than being patched onto the returned result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:54:53 +02:00
leonarski_fandClaude Opus 5	64c8117a96	Space-group search: absences always come from the all-observation merge ... The two-arm search is meant to use the Lorentz-filtered merge for the point group only - systematic absences live in the weak reflections a |zeta| cut removes, and reading them off the filtered arm is what cost four crystals their screw axes. That is what the code comment and CPU_DATA_ANALYSIS both say, but the filtered-arm-wins branch kept its whole result, screws and centering included. Let a search be pinned to a point group decided elsewhere (fixed_point_group) and re-run Stage B on the all-observation merge when the filtered arm rescues the point group. The point group is passed as its symmorphic representative, not by name: gemmi calls both P321 and P312 "32". Reporting that representative also lets the ambiguity check see two arms that disagree about which 2-folds are real - by name they looked identical - and the advice it prints now names a space group -S can actually be given. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:54:02 +02:00
leonarski_fandClaude Opus 5	c10d7ced17	rugnux: build the first-pass analysis engine once, not per image ... The first-pass spot cache constructed an MXAnalysisWithoutFPGA and an AzimuthalIntegrationProfile inside the per-image lambda, so every cache miss allocated a CUDA stream, the preprocessing buffer, the spot finder, the azimuthal integrator and the Bragg engine, used them for one frame, and freed them again - hundreds of times, serially, on the --redo-rotation-spots path. Both worker loops already hoist the same object out of their loop; only this path did not. Build them once for the whole first pass, and only when spots actually have to be found (with --reuse-rotation-spots there is nothing to allocate). Reuse is safe because every azimuthal-integration path - CPU, GPU and the fused adaptive engine - clears the caller's profile before adding to it, so each frame's output is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:38:11 +02:00
leonarski_fandClaude Opus 5	c40e09d458	rugnux: report an ambiguous space group instead of silently picking one ... The two-arm check for "same order, different symmetry" ran after the all-observation arm had already been adopted by value, so it compared that arm against a copy of itself: the point-group names were always equal and the branch was dead. In the other direction the orders were always unequal, so it was dead there too. The AMBIGUOUS warning - written for the case where the two merges support different symmetries of the same order, which implies two different molecular-replacement searches - could never be emitted. Check before the adoption, while both arms still hold their own result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:34:41 +02:00
leonarski_fandClaude Opus 5	157698d31e	Space-group search: report the order of the point group it chose ... The two-arm search compared its arms by the order of the space group each had picked, but Stage B leaves best_space_group unset whenever no candidate is eligible - no candidate had enough observed absences to trust. That is not rare on the Lorentz-filtered arm, and for a systematic reason: the filter removes the badly-measured observations, which is where the weak systematically-absent reflections are. An arm that confirmed 422 but stopped short of naming a space group therefore scored order 0 and lost to an arm supporting P2, and the demotion was logged as "taking the higher symmetry" - the comparison and the message both wrong, in the one direction the design says cannot happen. Carry the point-group order in the result, set from the order Stage A actually adopted, and compare on that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:33:29 +02:00
leonarski_fandClaude Opus 5	8cb1cacadf	rugnux: show the space-group search on rotation data ... The search runs in pass 1 of the rotation two-pass; pass 2 only reuses the group it decided, so the result the CLI renders carried no search at all and the whole report - operator correlations, the space-group candidate table, the refused higher symmetry - was silently dropped on every rotation dataset. Most costly of all, the "or <group> (indistinguishable from these data)" line never appeared, so an enantiomorphic pair the intensities genuinely cannot separate was reported as a single answer. Carry pass 1's search into the returned result. Also de-duplicate the alternatives when the centred-lattice test swaps the metric-matching candidate into the answer: the group it displaced was left out and the chosen one listed twice ("C2 or P21 or C2"). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 08:27:43 +02:00
leonarski_fandClaude Opus 5	25458265d3	Space-group search: ask twice - all observations, and only the well-measured ones ... --search-min-zeta rescues a point group that the full merge cannot confirm, but used on its own it is a trade: on the crystal it was built for it recovers the correct 422, and on four others it costs the space group outright, because discarding 40-80% of the observations starves operator correlations that were perfectly healthy. Both ways of applying it - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade the SAME crystals, so the cut itself is the problem, not where it is applied. Filip's observation makes it one-way: every disagreement between the two is a LOST operator, never an invented one. Discarding observations can starve a correlation; it cannot manufacture symmetry that is not there. So run the search on both merges and keep whichever found MORE symmetry, and the failure mode disappears - each arm rescues the other exactly where it fails. crystal all observations Lorentz-filtered adopted thaumatin (weak) 222 422 422 tetragonal lysozyme 422 222 422 cubic insulin x3 23 2 / 222 23 The filtered merge is used ONLY to rescue the point group. The screw and centering determination always comes from the merge with all the observations, because systematic absences are decided by the WEAK reflections and the filter throws most of them away. Preferring the filtered arm on a tie is not a conservative choice, it is a wrong one: it cost four crystals their screw axes (P2(1) read as P2, P4(1)2(1)2 as P42(1)2) with the point group and every intensity statistic identical - a regression invisible to CC1/2, R_meas and ISa. Where the two find the same ORDER but different symmetry, nothing can prefer one, so the run says so: it names both space groups, states that the data do not decide, reports which one processing continued in, and gives the flag to force the other. Two candidates of the same order imply different molecular replacement searches, and trying both is cheap next to reprocessing - much cheaper than a confident wrong answer. Rotation battery, 33 crystals, both spot finders: fixed-threshold finder 30/33 - ZERO crystals differ from the single search adaptive finder 30/33 - the same three mismatches, gap CLOSED The adaptive finder now matches the fixed-threshold one exactly, which it has not done before: its last remaining loss was the thaumatin set whose 4-fold sits 88.9 deg from the spindle, and it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken). A merohedral twin stays refused in BOTH arms at all three frame ranges where it over-promotes, and at one of them the second opinion is strictly better than shipping behaviour - the full merge collapses to P1 where the filtered one finds the correct H3. Cost is the extra scale-combine-merge on already-ingested partials, with no re-integration: 47.2 s against 47.8 s on the same crystal back to back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 21:11:05 +02:00
leonarski_fandClaude Opus 5	eb70684fa9	rugnux: report how close a symmetry axis lies to the spindle ... A rotation sweep never records the reflections whose reciprocal vector lies within the Bragg angle of the spindle - the blind cusp. Symmetry normally supplies them from an equivalent elsewhere in reciprocal space, so the hole closes. It cannot when a symmetry axis IS the spindle: the cusp is then mapped onto itself, every reflection in it is equivalent only to other reflections in it, and it stays empty however long the sweep runs. The user can fix this at the microscope - re-mount, or add a sweep on another axis - but only if they are told, and nothing in the output mentioned it. Report the smallest angle between any proper rotation axis of the adopted space group and the goniometer axis, always on rotation data, and warn when it falls under 15 deg. The axis is found by projecting onto each operator's invariant direction (the sum of its powers annihilates everything else) and mapping that fractional direction through the refined lattice into the lab frame; the angle is invariant under the sweep, so the reference orientation is enough. Cross-check: this reports 30.6 deg for a crystal whose 4-fold an independent analysis of the XDS orientation matrix put at 30.5 deg. Measured on three rotation sets: 13.6 deg (2-fold, warns), 16.2 deg (2-fold, 99.7% complete) and 30.6 deg (4-fold). The 15 deg bound is practical rather than derived - the blind cone's half-angle is the maximum Bragg angle, ~15 deg for 2 A data at 1 A wavelength - and the wording says what the diagnostic can honestly support: the angle is a risk indicator, the loss is confined to the cone rather than spread over the data, and overall completeness may still look reasonable while the region near the spindle is empty. It does not promise a completeness number, because across those three sets the overall figure does not track the angle (99.7% at 16.2 deg, 92.6% at 30.6 deg). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 17:59:47 +02:00
leonarski_fandClaude Opus 5	3171b071e6	Space-group search: judge a promotion against its parent, not against the error model ... The point-group decision moved with the AMOUNT of data at fixed physics: a partially twinned trigonal crystal was promoted into the twin's holohedry whenever the search happened to see a larger first-pass merge, and kept its true subgroup when it saw a smaller one. Simulation over 6 noise draws with only the merge multiplicity varying: the twin is promoted 0/6 at multiplicity 2 and 6/6 at 18, while the genuine control is promoted 6/6 throughout. The cause is that every existing gate is a ratio to the merge error model - b_parent grows toward the true systematic scatter as sigma shrinks with 1/sqrt(N), while b_cand is already saturated by the twin's disagreement, so the ratio slides down through a fixed veto. The parent statistic moves with data amount and the candidate statistic does not. Gate promotions on the operator disagreement H = <|I1-I2|/(I1+I2)> instead, as the ratio of the operators a promotion ADDS to the parent group's own operators on the same reflections. There is no sigma in it, so it cannot drift with the error model, and the parent normalisation cancels data quality. Measured over 27 runs, 5 promotion types and 450-1800 images: genuine symmetry 0.862-1.219, merohedral twins 1.270-2.084. On the synthetic grid it is flat across a 9x change in multiplicity - genuine pinned at 1.00, twins 3-12x the bound - which is precisely the property the old gates lacked. chi^2 and the systematic-b stay as secondary vetoes; they protect against non-crystallographic pseudo-symmetry, which is where correlation-based scoring is weak. Pick the parent carefully: 422 has two maximal subgroups of order 4, and on a tetragonal crystal twinned by 2[100] the rival (222) is CC-confirmed too and CONTAINS the twin laws, so normalising against it hides the twin among the promotion's own real operators (ratio 8.19 against the true parent, 0.78 against the rival). Where several parents tie, judge on the most damning. Also: - Report a refused promotion instead of silently processing lower. Merging a twin in the twin's holohedry averages non-equivalent reflections into each other and cannot be undone from the output; keeping the subgroup costs only redundancy. The refusal names the group and the number that caused it. - Stop the twinning report from arguing in a circle. It ran after adoption and conditioned on the adopted group, so a promotion into a holohedral Laue class made it print "no merohedral twin law exists" - the test was conditioned on the decision it should audit. Twinning is now also measured on the subgroup merge before adoption, and the post-adoption text says when its own conclusion is not authoritative. - Compare PRIMITIVE cell volumes in the first-pass scheme tie-break. A centred setting's cell is an exact integer multiple of its primitive one (a rhombohedral lattice in hexagonal axes is exactly 3x), so the integer-supercell test fired on a pure setting difference and demoted a good scheme to a threefold-smaller merge - which is what let the twin see the small merge to begin with. Rotation battery, 33 crystals: point-group agreement 30/33 -> 29/33, one crystal moved. That crystal (P422 -> P222) is the one with the known unresolved integration defect where reflections near the rotation-axis plane are wildly mis-integrated; its symmetry mates genuinely disagree, and its lower-symmetry merge is measurably better (ISa 2.72 -> 3.63, high-shell CC 75.4 -> 86.0). The threshold was not moved to accommodate it: 1.25 sits inside the measured gap and widening it would admit real twins. Separately the tie-break improved one crystal's CC1/2 from 77.7 to 84.0. Tests: a synthetic twin-fraction x multiplicity grid, which is what the search had never had - the existing tests are noise-free and exercise only Stage B absences. A NOTE ON WHAT WAS TRIED AND REJECTED, so it is not rebuilt: the obvious "physics-anchored" statistic is the disattenuated cross-validated correlation rho = corr(I_half0(h), I_half1(Rh)) / corr(I_half0, I_half1), which is 1 for real symmetry at any data quality and 2a(1-a)/((1-a)^2+a^2) for a twin. It passes the synthetic grid perfectly and FAILS ON REAL DATA IN BOTH DIRECTIONS - five false refusals of genuine symmetry on the battery, and it waves through a twin (rho 0.998) that H refuses. The reason is that cc_half correlates the two halves of the SAME reflection and so measures only random error, while cc_cross compares DIFFERENT reflections carrying different systematic error; dividing by cc_half removes the noise and leaves a systematic floor that varies by crystal AND by operator. Genuine rho measures 0.9987 on strong data and 0.73 on weak. A synthetic generator validates a statistic's arithmetic, never its premise, and this premise - that the only departure from exact symmetry is noise - is false for every real crystal. Any per-operator agreement statistic needs a same-crystal reference; an absolute threshold on one cannot be made to work by tuning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 21:05:16 +02:00
leonarski_fandClaude Opus 5	16bf3408f0	Address code-review findings; make detection limits detector-driven ... One changeset, developed together in response to a review of this branch, so the files carry several of the changes at once. Full test suite passes (733 cases). Spot finding - Split ImageSpotFinder into Detect() (flag strong pixels - the expensive per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with Run() = both. The per-image min-pix escalation now detects ONCE and repeats only the cheap extraction, instead of re-running the whole finder four times per frame as it did on the default path. It also keeps the winning attempt's spot list rather than re-extracting it, so the frame that is integrated is exactly the frame that was scored - which a GPU re-extract could not guarantee (float atomic ordering). - spot_finding_time_s no longer swallows indexing time, and indexing_time_s now sums every escalation call instead of reporting only the last. Detection limits follow the detector - The azimuthal-integration upper q and the spot-finding high-resolution limit are now std::optional, in the C++ structs AND in the OpenAPI schema, and resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_ recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a pixel outside that q range could never be strong - the integration range silently bounded what detection could see, regardless of the requested resolution limit. Regenerated the C++ and TypeScript clients; the viewer and the web frontend each gained a "to detector edge" switch. Detection defaults are now per workflow (measured, not assumed) - Stills: adaptive detection, min-pix chosen per image, no resolution clipping. - Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit. On a 33-crystal rotation battery, adaptive detection helped four hard crystals but deterministically broke three (a lost space group, a halved indexing rate, a collapsed merge), and the detector-edge limit cost indexing on a strong rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and --no-adaptive-spots is new. Indexer seed escalation - Stop escalating once a seed's lattice explains >= 90% of the seed spots. Previously any frame with >= 80 spots always paid three indexer calls, online broker included. Merge-consistency filter - --min-image-cc gated on a per-image CC computed BEFORE the stills partiality post-refinement and never refreshed; the refiner now recomputes it, so the reported CC describes the data that are actually merged. - Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the merge, the error model and MergeStats can no longer disagree about which images are in (the --scale path merged unfiltered while its statistics were filtered). Per-image B-factor refinement (-B) removed - Measured on four serial-stills datasets: it is a no-op where the per-image fit is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas +23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on 14-25% of images). It had also been silently DISCARDED since the partiality post-refinement landed - reported but not applied. Rather than fix and keep a knob with no demonstrated benefit, the flag and the whole image_scale_b_factor chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs. ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS. (The Wilson per-image b_factor is a different quantity and stays.) Stills partiality width now fits both of its components - sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d* with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2 on d*^2. The angular-only width is fitted over a d*^2-dense population, so it was pinned by the high-resolution edge and collapsed at low d*: median partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55% of them under the merge's partiality floor, and the survivors divided by those values - which inflated the merged low-resolution intensity scale 3.6x (~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x, no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining R-free are all blind to that ramp, which is why it survived earlier validation; the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged). Removed dead code from add-then-remove churn - Prediction-time "still partiality" (unreachable: no setter), the phantom IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the constant it always was), StillsPartialityRefine's caller-less Settings constructor and its reference to a long-gone env var, ProcessImage's unread bool return, an unused include, and a dead viewer overlay hook. Also - Viewer: the magnifier compared a QImage with itself, so its scene rect was set once ever and it could not pan into a larger dataset; the hover tail timer could fire after leaveEvent and resurrect the resolution readout outside the image. - update_version.sh regenerated the frontend lock file BEFORE bumping the version (every release shipped an off-by-one lock), and did git rm/git add on a path that has not existed since the client moved to src/client - with no set -e, both failed silently. - fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a redirect, not a test, so dkms add never ran. - Unit tests for the adaptive-threshold host functions, which had none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 09:07:00 +02:00
leonarski_fandClaude Opus 4.8	9fdeed282a	Add fused GPU adaptive spot finder (azint + spot finding in one pass) ... AdaptiveSpotFinderGPU does the per-resolution-ring reduction once on the GPU and drives both products from it: the azimuthal-integration profile (corrected space) and the self-calibrating adaptive spot-detection threshold (raw counts). This replaces the separate GPU azint pass and the host-side adaptive spot finder that runs on the GPU path today. On a ~4.5 MP detector it does both jobs in ~1 ms/frame versus ~40 ms for the CPU adaptive finder (~42x), with an identical spot list and azimuthal profile. The per-ring threshold math (Poisson tail + read-floored Gaussian, operating point from the false-pixels-per-frame knob) is factored into AdaptiveThreshold.h so the CPU and GPU finders share one source of truth and cannot drift. Wired opt-in via a MXAnalysisWithoutFPGA constructor flag, default on for the rugnux offline path and the interactive viewer, off for the online receiver (so the broker path is unchanged). When on, Analyze() skips the separate azint pass and lifts the profile from the fused engine. The viewer gains an "Adaptive threshold" checkbox that greys out the signal/noise and photon-count sliders (the adaptive finder uses neither). Dedicated tests exercise both products (spot-finding parity vs the CPU finder, azimuthal profile vs a standalone GPU azint) plus a speed benchmark. Validated end-to-end on lysozyme serial stills: fused == CPU-adaptive index rate and merge stats. Docs: new section 3.2 in docs/CPU_DATA_ANALYSIS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 20:10:45 +02:00
leonarski_fandClaude Opus 4.8	014e43a4c9	Remove non-helping stills merge/scaling knobs ... Trims three opt-in stills parameters that did not improve data quality on the external-reference (PDB R-free) battery and only added code: - --partiality-uncertainty: the (1-p)/p merge-sigma term was null on all four serial-stills datasets of the battery vs their reference structures (and neutral-to-harmful at higher coefficients); removed the flag, setting and CorrectedSigma term. - --stills-modulation: the detector-plane flat-field surface was net-negative on flooded data; removed the flag, setting and MergeOnTheFly::RefineModulation (the rotation modulation in RotationScaleMerge is unaffected). - --min-indexed-fraction: every value other than the 0.20 default collapsed CC1/2; removed the override flag/setter, keeping the fixed 0.20 acceptance floor. Default behaviour is unchanged (all three were off / at their default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 17:40:48 +02:00
leonarski_fandClaude Opus 4.8	503bd36738	Apply --min-image-cc merge-consistency filter on the stills merge path ... The per-image CC-to-reference filter (--min-image-cc) was only honoured on the rotation merge; the stills merge added every crystal unconditionally. Extend it to stills so the flag is meaningful there too: on flooded frames that produce many spurious lattices (large-cell serial data), the crystals whose per-image CC to the reference falls below the limit are dropped, keeping only the coherent ones in the merge. Opt-in and default-off (limit 0 -> the loop passes cc_filter=false and the merge is bit-identical to before), so no existing behaviour changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-24 08:41:48 +02:00
leonarski_fandClaude Opus 4.8	1a2b0181a5	Add physical partiality post-refinement for stills (default on) ... Replace the frozen scalar-sigma stills partiality with a physical, refined model. Per crystal, refine an orientation tilt (dpsi_x, dpsi_y) against the running merge and recompute each reflection's partiality analytically from the refined geometry (angular Ewald-proximity model, sigma(d*) = gamma_e*d*), with the per-crystal scale G profiled out by the existing robust IRLS - no re-integration. A soft Gaussian prior on dpsi tames weak-data overfit while staying inert on strong data. The merge <-> refine loop iterates a few times. This is now the stills default via ScalingSettings::stills_partiality_refine (on). A single opt-out flag `--simple-stills` reverts to treating every reflection as a full (p=1, single pass). Retires the experimental `--still-partiality` flag. The viewer gains a "Partiality post-refinement (stills)" checkbox in Scaling settings. Validated (integrate-once / --scale): CC1/2 and R_meas both improve on three monochromatic serial-stills datasets (+2.8 / -10, +5.6 / -3.4, +2.1 / -4); neutral on a pink-beam DMM set (already-full reflections); R-free/R-work down vs a fixed model; competitive with CrystFEL partialator on matched frames. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 14:45:43 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00