194 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	59d92a7238	ParallelSort for the two large sorts on the merge path ... Two whole-dataset sorts sat on the main thread at the end of a rotation run: WilsonOutliers orders every full by resolution, and the unmerged MTZ is put in H K L M/ISYM BATCH order by Mtz::sort(5) - together about 2 s of one thread on a 1.4 M-observation set. ParallelSort (common/ParallelFor.h) sorts one piece per worker and merges them pairwise. It is only for comparators that are a strict total order, where the sorted sequence is unique and the result is the serial sort's bit for bit: WilsonOutliers already breaks ties on the index, and the unmerged writer now sorts the rows itself on the five key columns and then the row number - the order Mtz::sort's stable sort gives - and sets sort_order as it did. An empty table still fails the way Mtz::sort does. md5-identical p.hkl, p.mtz and p_unmerged.mtz; 44.4 -> 43.6 s on a 16M set. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 18:13:41 +02:00
leonarski_fandClaude Opus 5.5	c244d384b0	tNCS: evaluate the class-vector ladder's neighbours in parallel ... Where a pseudo-translation is detected, the class vector is re-refined up a ladder on the whole data by a greedy sweep over the 26 neighbours of the current vector, each a cosine correlation over up to every acentric reflection - seconds of one thread on the main path of the run (3.4 s and 1.5 s for the two calls on a P3_121 16M set). The neighbours of the current vector are now evaluated together; the first improvement in the sweep's own order is taken, and the neighbours after it are evaluated again about the new vector - exactly the moves the serial sweep makes. Each correlation is still one serial sum, so the result is bit-identical. AnalyzeTranslationalNCS takes the thread count. md5-identical output and report; 47.1 -> 44.4 s on that set. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 18:08:36 +02:00
leonarski_fandClaude Opus 5.5	50098d20ba	CPU adaptive spot finder: second local pass only at the ring pixels ... The adaptive finder keeps (local test & ring threshold), so the local test's second pass is only wanted at the ring pixels. Its window there is the first pass's window with the first pass's strong pixels taken out, and those are few: the first pass now records its window sums at the ring pixels, and the second pass subtracts the strong pixels in each window instead of sweeping the whole image again. Integer sums and the same acceptance test (factored into StrongInWindow), so the bits are exactly the dense pass's. CPU-only rotation run of a 16M set: md5-identical output, 262 -> 242 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 17:30:44 +02:00
leonarski_fandClaude Opus 5.5	67ec3649a4	CPU Bragg integration: keep the full-frame scratch between images ... RunImpl allocated the reflection mask (1 byte/pixel) and the owner map (4 bytes/pixel) afresh for every image - 90 MB at 18 Mpixel, above the malloc mmap threshold, so each call paid an mmap, a zero-fill page fault per 4 kB and an munmap (with its TLB shootdown across the other workers). On a CPU-only rotation run of a 16M set that was 79 M page faults and 794 s of system time. Both are now members of the engine, and each call clears the rectangles the previous one wrote before it starts. Contents at every read are unchanged, so the output is byte-identical; CPU-only wall 297 -> 262 s, system time 794 -> 214 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 17:23:12 +02:00
leonarski_fandClaude Opus 5.5	aac1c54fb7	GPU spot finder: warp-per-32-columns local test, second pass only at ring pixels ... The local-box SNR kernel (analyze_pixel) was 55% of all GPU kernel time on a rotation run and held the image loop GPU-bound. Three exact changes: - analyze_pixel is rewritten as one warp per 32 output columns, each lane holding the vertical sums of two input columns in registers and the 31-wide horizontal window taken from warp prefix scans. No shared memory and no block-wide synchronisation; the sums are modular 64-bit integers, so the result is the same bits as before. - The adaptive finder keeps only (local-test & ring threshold), and a pixel's second-pass result depends only on its own window, so the second pass is evaluated at the ring pixels alone (analyze_candidates) instead of densely followed by and_bits. - The first pass is then only read within NBX of a ring pixel, so a warp whose tile no ring pixel can reach skips it; it also no longer reads an all-zero previous-pass buffer. Checked bit for bit against the old kernel on every frame of a 16M rotation run, and p.hkl / p_unmerged.mtz md5-identical on two inhouse EIGER2 16M sets. Total kernel time 19.8 -> 12.1 s, image loop 2.69 -> 1.37 ms/image; wall 44.0 -> 37.2 s and 56.0 -> 48.7 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 17:16:06 +02:00
leonarski_fandClaude Opus 5.5	e9da892790	Rugnux report: low warning thresholds, worded as prompts to check; ice apart from powder ... Owner decision: a warning is a prompt to check and must catch the real cases (9min) at the cost of some spurious ones. The physically motivated corrections stay (<|L|> outside its physical range is not twinning; a single sweep's indexing choice; NO_LATTICE on rotation; the no-crystal report); thresholds raised only to cut noise come back down: - SUPERCELL_POSSIBLE warns wherever the class measures and rocks (as before rc173's audit fix), worded as "check the cell", naming weak ordered intensity of a correct cell and spots of further lattice domains as the other readings. 9min (rock 4.2%) warns again. - LATTICE_TRANSLATION warns on every admitted vector (>=75% of the origin); below 90% the wording names a very strong pseudo-translation as the other reading. - PSEUDO_TRANSLATION warns on every detection; below a 20% peak it is worded as weak, check. - SWEEP_GAPS warns where the degraded ranges cover at least 1% of the sweep (the 4 sets of 77 below that had 1-2 frames, 0.4-0.6% of the sweep). - Powder rings are split between hexagonal-ice positions and the rest (MeasurePowderRings, report-only fields); ICE_RINGS and POWDER_RINGS warn separately from 5% of the spots, and ICE_RINGS also where the merge's ice gate found ice. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 14:02:05 +02:00
leonarski_fandClaude Opus 5.5	8a8c992cca	Rugnux report: warn only where the evidence supports the claim ... Report-layer fixes from the warnings audit of the rc173 full battery. Processing is unchanged (merged MTZ/HKL bit-identical on the sets checked). - SUPERCELL_POSSIBLE stays a key and a summary line; a warning only where the rocking part reaches 20% (sub-cells that refine normally rocked at 2.1-16.2%, the accepted doubled cells at 4.2% and 29.1%). The summary says where further lattice domains may put spots on the half-integer nodes. - LATTICE_TRANSLATION only where the Patterson at the vector reaches 90% of the origin (UNDECLARED_LATTICE_TRANSLATION_PCT, new); the two admitted vectors on the battery read 79% and 85% on crystals refining in the deposited cell, and are now reported as a very strong pseudo-translation. PSEUDO_TRANSLATION warns from a 20% peak (xtriage convention; detected peaks were 8.1% on one set, 24.8-63.2% on the rest). - <|L|> outside 0.375-0.55 (on the adopted or the pre-search merge) is TWINNING_VERDICT= NOT_READABLE with no warning, instead of a twin or SYMMETRY_SUSPECT reading. - NO_LATTICE no longer fires on a rotation run that indexed the sweep with no frame indexed on its own; INDEXING_RATE is --developer on rotation. - A rotation run that finds no lattice writes a report (VERDICT= FAILED, NO_LATTICE) before exiting 1, instead of an input-parameter error and no report (NoLatticeFound). - Powder and Ice summary lines, ICE_* keys, and a POWDER_RINGS flag from a 0.5 spot fraction. - SWEEP_GAPS warns only where the degraded ranges cover 10% of the sweep (77 -> 48 sets). - A single rotation sweep no longer carries an INDEXING_AMBIGUITY warning (one orientation matrix, consistent hand); INDEXING_AMBIGUITY_OPERATORS and an Indexing choice line instead. - A DOMAIN more than 10 deg away is a second crystal; the MULTIPLE_LATTICES sum is documented. - Mosaicity line and docs name XDS's suggested REFLECTING_RANGE_E.S.D. as the comparable number (0.90x median over 34 in-house sets; per-image SIGMAR runs ~1.35x higher). - HARMONIC_CONTAMINATION, SCALING_NOT_CONVERGED, POWDER_RINGS in the documented vocabulary. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 14:02:05 +02:00
leonarski_fandClaude Opus 5.5	a1816e905e	ShadowFinder: add beam-stop holder arms that let part of the beam through ... After the existing mask is complete, a second step takes the pixels it left out that are significantly dimmer than their ring, joins them through a 6 px bridge and across module gaps of any width, and adds a piece whole when it holds >= 2000 dim pixels of which >= 200 are deep. The existing mask is never touched, so a sweep with no such piece keeps its mask bit for bit. Quick subset tests (battery --only, no model check) against the rc173 a0518abe6 full battery: 152 of 233 masks identical; 10 of 10 controls (including the sets where earlier shadow changes regressed through marginal decisions) give byte-identical merges. On the 21 sets that gain a piece the space group never changes, merge outlier rejections fall on 19, R_meas falls and ISa rises on 17, and the shell-scaled model R improves on 16 of 18; a transmitting arm with an over-subtracted background strip is recovered (R_meas 13.3 -> 12.0 %, ISa 13.4 -> 15.5). Known costs: on one set with background bumps at ring radii the low-resolution agreement with the model falls (CC 0.89 -> 0.85) while its internal statistics improve, and two sets cut slightly coarser (1.50 -> 1.55 A, 1.72 -> 1.80 A) with a better model R. Squashed from branch hq-beamstop (cfa080a59..60b6c6822), whose history records the variants tried and dropped: a straight port of the earlier arm finder moved every mask, a two-shape construction filled almost a whole sweep through hole filling, and a deep-fraction floor rejected a real arm that is dim along its whole length. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 14:01:17 +02:00
leonarski_fandClaude Opus 5.5	5a9aca682b	Rugnux: spherical-harmonic crystal-frame absorption as a candidate surface ... A new correction surface, fitted after the time x detector surface and before the goniometer-frame 8x8 grid: log A is a sum of real spherical harmonics (l = 1..6, 48 terms) of the diffracted-beam direction de-rotated into the crystal frame. The incident-beam path depends on phi alone and is in the per-frame scale already. It runs through ApplyCellSurface unchanged in everything but the update: the cells are 32 x 64 equal-solid-angle direction bins, and each round the per-cell sums (ref2, cross, the same damping) become one ridge-regularised Gauss-Newton step on the coefficients (prior width 0.1/l per degree-l coefficient) instead of independent per-cell steps. The half-set Fisher-z gate adopts or refuses it exactly as it does the grids; where it is refused, the grid after it sees what it saw before. Why: the folded 8x8 grid (hemispheres share a cell) is the weak basis for long-wavelength absorption. Offline, held out by unique reflection, this basis lowered held-out scatter 4-8% on 6 of 11 long-wavelength sets where no cell grid did, raised model-phased anomalous peaks 0.02-0.2 sigma, and was neutral on hard-X-ray controls. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 13:43:59 +02:00
leonarski_fandClaude Opus 5.5	23f446acbb	Rugnux: French-Wilson amplitudes as ctruncate gives them ... Compared with ctruncate (CCP4 9, version 1.17.29) on rugnux's own merged intensities of the open battery arm, three differences: - ctruncate gives no amplitude to an intensity below -3.7 sigma (exactly that bound on every set checked; up to 2498 reflections on one set). Rugnux turned them into small, confident amplitudes (F/Fc ~0.3 on the sets whose background is over-subtracted on powder/ice rings). They now get F = NaN (missing in MTZ, '?' in mmCIF); IMEAN is kept. They are also left out of the shell mean that sets the Wilson prior. - The switch to sqrt(I) at I/sigma = 4 left 4-6 sigma amplitudes 2-3% above ctruncate's on every set (1.019-1.028). The posterior now applies up to 20 sigma (emulated: 0.995-1.000). - A shell whose mean intensity is not positive gave a prior at the 1e-10 clamp and amplitudes of ~0 (one set's outer shell); it now takes the nearest lower-resolution shell's mean. The remaining gap (weak amplitudes 2-5% below ctruncate's in the outer shells) is ctruncate's anisotropy-corrected prior; not attempted. An offline R-free ablation put the whole ctruncate conversion at -0.0006 median (15/18 sets better) and dropping its rejected negatives at -0.0009 mean (-0.009 on the worst set). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 13:43:59 +02:00
leonarski_fandClaude Opus 5.5	6b061880d8	Rugnux: bin merged reflections by the cell the completeness denominator uses ... A merged reflection's d came from whichever frame first observed it, while the possible reflections per shell are enumerated from the reference cell, so reflections near a shell edge fell on the other side from their possible twin and shells read 100.1-100.9 % complete (17 of 210 battery sets). The group's d is now computed from the reference cell before the export, and the per-hand table reads the group's d instead of an observation's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 13:43:59 +02:00
leonarski_fandClaude Opus 5.5	02bc0b7e0e	Rugnux: test angle promotions of the metric symmetry; score model frames at the data's best indexing ... The lattice walk admits an angle up to 3 deg from 90 and the constrained refinement then holds it there. The existing two-arm test (promoted vs demoted class, judged on the held-out residual) only covered length equalities (a = b). It now also covers monoclinic/orthorhombic classes: pass 1 reads how far the free refinement leaves a held angle from 90 (AngleEqualityDeparture_deg), turns it into pixels at the far corner, and where that exceeds the integration disc runs the demoted arm on the indexer's free (triclinic) cell. A primitive monoclinic crystal with beta 1.4 deg from 90 was indexed and integrated as primitive orthorhombic on 60/60 validation frames, so the per-frame guard never fired; demoted, the held-out residual halves (1.54e-6 -> 8.16e-7), ISa 5.5 -> 6.1, d_min 3.70 -> 3.37 A, CC1/2 at 3.9 A 0.54 -> 0.76. Unchanged on five controls (three oP/oI, two near-90 monoclinic). Model validation scored alternative cell frames against the data in the merged indexing, before the indexing-ambiguity probe. On a pseudo-orthorhombic P2 crystal whose data need h,-k,-l to match the model, every frame read random (R 0.77-0.84) and noise picked an axis swap into P1121; each frame is now scored at its best over the data's alternative indexings. CC_model 0.23 -> 0.45, R-free 0.66 -> 0.44 on that set. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 13:43:58 +02:00
leonarski_fandClaude Opus 5.5	ae444437f4	rugnux: accept a PDB structure-factor mmCIF as the -z reference ... -z (now also --reference; --reference-mtz still works) takes an SF-mmCIF (e.g. a deposited -sf.cif, gzipped or not) as well as an MTZ. The format is recognised by content: a file starting with "MTZ " is an MTZ, anything else is parsed as CIF. The first merged reflection block with the requested column (or, by default, one the auto choice accepts) is converted to a gemmi::Mtz in memory with GEMMI's CifToMtz and then read by the unchanged MTZ loader, so the in-memory reference is exactly what the MTZ path yields. Unmerged (_diffrn_refln) and anomalous-only blocks are passed over; the log names the block used. The R-free set comes from _refln.status (f -> FreeR_flag 0, o -> 1: the CCP4 convention the loader already reads) and is preferred to _refln.pdbx_r_free_flag, whose convention varies by program; a status column with no 'f' is ignored and pdbx_r_free_flag is used instead. Checked on two open-arm sets in the deposited setting (one with F_meas_au only, one with intensity_meas): reference loaded with the deposited cell and group, the inherited free set agrees with the deposited status 'f' on every common reflection, and the merge correlates at CC 0.99 with the deposition. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 20:17:03 +02:00
leonarski_fandClaude Opus 5.5	19f399a973	Rugnux: write the standard setting, or the user's; match a reference MTZ on its own axes ... The files were written on the axes the space-group search named, which for P2221/P21212 puts the unique axis wherever the a<b<c indexing put it (a 52.51 87.87 137.72 crystal came out P 2 21 21 where XDS writes 87.87 137.72 52.51 P 21 21 2), and a reference MTZ was matched in the data's frame: on permuted axes its free-R flags landed on unrelated reflections while the log reported a high matched count. - New CrystalSetting (scale_merge): changes of basis between settings of one lattice (cell, group, index operator, basis matrix), the {-1,0,1} det +1 candidates (CellMappingOperators, moved from ModelValidation), MetricViolation (moved from Rugnux), ChooseOutputSetting and SeatGroupByAbsences. - Output setting: after every decision the merge, the integrated reflections (unmerged MTZ), the P1 cross-check, the lattice and the _process.h5 reindex matrix are relabelled into the ITA standard setting - or, in priority order, a reference MTZ's, a fitting model's, the -C axis order, a non-standard -S symbol's. Free-R flags are drawn again on the written axes. Reported as SETTING_OPERATOR / SETTING_SOURCE. - Reference MTZ: the group is kept in its setting; after the merge every cell mapping onto the reference cell (times the twin laws) is scored by the reference CC, the best is re-seated and re-merged, and the free flags are inherited only where CC >= 0.5 over >= 50% of the reference range (REFERENCE_MISMATCH otherwise; --mode scale gates the same way). REFERENCE_OPERATOR / _CC / _MATCHED_FRACTION / _FREE_FLAGS_INHERITED. - -S: a fixed group is put on the axes its absences name before merging (SeatGroupByAbsences), fixing -S 18 on a cell whose pure axis is not c. - --model: a model in another setting is now a claim the null tests; where it fits, the data are written in its setting and the validation is remade on those axes (KeepModelVerdict carries the decisions over). Tests: [setting] (synthetic #18/#17/I222/C222/c-unique P21/C2 beta/I2->C2/P1, -C and -S order, absence seating, permuted reference with flags). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 20:16:42 +02:00
leonarski_fandClaude Opus 5.5	6c9205a670	Report the strongest index-2 superstructure class and the cell it doubles to ... The report-only supercell probe now reaches the results report: SUPERCELL_CLASS (parity of the primitive indices), its occupancy and Bragg-like (rocking) part against the lattice's own reflections, its <I/sigma>, and SUPERCELL_DOUBLED_CELL, the Niggli-reduced cell to give with -C. No decision is taken on it: correct cells whose half-integer class is diffuse, and cells whose depositor kept the sub-cell, cannot be told from a real doubling by the data alone. On a crystal with a real doubled axis the reported cell matched the deposited one, and processing on it with -C brought R_free against the deposited model from 0.60 to 0.29. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 08:57:28 +02:00
leonarski_fandClaude Opus 5.5	9e510816e0	Merge hq-cure-merge into hq-pool3; R_MEAS stays the ordinary statistic, the weighted one is R_MEAS_WEIGHTED ... R_MEAS is what users set beside XDS/AIMLESS, so its meaning does not change; the merge-weighted value from hq-cure-merge is reported beside it (R_MEAS_WEIGHTED, REFRES_R_MEAS_WEIGHTED). The per-hand table keeps the ordinary statistic. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 06:05:46 +02:00
leonarski_fandClaude Opus 5.5	2b476336ae	rugnux: report-only supercell probe - the eight parity classes of the doubled cell ... Every lattice decision counts spots or frames, and the sub-lattice's strong reflections win every count, so a crystal whose cell is doubled by a weak superstructure class (9min, 6z9g) is adopted at the half cell. This asks the question in intensities instead. On 60 frames spread over the sweep, after each frame's own integration, the 2a x 2b x 2c supercell of its primitive lattice is predicted to 3 A with the frame's own refined orientation and geometry and integrated on the same engine; the reflections are summed per parity class in two shells (20-5, 5-3 A), with a fit of intensity against partiality (I = a + b p) that separates what rocks like a Bragg reflection from what sits at the node whatever the rocking. Only the tested frames are predicted and nothing is retained, so memory is bounded (the previous prototype predicted the whole run through the merge and ran out of GPU memory). The probe's integrations are kept out of the engine's own counts, which the two-pass stencil guard reads. Results are bit-identical with and without it. REPORT ONLY - it decides nothing, because on the battery it does not yet separate a weak real class from what sits at the half-integer nodes of crystals whose cell is right. Real classes: 6z9g class 101 at 24 % of the lattice's intensity (29 % rocking), 9min 100 at 19 % (4 % rocking - its real class does not rock like the lattice either). On correct cells the largest classes reach 9-12 % raw (7n2s, 9i0a, 7os3) and 3-4 % rocking (7dkp, 7os3), and 7mzt reads 40 % / 19 % on a class the deposition does not have. The log line is the population a decision has to be calibrated on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 06:02:50 +02:00
leonarski_fandClaude Opus 5.5	96b681e50d	PostRefine: commit the geometry when the held-out residual falls by more than its noise ... The joint crystal + detector fit was committed only when the held-out residual fell by at least a fixed 2 %. On a crystal whose geometry is already right that bound cuts through the noise: two builds read 1.9 % and 2.05 % for the same 0.03 px / 0.01 % move (lyso_x06da_half_image), one committed and the other did not, and everything downstream (smoothing window, stretch segmentation) followed the coin. The bar is now the residual's own noise - the standard errors of the two held-out means combined - which is the bar a round of the geometry walk already has to clear (HeldOutResidualFell). The log line carries the joint residual and that noise. Noise on the battery is 2-7 % of the residual, so the gate is about as strict as before but no longer at a fixed edge. Checked on 57 open/in-house sets and 11 private ones against the pool battery. The verdict changed on 8 open/in-house sets besides the passes that follow the 7ris/8pqd lattice change: half_image, 6qaj, lalanine, 8xtf, 9ig7, 8sqt (commit -> reject of a 1-step move) and myob_x06da_split, 6cdl (reject -> commit). Merge statistics are equal to the last digit everywhere except half_image (REFRES R_meas 1.16 -> 0.95, CC1/2 0.79 -> 0.86), 9ig7 (5 fewer rejected observations) and 8xtf, whose stretch disposition followed the coin: multiplicity 20.8 -> 15.6, R_meas 0.91 -> 0.60, R_free 0.193 -> 0.189, REFMAC R_free 0.179 -> 0.176. Private arm unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 06:02:50 +02:00
leonarski_fandClaude Opus 5.5	5c5eb00904	Merge statistics: R_meas weighted as the merge weights each observation ... The delta-CC1/2 disposition keeps a weak stretch in the merge (downgraded) wherever removing it would not raise CC1/2, and the merge then carries each of its observations at the small 1/sigma^2 its scaled-up counting error gives it. R_meas counted those observations at full weight, so the frames that add next to nothing to the intensities set the number. hq-pool battery: 8xtf kept 115 weak frames (scale 0.1-0.2 of the run's) with the same CC1/2, <I/sigma> and CC_model per shell as rc173 with 84 frames rejected, and R_meas was 1.3-1.5x higher in every shell; 9w3y (33 deg rejected -> 0, every model metric better) and lyso_x10sa_strong read the same way. The disposition itself is not segmentation-dependent: conviction is on the batch grid, not the ledger ranges, and those sets' dispositions changed because the corrected data changed. R_meas now weights each observation by its merge weight v = 1/sigma^2 under the error model (corrected_sigma on the host, ModelSigma on the GPU, with the error model of the last MergeAccum), normalised per reflection to Kish's effective count, so equal sigmas give the ordinary formula (WeightedRmeasTerms). Where the proportional term dominates - strong reflections - frames are weighted alike, as in the merge: 8xtf's lowest shell reads 15.0%, the rc173 run with 84 frames rejected 15.2%. The per-hand table is weighted the same way. R_MEAS_UNWEIGHTED / REFRES_R_MEAS_UNWEIGHTED keep the XDS/AIMLESS convention and are what to set beside XDS; the battery scorer records them. MULTIPLICITY stays a count. Effect (weighted / unweighted): lyso_x06da_ref 0.0454 / 0.0479, 8xtf 0.225 / 0.907, insu_I_x06da_ref REFRES 0.072 / 0.232. GPU and CPU paths agree to the last printed digit on lyso_x06da_ref. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 04:52:24 +02:00
leonarski_fandClaude Opus 5.5	94ed6aec04	Scaling: a frame below the credible floor does not hold the loop's step open ... ComputeSmoothGWindow leaves a frame below MIN_CREDIBLE_SCALE_RATIO of the median out of every window, and DropCollapsedScales drops it after the loop, but RunScalingLoop still counted its step. Such a frame has lost its vote in its own references and keeps moving, so the loop was stopped as "unsettled". On 8xtf (hq-pool battery) the first-pass partials loop stopped after 16 rounds at rms dlogG 1.8e-01 (base 9.5e-04); with the frame left out of the step it settles in 23 rounds (8.0e-04), and every other loop of that run that had stopped unsettled (pass 2 at 1.6e-03) now settles too. Every loop of that sweep that stopped unsettled was one that went on to drop a frame. The final 8xtf merge is bit-identical; the false SCALING_NOT_CONVERGED warning goes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 04:44:50 +02:00
leonarski_f	5893a524a5	Merge branch 'hq-errmodel' into hq-pool	2026-09-24 14:36:50 +02:00
leonarski_fandClaude Opus 5.5	aa1f939347	Error model: calibrate each bin on the median NORMALISED deviation, binned by counting I/sigma ... The rotation merge fitted var = a*s2 + b^2*<I>^2 from three separate medians (s2, I2, dev2) per bin of I2. A median of dev2 over observations whose variances differ is not 0.455 times their mean variance, so the ratio of medians read a too low and b too high: on the scaled fulls of 28 sets (in-house, open and private) the core of the normalised deviations scattered at up to 1.8x its stated variance in the weak and middle bins and at 0.1-0.7x in the strongest. Reproduced on synthetic samples with a known model (a 1.3 read as 1.12, ISa 33 read as 31). Now (ErrorModel.h/.cpp, host-only, so the GPU and CPU paths share it): - bins are equal counts in counting I/sigma (I2/s2), where b is identified; - each bin is calibrated on the median of dev2/var with var from the previous iteration, iterated to a fixed point - heterogeneity inside a bin no longer biases it, and the median keeps it robust to tails; - s2 is the counting variance the merge actually applies the model to (rebuilt at the reflection's mean), not the observation's own sigma^2. The separate 6-sigma misfit refit is gone: the median does not need it. A mean-based fit (misfits cut at z^2 > 2 ln N) was tried first: it calibrates the total variance best (median rms log chi2 over the 28 sets 0.14 vs 0.19 here) but on heavy-tailed data it sizes the sigmas on the tails, the merge's outlier test widens with them, and CC1/2 fell 0.80 -> 0.71 on a powder-contaminated set (0.83 with this fit). Rejected for that. Offline, 28 sets: rms log chi2 of the median normalised deviation over (counting I/sigma x resolution) 0.208 -> 0.139 (better on 23), of the mean 0.239 -> 0.191 (better on 22). Battery, 45 of 48 sets against the 9b6736 run (3 lost to CUDA OOM from GPU contention): space group unchanged on all; d_min unchanged except two poor multi-lattice sets (1.69 -> 1.56, 1.96 -> 1.90); ISa x1.13 (median), in-house ISa/XDS 0.73 -> 0.93; ISa*R_meas_lo/0.8 0.93 -> 1.05 (XDS ~1.2); CC1/2 over the XDS range +0.002 (mean; up 0.014-0.031 on the three poorest sets, else +-0.0001); CC_model +0.0011, R_model_shell_scaled -0.0007 (mean over 17 open sets); CC_anom +0.003 (mean). Six private sets: space group, d_min and CC1/2 unchanged, ISa up by 14-67% towards XDS's. Remaining misfit, not addressed: the excess variance grows slower than <I>^2 (the effective fractional error falls 2-2.5x from counting I/sigma 5 to 200), so the strongest reflections still scatter below their sigma on open sets. A third, linear term (as in Aimless) fits it better on most sets but leaves b unidentified on some (b -> 0 on 4 of 28); not landed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-24 14:32:13 +02:00
leonarski_f	c0b7391159	Merge branch 'hq-rotrate2' into hq-pool	2026-09-24 14:06:24 +02:00
leonarski_fandClaude Opus 5.5	e7ef72ba40	Rotation prediction: place each partial at its slice of the rocking curve ... A rotation frame records the part of a reflection's rocking curve inside its own oscillation, and while the crystal turns through the curve the spot walks along its Debye ring. The predictor put every partial at the exact diffracting condition, so a partial recorded on the curve's flank was integrated pixels away from where its flux landed. The walk is largest where the reflection moves nearly tangent to the Ewald sphere (low |zeta|), whose curves are widest. The prediction now turns S about the beam by the rotation's component along the ring times the flux-weighted centre of the frame's slice of the curve (a truncated-normal mean, RockingSlice.h), on the CPU and the GPU predictor alike. 2theta is unchanged; a frame that straddles the condition symmetrically gets no shift. Measured (observed r1 centroid minus prediction, tangential, I/sig > 10): against the slice model correlation 0.93-0.98 before, 0.0 after; residual rms 1.0-1.8 px -> 0.35-0.48 px at |zeta| < 0.3. Emulated capture of low-|zeta| partials at high angle 0.50 -> 0.94 of a 12 px aperture. CPU runs on four rotation sets (400-800 frames): R_meas -0.03..-0.09 pp, ISa +1.2..+1.8, rejected observations roughly halved, space groups unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-24 14:06:24 +02:00
leonarski_fandClaude Opus 5.5	5c28f35f85	Rugnux: report the oscillation the run integrated at; test the rotation-scale fit on a simulated sweep ... OSCILLATION_RANGE was read off the caller's experiment, so a run that adopted a goniometer rotation scale still reported the file's oscillation; it is now scaled by the adopted k (the starting angle is the stage's and stays). The post-refinement logs the range of k over the leave-a-fifth-out folds instead of a ratio to k - 1, which is meaningless near k = 1. PostRefine_RotationScale: partials of a 180 deg sweep simulated at a known stage rate are fitted back to it (0.97 and 1.0, to 1e-3). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-24 11:37:38 +02:00
leonarski_fandClaude Opus 5.5	032bf9fe2b	Rugnux: walk the goniometer rotation scale to its fixed point, decided on the whole sweep ... The pass-1 post-refinement fits the rotation scale k only on the frames the stored angles still track, and a rate error is exactly what stops them tracking the rest: on a sweep whose stage turned ~3 % slow the fit read 0.979 over 94 deg, failed its leave-a-fifth-out test and was thrown away, leaving half the frames unscaled. The pass no longer decides. Between the passes, at the pass-2 detector geometry, the lattice is indexed (index-only probe) under the stored angles and under the fitted k and scored on the validation frames of the whole sweep: share of the spots on the lattice beyond the wrong-spindle null. k is adopted only where it scores higher by more than the binomial noise of the two (ValidationEvidencePrefers - the test the beam-centre arms already used, now one function); the run then integrates and post-refines at k (post-refine-only probe), fits again on top of it and repeats until the next k no longer scores better (WalkRotationScale). The stored angles are the first hypothesis. Measured: 1.000 25.1 %, 0.97874 58.9 %, 0.97041 90.0 %, 0.97006 90.7 % (not significant) -> 0.97041 adopted. Probes restore the experiment, the pass-2 geometry, pass-1 mosaicity and the beam-centre-search flag; a probe opens no beam-centre search. Forced pass-1 results get their axis scaled; the header revert drops the scale. GONIOMETER_ROTATION_SCALE reports the adopted k (SUSPECT = adopted). The leave-a-fifth-out figure stays in the log only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-24 11:34:06 +02:00
leonarski_fandClaude Opus 5.5	a2c2810551	Merge: a clipped observation is no witness in the Wilson test ... The pair rule drops the improbable (higher) member of a discordant pair. If the lower member lost its core to saturation or the mask, its profile estimate of what remained may be low, and a genuinely strong reflection would be replaced by the clipped value - the failure Aimless guards against. Integration now marks a reflection whose signal disk was not fully readable (Reflection::clipped, from the engines' existing full-disk test); the flag is carried through the partials to the fulls (OR over an event's partials, CPU and GPU combine alike), and the Wilson test never counts a clipped observation as a probable witness against a larger one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-24 07:26:42 +02:00
leonarski_fandClaude Opus 5.5	257686c6fd	Merge: Wilson outlier test ... The median outlier test needs three observations, so a reflection measured once or twice (one good observation and one artefact, the usual case after Friedel merging) was never tested, and where there were more, a precise-looking artefact (a hot pixel, thousands of counts) could outweigh mates from weak frames and become the weighted median itself. Every observation of the written merge is now also judged against Wilson statistics beyond 4 A: E^2 = I / (epsilon <I/epsilon>_shell), centric and acentric laws, a bound set by alpha = 0.01 expected false rejections per dataset (ln(2N/alpha); twice that for centrics), the lower confidence limit I - z*sigma (own sigma, z from the same budget) required to exceed it, shells whose <I> is not established at that significance not judged, and the bound widened by the dataset's measured tail scale (peaks over threshold on the well-measured shells; 1 for a Wilson crystal, larger under tNCS/anisotropy). An improbable singleton is rejected; one with company only when most of its reflection's other observations are probable and it disagrees with them. The reflection is the group, or the Friedel pair under -A. Search merges are not tested (epsilon is 1 in P1, and the screw rows are the reflections a wrong epsilon would call improbable). The flags are handed to the device merge as pre-rejections, so CPU and GPU agree. Report: OBSERVATIONS_REJECTED_WILSON= (part of OBSERVATIONS_REJECTED=), and the developer report lists each rejected observation (hkl, d, E^2, image, x, y). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-24 07:26:42 +02:00
leonarski_f	cab91e2663	Merge branch 'hq-prescan' into hq-integ	2026-09-24 07:26:42 +02:00
leonarski_fandClaude Opus 5.5	23e1976e55	Surfaces: compare held-out half-set CC on Fisher's z; fit modulation, time, goniometer frame in that order ... The correction-surface gate averaged the per-shell change of the held-out half-set CC over 20 equal-occupancy shells and adopted a surface on its sign. On the shells where a multiplicative error shows, CC1/2 sits at 0.99-0.9996 and a large reduction of the error moves it in the fourth decimal; the shells beyond the data's reach (up to 12 of 20 at CC ~ 0) each add +-0.01, so the sign was set by noise (measured spread of the mean +-0.001-0.003 against effects of -0.0003..-0.0008). It refused the 24x24 detector surface on four EIGER2 16M sweeps. The change is now averaged on atanh(CC) - the candidate from the hq-bisect investigation. The order of the three overlapping surfaces also decides what is adopted. With the gate fixed, goniometer-frame first took a 5 keV insulin sweep from ISa 31.8 to 20.1 (the time surface then refused; R_meas 7.8 -> 8.9%, CC to the model 0.8307 -> 0.8282) and cost a 0.1 deg-sliced thaumatin sweep R_meas 18.2 -> 19.6%; time x detector first refused the detector surface on the 16 keV thaumatin sweep (ISa 33 instead of 40). Modulation, time, goniometer frame avoids both. Bare runs, --report-resolution at XDS's range, against rc173 (0252880e0): ISa / lowest-shell R_meas thaumatin 16 keV 32.9/.030 -> 39.8/.028, lysozyme 90 deg sweeps 10.1/.059 -> 14.1/.048 and 10.1/.060 -> 13.8/.049, cytochrome c 19.2 -> 20.4 and 17.3 -> 19.3, thaumatin 3.8 keV 13.3 -> 14.8; CC to a deposited model (standard-protein models on 14 in-house sets, the deposition on 13 open sets) within +-0.0045, R_model_shell_scaled within +-0.002 except where d_min moved. 46 sets, one regression: a half-image lysozyme test set, CC1/2 0.878 -> 0.862. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-24 02:13:33 +02:00
leonarski_fandClaude Opus 5.5	19a850cd20	rugnux: measure the X-ray bandwidth from the pre-scan's spot shapes (reported only) ... Per isolated strong spot of the spot-width sample, second moments along and across its radius, with the exact per-spot Jacobians (pixels per radian of 2theta and of the angle across the scattering plane, from the geometry) and the sensor parallax fixed from physics (conversion depth exponential with length L cos(psi), truncated at the thickness, smeared z tan(psi) along the ray's in-plane direction). y = (m_rad - 1/12 - par_u) - (jr/jt)^2 (m_tan - 1/12 - par_v) is linear in (2 jr tan theta)^2 with slope sigma^2; fitted over eight equal-count 20%-trimmed bins, error from 200 seeded bootstrap re-draws inflated by the reduced chi^2, significant at z > 3. The pre-scan logs the FWHM, its standard error, z and chi2 next to the bandwidth the run uses; nothing consumes it, so output is unchanged. The truncated-exponential depth variance moves into SensorAbsorption.h (ConversionDepthVariance_um2), shared with the integrator's parallax_var_px2 (same arithmetic). Catch2: a synthetic spot population painted with 0.45% FWHM bandwidth and 450 um Si parallax recovers 0.449% (z 82); the same spots without bandwidth give 0.06% at z 0.7. On the battery sets (67d49f base, estimate as logged): MicroMAX pink lyso 0.41% z 7.3, thau 0.42% z 6.9, CHESS 7B2 9q41 0.36% z 14.6, ALS 8.2.1 8u0i 0.27% z 5.6; mono controls lyso_micromax_mono, lyso_x06da_ref, 5mln and near misses 9z44, 7orr, lyso_x06da_5keV all below z 3. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-23 20:42:46 +02:00
leonarski_fandClaude Fable 5.1	38bb511d23	Make a first-pass detector tilt no mounting can have prove itself on the spots ... The first-pass rotation fit refines the spindle-perpendicular detector tilt freely, and on a sweep whose seed spots reach only a few degrees of 2theta the keystone that would determine it is a pixel or two. The fit then commits whatever the centroids' own systematics prefer - measured 2.5 deg on one sweep seeded to 11.7 A - and that tilt mispredicts the detector corners by 20-60 px against a 6 px integration disc, collapsing the run from 2.8 A to 6.35 A and biasing the metric-symmetry arbiter to a = b on the way. On synthetic data the same runaway is reproducible: a 1.5 deg detector error on 6 A reflections walks the free fit to 3.9 deg, on 8 A reflections to 37. Three data-driven discriminators were tried first and refuted on the corpus: a blanket restraint (0.13-0.16 A and up to 28% of ISa lost on the crystals whose tilt is largest and real, and in-house runs pinned at a header tilt known to be wrong), the distance's held-out excitation criterion (the rocking angles move by a tenth of their noise whether the tilt is real or not), and a keystone comparison of the fitted and header tilts with the beam free in both arms (236-set battery: ~30 sets moved, one collapsed from P1 to C2, one lost a screw axis; differences of 0.007-0.6 sigma held the header on right and wrong cases alike). For a tilt of a few tenths of a degree the keystone is a fraction of a pixel and nothing in the spots says whether it is real. What separated the cases was the tilt's size: every set the keystone arm moved carries 0.08-0.46 deg, and a survey of 211 corpus sets leaves the file's tilt by more than 0.56 deg on exactly one - a 2theta arm swung out 12.8 deg that its file records as square, which the free fit recovers and the held arm refuses by 90 sigma. So the hardware prior decides whether to ask, and the spots decide. Below 1 deg of walk from the tilt the pass started at the fit is trusted as before, by construction. Beyond it the winning candidate is refined again from its start with the tilt held there - the beam centre taking the shift the tilt is equivalent to, so the header tilt is never paired with a beam fitted beside a refused tilt - and the two are judged as the rounds of one chain already are, on the spots each indexes inside the wide gate: the walked tilt stands only when it leads by more than the count's own noise. A real tilt of degrees has a keystone of tens of pixels over the seed spots and wins outright; an artefact has none and loses on a tie. Held rather than bounded, because a box the fit lands on is the same wrong answer at a smaller size. Decided at the fit, so every pass and arm of a run handles itself and nothing is carried between passes; the unconstrained alternative is re-solved beside a refused tilt too. The chain that drives a candidate to its fixed point becomes a lambda so it can be run on the held candidate. The run logs the walk, what it would have moved the far corner by, the shift the beam took instead and both spot counts; where it was refused the report's REFINED_DETECTOR_TILT is the starting tilt and REFUSED_DETECTOR_TILT what the fit had walked to. Tests: a half-degree tilt is found, kept and not reported as a walk; a walk past the prior is made on synthetic data, and the result's verdict, counts and geometry agree with each other. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-23 19:01:08 +02:00
leonarski_fandClaude Opus 5	541e523f60	Merge branch 'fft-sort-key' into rc173 ... Sort keys computed once, not in the comparator: an FMA-contracted Length() can round differently at different inlined call sites, which breaks the strict weak ordering std::sort needs and lets its partition scan run off the array. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-23 12:46:02 +02:00
leonarski_fandClaude Opus 5.5	4591837888	Fix broker SEGV in FilterFFTResults: sort keys computed once, not in the comparator ... The length sort in FFTIndexer::FilterFFTResults recomputed Coord::Length() inside its comparator. Under LTO with -march=x86-64-v3 (CI and production builds) GCC inlines Length() at several sites in std::__introsort_loop and contracts x*x+y*y+z*z into FMAs differently at each (fma(x,x,y*y) for the element keys, fma(y,y,x*x) for the pivot recomputed after a swap). The same element's key then differs by one ulp between comparisons. Shortlist lengths come from quantised FFT bins, so exact ties are common on noise frames; on such a tie the unguarded partition scan passes its sentinel and runs off the index array. Evidence: - rc.172 jfjoch_broker disassembly: pivot key after swap at 0x925a83 is rounded differently from the scan keys. - The deployed rc.172 introsort, called directly on finite golden-spiral directions x binned lengths, faults at binary +0x5259ca (the journal's crash address) in up to ~0.5% of sorts. No NaN is involved; the earlier NaN guards could not help. - New test FFTIndexer_ManyNoiseFrames: unfixed rc.172 built with the CI flags (-march=x86-64-v3 -flto=auto) segfaults in the same introsort from FilterFFTResults on the GPU FFT path (noise frame 1632), 3/3 runs; passes with this fix. A non-LTO build keeps Length() out of line and cannot crash, which is why the suite never caught it. Keys are now precomputed and sorted with stable_sort. The same pattern was fixed in SpindleBlindFraction (broker-reachable, 62 lattice rows, |a|=|b| ties) and LePageLattice::PlaneBasis (rugnux, v/-v exact ties); tie order in the latter may change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-23 12:43:52 +02:00
leonarski_fandClaude Opus 5	d5f7e95d41	Scaling: the correction surfaces leave out observations on ice rings ... Ice-ring reflections are flagged and "excluded from scaling, kept for merging", but ApplyCellSurface never read the flag. While the fit dropped Is <= 0 this was mostly hidden: the over-subtracted half of the on-ring observations went with the filter. Once the negatives were admitted (1d1a40d90), a set with strong, azimuthally uneven rings fitted its surfaces to the rings: with the surfaces off the error model's b is 0.018, and the surfaces no longer took it down. On the two in-house lysozyme sets with heavy ice (25% of reflections flagged): ISa 7.57 -> 10.08 and 7.08 -> 10.16, b 0.0155 -> 0.0090; on a third, 3.57 -> 4.37 (4.41 before the filter was removed). A run where no ice is detected sets no flag and is unchanged. The remaining gap to the filtered surfaces (ISa ~15) and to XDS (~20) is not ice: extending the flag to the measured powder rings moved nothing, and a symmetric residual cut and a variance rebuilt at the reference did not either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-22 10:00:47 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00