195 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	ba24946c0f	twinning: do not report a twin the L-test rules out ... The two indicators can only move one way under twinning - <|L|> down from 0.500 toward 0.375, the second moment down from 2.0 toward 1.5 - but they were combined with an OR, so a narrow intensity distribution could report a twin on its own while the L-test said the opposite. That is not a marginal disagreement. It fires on small-cell rotation data, where a twin fraction of 0.50 was reported for a crystal whose <|L|> was 0.63: a value no twin can produce, and one that points at the opposite situation - a structure whose intensities behave centric. Suppress the flag when <|L|> sits at or above its untwinned value; the two indicators are otherwise combined exactly as before. Measured over 113 stored reports: two flip, both of them wrong today, and nothing else moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 15:34:16 +02:00
leonarski_fandClaude Opus 5	71cca91ec1	rugnux: rank screw candidates without the control count ... A screw's control class is the complement of its absent class on one axial row, so the two move together: a candidate that predicts more of the row absent leaves fewer reflections to be judged against. The Beta tail AbsenceEvidence computes grows with that control count, so the stricter candidate was charged for the very reflections it correctly called extinct, and a group whose predicted-absent class is a strict superset of another's - with the extra reflections equally dead - could score LOWER than the group that explains only part of the row. Two mechanisms, one dataset each. * The control-count minimum gated the row MEAN, which is the scale the zone's evidence is stated in, on the same count as the row MEDIAN, which is a violation threshold. A candidate could therefore forfeit a whole zone by being right: on a tetragonal 4_1/4_3 wedge, thirteen 00l reflections measured at 0.1% of the two l = 4n beside them scored ZERO because only two control reflections were left, while the nine of them a 4_2 also predicts absent scored 38.7. The mean now stands on two, the median still on three. * The Beta tail is replaced, for a SCREW zone only, by its b -> infinity limit - the same statistic with the control count dropped. A p-value computed against each candidate's own null is not one scale across candidates; what is left is the likelihood ratio of the absent class against Wilson at the row's own mean, which is a sum over reflections and therefore comparable. Asymptotically it is n_absent * (log(1/ubar) - 1), so an equally dead superset can no longer score lower. Measured on a tetragonal 4_1/4_3 crystal: 29 dead 00l against 8 control read 47.7 nats where a subset of 19 of them against 18 control read 55.5. The centring statistic is untouched: its control is the whole present population, not the complement of a claim on one row, so neither mechanism applies to it. Measured. Analytic superset monotonicity, on a 176368-point grid with no calibration and no corpus: 1.29% violations -> 0.00%, exactly monotone wherever both candidates keep a control class. Over 112 stored merges (open, in-house and private arms) exactly ONE decision changes and it is a gain, a 4_2 -> 4_1/4_3 swap on a deposited 4_3; the screw column over the deposited arm goes 33/41 -> 34/41 with zero screw over-calls before and after, and the four centring outcomes are bit-identical. An eight-wedge reproducer whose 00l row a strict candidate eats goes from two mis-calls to none, end to end. The refused side moves the safe way: 103 zones the old statistic scored negative go lower, none turns positive, and the minimum evidence of an adopted screw rises 20.4 -> 23.8 against an unchanged bound of 20. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 15:33:29 +02:00
leonarski_fandClaude Opus 5	c25418319a	report: the twinning measured before the search, and the strong-direction limit ... Three additions to the report, none of which changes a number the run computes. The twinning statistics printed today are measured in the Laue class the run ADOPTED, so where the search promoted the point group they can say no more than "no twin law exists inside the class it chose" - and a twin is precisely what would have caused that promotion. The uncontaminated pair was already computed on the subgroup merge the search was given and reached only a log line; it is now printed as L_TEST_MEAN_ABS_L_BEFORE_SEARCH and SECOND_MOMENT_I_BEFORE_SEARCH, beside the post-adoption pair rather than instead of it, since the two answer different questions. ANISOTROPY_D_MIN_BEST names the finest of the three principal limits so it can be grepped, and a paragraph distinguishes the three resolutions the report now carries: what was written, where the isotropic fit crossed, and how far the crystal reaches where it reaches furthest. It is read off the fitted tensor rather than off the cut, so it can land either side of the other two - measured on one dataset at 2.01 A against a 1.81 A cut - and the text points at ANISOTROPY_D_MIN_CENSORED for when it is the edge of the data instead. A warning when CC1/2 falls and then climbs again, which means the fall-off the cut is read off does not describe the data. A climb counts only above CC1/2 0.30 and only if it exceeds 0.05: past that the number is oscillating in noise beyond any cut this run would take. It advises no re-cut - the generous cut is deliberate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 14:06:42 +02:00
leonarski_fandClaude Opus 5	d189faeec7	viewer: draw grid scan cells in the proportion of the scan steps ... A grid scan with, say, a 20 um step in x and a 5 um step in y was drawn as a square grid, so the picture had nothing to do with the shape of the area that was scanned. The cells now carry that proportion. The anisotropy lives only in the view transform: JFJochImage gains a pixel_aspect_ (the drawn height of one pixel in units of its width, 1.0 for a detector image), the initial fit fits the image as if it were that much taller and then puts the factor back into the transform, and the grid view sets it from |step_y / step_x|. Scene coordinates stay one unit per cell, so the mouse mapping, the selected-image box, the pixel labels and the ROI code need no change and the uniform wheel zoom keeps the proportion. Measured on an 8x4 grid driven offscreen: m11/m22 comes out 4.000 for steps 20/5 um, 0.250 for 5/20, 12.333 for 37/3 and exactly 1.000 - the old behaviour - for a square step. Every cell centre still maps back through mapToScene to its own cell in all four cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hcoh6VNrmSswjfeMeDQqWP	2026-08-31 13:38:51 +02:00
leonarski_fandClaude Opus 5	756c3a22bb	docs: confirmed cells for three small-molecule sets; drop the unprocessable CCD one ... The four Diamond I19 small-molecule datasets had no reference of any kind, so a run on them could not be scored at all. Three now have one, each from a published structure with its DOI verified against Crossref; the fourth has a published space group but no numeric cell anywhere, which is recorded as such. Also removes the "obtained but not in the table" section. That dataset is stored as CCD TIFFs the reader does not support, so it is not processed here and adds nothing to a page about what the test corpus contains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 11:02:38 +02:00
leonarski_fandClaude Opus 5	6d37b31948	report: give the resolution the CC1/2 fit reached, not only the one it was cut at ... The automatic cutoff fits the CC1/2 fall-off, takes the crossing of the target (0.30 by default) and then deliberately keeps one more shell. Only the extended limit was reported, as INCLUDE_RESOLUTION_RANGE, and that is the number a reader takes for "the resolution of this dataset" - so the report quoted a limit that is generous on purpose as if it were the measurement. The generosity itself is right and stays: a shell that is included can still be downweighted or dropped by refinement, while one that was truncated cannot be put back. What was missing is the other number. ComputeCCHalfLogisticCutoff now also returns the crossing, ApplyResolutionCutoff passes it out, and the report prints it as FITTED_RESOLUTION beside the range the reflections were written to, with a sentence saying which is which. The log line names both as well. Nothing about the cut, the merge or the written reflections changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 11:02:37 +02:00
leonarski_fandClaude Opus 5	55d9e2536f	release: 1.0.0-rc.166 ... The changelog entry for this release is six lines, not the fifty-four the development log had accumulated: what a user of rc.166 gets is native miniCBF input, masters from other facilities opening, the beam centre measured on every run, a 2theta-swung detector placed where the file says, the unmerged MTZ and a P1 merge written by default, and a substantially reworked symmetry determination. The per-change detail is in the commits. update_version.sh regenerates the three clients from broker/jfjoch_api.yaml, so this also carries the one API description that had drifted from the generated code - fft_high_resolution_A, which sizes the FFT's projection histogram and does not filter spots by resolution. Everything else in the regenerated tree is the version string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 09:11:22 +02:00
leonarski_fandClaude Opus 5	3a30dc8ab5	rugnux: a space group that names twice the cell is a signal to halve it ... C 1 1 2 and C 1 1 2_1 are the only two monoclinic settings whose change of basis from the reference has determinant 1/2, and the reason is physical: with unique axis c the C-centring vector lies wholly in the oblique a-b plane and reduces away. Such a group is never a distinct Bravais centring - it is a primitive lattice written on twice the cell - and it wins the candidate table only because the table is scored on that doubled cell, where half the reflection positions are absent by construction. Refusing to enumerate those settings was measured and is worse: the volume-preserving alternative keeps the by-construction absences as real observations and merges no better than P1. So keep the enumeration as it is and act on the win instead: halve the cell, re-classify in the group's own system, carry the reflections across (the invented positions map to non-integral indices and are dropped, which is what the centred candidate was already doing) and search again there. Measured on one rotation crystal: C 1 1 2_1 on 68.6 73.0 170.3 90 90 90 becomes P 1 2_1 1 on 50.1 170.3 50.1 90 93.5 90, exactly the deposited setting, with every merge statistic unchanged (R_meas 0.1209, CC1/2 0.9979, d_min 1.238) and ISa 18.27 -> 18.31. Stable across +/-0.5 mm of detector distance. Eight other crystals are bit-identical, and across 112 stored battery reports exactly one adopts a volume-doubling setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:19:38 +02:00
leonarski_fandClaude Opus 5	7c3ab72fed	reader: refuse a CBF that is not a detector image instead of misreading it ... Three ways a miniCBF opened silently wrong. A byte-offset CBF with no PILATUS header at all was accepted: Count_cutoff then defaulted to 0, SaturationLimitFromValue(0) is 1, and every pixel at or above one count was flagged saturated - the integration accept gate drops the whole reflection, so the run comes out empty for a reason nothing reports. The pixel size defaulted to 0 with no validation anywhere downstream, which collapses every resolution, every scattering vector and the beam centre in millimetres. XDS writes its correction files in exactly this shape, so this is not hypothetical. A pixel size is now required to claim the file at all, and a missing Count_cutoff leaves the saturation limit unset - falling back to the container's own overflow, which can only fail to call a pixel saturated - with a warning saying so. And the header captures are character classes, not number grammars: "[\d.eE+-]+" matches a bare "." and "(\d+)" matches a digit string too long for int64. std::stod and std::stoll answer both with a raw std:: exception, which escaped the format probe - CanRead catches JFJochException only - so merely LOOKING at a corrupt file threw out of the viewer's open path. A header field that does not parse is now reported as a malformed header. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:40 +02:00
leonarski_f	e0ff51e2a0	lattice: transpose the change of basis to a primitive cell, as everywhere else ... gemmi states centred_to_primitive as an operator on COORDINATES and CrystalLattice::Multiply combines BASIS VECTORS, so the matrix has to be transposed on the way in - as it already is at both of the other places a gemmi Op::Rot reaches Multiply, one of them in this same file. A, B, C, I and F are symmetric matrices, so for them the transpose is a no-op and the omission never showed. R and H are not. Measured on an R-centred hexagonal lattice, ToPrimitive('R') returned 59.5 81.7 43.3 / 145.6 124.5 46.7 where the rhombohedral primitive cell is 49.3 49.3 49.3 / 60.9 60.9 60.9. What hid it is that a determinant is unchanged by transposition, so the VOLUME came out right - and most callers only take the volume. It is not only cosmetic: the result feeds the re-seating path that puts a lattice into a space group the user fixed by hand, so an R-centred lattice was handed the classifier a "primitive" cell that is not that lattice - broken for exactly the centring whose setting most needs re-seating. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit 6ca00e927d664e870515c04164defa81d8a18725)	2026-08-31 07:17:00 +02:00
leonarski_f	527a5187f4	lattice: restore the character that names a centred monoclinic mI reduced form ... ITA character 43 was absent from the table. It is the type-II reduced form of a centred monoclinic lattice with no length equality - mC, mI, mA and mF are one Bravais lattice in four settings, and this row names the one whose conventional cell comes out I-centred. With the row missing such a cell reaches character 44 and is reported as triclinic, losing its centring outright. It accounted for 30 of the 31 demotions left after the two fixes before this one, and for 4.1% of random centred-monoclinic lattices. Both of its conditions are equalities on scalar products - International Tables gives them as 2|D+E+F| = A+B and |2D+F| = B - so the three angle tests are vacuous for it and the second equality is what selects it. cond_2DF was declared in the character struct and never tested against anything, because until now no row used it. Audited over 14000 exact lattices, the row fires 35 times: 32 are centred monoclinic lattices it recovers correctly and 3 are triclinic cells it promotes, all three of which Le Page promotes at the same tolerance. No over-call is attributable to it. The row is taken from International Tables rather than derived - no single integer matrix of determinant 2 covers more than 68% of these cells - and the test checks the conventional cell it produces has two right angles and twice the primitive volume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit 4c2baf0cc3b2177196a9448df06e24b5446de96a)	2026-08-31 07:17:00 +02:00
leonarski_f	931d7acc7c	lattice: judge a structurally-zero scalar product against the size of the cell ... The Niggli type of a reduced cell is the sign of its three scalar products, and gemmi's reduction was asked to decide those signs against an absolute tolerance of 1e-9 while the products themselves are 10^3 to 10^5 A^2 and the cell is held in float. A product that is structurally zero therefore arrives carrying about 1e-4 A^2 of rounding and is read as definitely signed. The reduction lands on the wrong side of the type-I/type-II boundary, the character written for the other side matches nothing, and the lattice comes back with less symmetry than it has. It is not a corner case. Take an exactly body-centred tetragonal lattice with c > a*sqrt(2) and merely ROTATE IT IN SPACE: 38 of 60 rotations lose the 4-fold, and it comes back C-centred. Every lattice this code classifies is a refined, rotated one; the two tetragonal-I cases already in the tests are axis-aligned, which is the one corner where the rounding vanishes. The tolerance is now scaled by the cell's own magnitude. Over 14845 exact lattices that puts it on a plateau three decades wide - demotions 461 -> 31 - with the over-call count unchanged at every point of it, and with the three fixes that follow the plateau is four decades wide and the demotions are zero. The constant is 100x the one Grosse-Kunstleve et al. give because theirs is calibrated for a double-precision cell and this one is float: measured, their value recovers 6% of these lattices and this one 93%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit b82f217719450e1e46e94be5b5a43ed8ffbf2ca4)	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	b31fb0a55c	symmetry: take the point group from whichever merge found it, the absences from all observations ... The search runs on two merges - one holding every observation, one holding only the well-measured ones - and the second was allowed to report but never to decide. That cost real symmetry: over the corpus the two arms disagree on eight crystals and the arm that found the LARGER point group is right on all eight. A promotion is refused by the most damning statistic it can be shown, so only one of the two arms has to be starved for a genuine operator to be thrown away, and which arm that is depends on the crystal. But the two halves of the answer do not come from the same place. Absences live in the weak reflections the filter discards, so adopting the filtered arm whole recovers the point group and loses the screw - a P6_5 crystal comes back as P6. So the arms are split rather than ranked: the point group is taken from whichever supports the larger one, and its absences are then judged here, on every observation. SearchSpaceGroupOptions::fixed_point_group was written for exactly this and had no caller. Measured on the deposited arm: a trigonal-3 answer becomes P 6_1 against a deposited P 6_5 (an enantiomorph pair, so correct), and a monoclinic answer becomes P 2 2 2 against a deposited P 2_1 2_1 2. The screw survives the split, which it did not when the filtered arm was adopted whole. No crystal that was already right moved, and the over-call column is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	d35e8f680f	symmetry: say when the cell metric hosts more symmetry than the group adopted ... Every symmetry under-call in the corpus has the same signature: a lattice whose metric carries rotations the adopted group does not, with nothing in the run saying so. The user is left to notice that a P1 answer sits on a cell whose axes are equal and whose angles are 60 degrees. So a run that determines its own group now compares the two and says what it sees. It decides nothing - no threshold, no promotion, no demotion, no reprocessing - and the message says as much, because a pseudo-symmetric metric is ordinary and only the intensities can settle whether the extra rotations are real. Measured over 83 crystals it fires on 19 and covers 6 of the 7 point-group under-calls; on the crystal whose deposited group is F 4 3 2 and which rugnux reports as P 1, the line reads "the cell metric is cubic - it admits 24 rotations where P 1 has 1". The metric symmetry is Le Page's, taken through GEMMI's implementation, at an obliquity of 1 degree - the middle of the band over which the answer is stable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	493be2cf19	symmetry: enumerate the settings the cell can host, not only the reference ones ... The space-group search offered a candidate only if gemmi calls it the reference setting. A setting is a statement about direction, so that restricted the search to the axes the convention chose: a crystal whose 2-fold lies on c had no rung between P1 and 222 and fell to P1, and one whose screws lie on b and c was reported as the group with a single screw on c - the wrong group, not a lower one, because the candidate that predicts a subset of the real absences and nothing else wins on no evidence at all. Both stages now enumerate more, under refusals rather than thresholds. Stage B offers the non-reference settings of the chosen point group. Their rotation set is equal to the chosen one, not merely contained in it, so this cannot raise the symmetry; what it adds is a screw or a centring on the axis the data show it on. A candidate is offered only if the cell's own metric admits the rotations its setting names, and one predicting exactly the absences another candidate already predicts is dropped as the same hypothesis under a second name. A non-reference candidate whose centring class this merge does not contain is refused outright: the reference path may adopt an untested centring because the caller's centred-lattice re-test backs it, and a non-reference setting has no such backing. Stage A offers the rotation sets no reference setting carries - the a-unique and c-unique monoclinic 2-folds, and the two rhombohedral-axes trigonal groups - and only those, so every point group reachable before is still reached by the same group in the same setting. A rung reached only that way may be ADOPTED but does not judge anything else: it is skipped when the reference chi^2 is formed and when a higher promotion's parents are collected. Without that it made higher promotions strictly harder - a promotion answers to the most damning of its parents, and offering two more order-2 subgroups of 222 refused 222 and 432 on crystals that had them. The run report gains SPACE_GROUP_NAME beside SPACE_GROUP_NUMBER, since the number alone does not say which axes a group's symmetry lies on, and a group that is not a reference setting is printed as its extended Hermann-Mauguin name. The two-arm reconciliation compares point groups by that name rather than by number for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	26fc4b02b3	symmetry: carry the space group as the group, not as its number ... The adopted space group travelled the pipeline as a bare int and was rebuilt downstream with find_spacegroup_by_number, which returns the reference setting. So every setting a number cannot name was destroyed one line after it was determined: P 1 1 2 came back as P 1 2 1, I 1 1 2 as C 1 2 1, R 3:R as R 3:H. DatasetSettings now holds the gemmi::SpaceGroup itself, DiffractionExperiment exposes it as GetGemmiSpaceGroup() / GetSpaceGroupOrP1(), and everything that used to take an int - HKLKeyGenerator (its int constructor is gone, so the compiler finds the callers), the merge, the R-free flags, French-Wilson, the reindexing ambiguity, the completeness enumeration, the MTZ and mmCIF exports, the model validation - takes the group. -S keeps the setting the symbol names rather than reducing it to a number. The end message carries both spellings and a reader prefers the name, since only the name keeps the setting while the number is what a reader written before the name understands. It carries them over CBOR too: the determined group was never serialised at all, so a group rugnux chose reached the master file only when the same process wrote it, and an online writer fell back to whatever the user had supplied at the start. Both keys are optional additions, so an older reader skips them and a newer one reads an older sender. On disk the master's /entry/sample/space_group carries the extended Hermann-Mauguin name and is what the reader takes the group from, so a setting survives a _process.h5 and the --mode scale that re-reads it; the number stays beside it and is the fallback for files written before. Every one of the 230 reference settings the old writer could produce reads back as itself, so older files are unaffected. Stage A and Stage B of the search still enumerate reference settings only, so this determines no group differently today - it is what the enumeration needs before it can be widened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:16:43 +02:00
leonarski_f	6ed4ea541e	reader: a link to a file that is not there is not a dataset that exists ... Every DECTRIS Eiger master links saturation_value, pixel_mask, bit_depth_readout and serial_number into a companion <prefix>_meta.h5, and that file is routinely not kept when a dataset is archived or deposited. The existence test asked only whether the LINK was written, which it is, so every optional-field guard in the reader answered yes and the read that followed threw. A deposited Eiger 16M set could not be opened at all, over values the reader was perfectly prepared to do without. Exists() now asks the second question too - whether the object the link names can be reached - so an orphaned link reads as absent and the fallbacks behind it do their job. The saturation value is then allowed to be missing outright, because on such a file it is: neither the NXmx name nor the DECTRIS one is readable, and there is no third place to look. Left unset, GetSaturationLimit() falls back to the container's own overflow. That is the safe direction - it can only fail to call a pixel saturated, where too LOW a value drops the whole reflection and silently removes the strongest data - and the run says out loud that nothing will be called saturated. The set that could not be opened now processes to 2.17 A against a deposited 2.40 A, in the deposited space group, with a cell agreeing to 0.08%. Output is byte-identical on datasets that already opened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit 0637979f6d95b446406ab70d1f3982841d195b36)	2026-08-30 21:43:54 +02:00
leonarski_fandClaude Opus 5	fb18457e6f	docs: bring the changelog and the method notes up to rc.166 ... The rc.166 changelog was missing fourteen user-visible changes and carried rationale and measurements that belong here instead. Added: the native miniCBF sweep reader, the third-party and firmware-1.x NXmx masters, the plain LZ4 filter, the image orientation taken from the file's module direction vectors, the two beam-centre flags and their rescues, and the FFT reach past 500 A. Trimmed the rest to one line each, moving the numbers out of the user-facing file. RUGNUX.md described the input as a single Jungfraujoch master file, which it has not been since this branch; it now covers the foreign and legacy masters, the accepted compression filters and the miniCBF sweep, including how a sweep is collected from one named frame. Six options existed with no entry in the table - --beam-center-check, --beam-center-search, --fft-min-unit-cell, --min-indexed-spots, --rot3 and --no-p1-crosscheck - and -C now moves both FFT cell bounds, which was not written down anywhere. CPU_DATA_ANALYSIS.md carried two statements this branch made false: 7.5 still said pass 2 reuses pass 1's space group, and 13.1 still said centrings are ranked by net absence count. Both now describe what the code does - the group is determined after pass 2, and centrings are ranked by the same Beta-tail likelihood the screw test uses. Also documents the per-zone screw scoring, the coplanarity volume-fraction guard, the plane-normal transform, the FFT cell bounds and the twelve refined candidates. The miniCBF reader implements the x-CBF_BYTE_OFFSET scheme and reads the imgCIF axis table from the specification alone. No CBF code is vendored or linked, so there is no licence obligation, but reimplementing a published specification carries one of credit: ACKNOWLEDGEMENT.md gains a section and the two algorithms carry a one-line reference each. Both DOIs were resolved before being written. Rottger's initial was wrong where this branch first cited it - K, not A. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 21:15:31 +02:00
leonarski_fandClaude Opus 5	9924dd9fc3	docs: the open test battery, updated - 82 datasets, and not only non-SLS ... The public data the pipeline is exercised on has grown from 59 datasets to 82, 77 of them with a released PDB entry and released structure factors, so the page that credits the depositors and carries the DOI to cite for each is brought up to date with what is actually run. Renamed from NON_SLS_TEST_DATA to EXTERNAL_TEST_DATA, because the old title stopped being true: a few of the sets were collected at SLS beamlines, where the data are still written by someone else's detector and someone else's acquisition system. What the battery tests is foreign files, not a foreign facility. Also rewritten from the current archives rather than the earlier sample: - Multi-collection archives: eleven are not a single continuous rotation, not four. Seven IRRMC archives hold more than one collection; one sweep is kept in six of them, and both are kept in the one whose two sweeps are at different wavelengths. A repository project page is not a reliable guide here - one describes a 900-frame sweep its own tarball does not contain. - Detector labels: 76 rows can be compared against the PDB entry. Seven genuinely conflict, and one of those the file settles outright - pixel count, pixel size, sensor thickness and firmware string agree with an EIGER2 9M against the entry's PILATUS4 4M. A further 29 differ only in how much they state, which is not a conflict. - One archive ships 30 placeholder files named like images that are 64-byte text; named on the page so a reader that globs the directory is not surprised by them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 20:50:10 +02:00
leonarski_fandClaude Opus 5	03456dcddd	calibration: an option to hold the detector tilt fixed ... --mode calibration fits five parameters - beam centre, distance and the two PONI tilts - and a program that cannot express a tilted detector has nowhere to put the last two. Dropping them after the fact is worse than never fitting them: the centre and the distance of a tilted fit have already absorbed the tilt, so the flattened geometry is right nowhere. rugnux --no-refine-tilt, the "Refine detector tilt" tick box on the viewer's Calib page and RingOptimizer's refine_tilt argument hold rot1/rot2 at the value the geometry came in with and fit the remaining three. That is the best flat-detector answer, and the one such a program would refine to itself. Measured on a five-distance calibrant series. At short distance the tilt is real and reproducible - three independent fits agreeing to 0.01 deg, radial rms 1.4 -> 0.4 px - and its direct beam agrees with the background beam-centre estimator to 0.05 px, so the tilted model is the physically right one. The pinned fit then displaces the centre 2.6 px to absorb the tilt and lands within 0.03 px of the same place at every distance. Past ~300 mm, where only two rings reach the detector, the tilt is instead under-determined: it comes out with the opposite sign to every short-distance fit and drags the PONI 28 px while the rms does not move (0.960 against 0.962). The existing degeneracy guard only fires on a strictly single ring, so it does not catch that; declining a tilt that does not pay for itself in rms is left for a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 20:44:12 +02:00
leonarski_f	d5818713cb	rugnux: also merge in P1, so a wrong space group is recoverable ... A rotation run that determines its own space group now also writes <prefix>_P1.mtz: the same observations, the same scaling, merged in P1. If the group was wrong there is no route back today except processing the images again, and the P1 data settle it - re-merge, re-solve or re-refine in any subgroup. Written whether or not the search adopted P1, because a file whose presence depends on what the pipeline decided cannot be harvested by a script. A run given -S writes nothing: its group's centring absences were never integrated, so the P1 merge would be missing whole classes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f # Conflicts: # docs/CHANGELOG.md # rugnux/rugnux_cli.cpp	2026-08-30 14:15:20 +02:00
leonarski_f	123dc22be2	rugnux: write the unmerged MTZ by default ... A run that does not ask for it still has to hand its data to aimless, pointless or careless eventually, and the file it needs is one a user had to know a flag to get. It is the largest a run produces, but it is 3.4 to 5.8 per cent of the images it replaces - and the better it is, the less anyone needs to keep those. The batteries pass --no-export-unmerged: they process the whole corpus and would write a few gigabytes nobody reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f # Conflicts: # docs/CHANGELOG.md	2026-08-30 14:14:28 +02:00
leonarski_fandClaude Opus 5	a27c4cf26f	reader: take a miniCBF's mounting from the imgCIF axis table its header states ... A miniCBF header states three things about how the instrument is put together that the reader was assuming instead: which laboratory direction the image's columns run along, which its rows run along, and which the spindle turns about. Some beamlines append a CBF template block holding the full imgCIF axis table, which says all three outright. Two instruments in the corpus are not what was assumed, in two different ways. One mounts its detector a quarter turn round, so the image's columns run vertically. Another turns its spindle about the VERTICAL, with the image mounted the usual way; its table says so, and its "# Oscillation_axis" line says so a second way, by naming the image direction the spindle runs along rather than a vector. Either error leaves the spindle 90 degrees from the image. That is not a sign, so the run's axis-sign rescue cannot reach it, and no refinement recovers it: all three affected sweeps indexed nothing usable. So the table is read. The element axes give the image orientation, matched against the eight discrete mountings exactly as the NXmx module directions already are - the match itself moves to DetectorOrientation, so both readers share one definition rather than two copies. The goniometer axis with no parent gives the spindle DIRECTION; its sign stays the rescue's business, which is the part a convention can legitimately differ on. The detector axis with no parent gives the 2theta arm, replacing the assumption that the arm shares the spindle's axis - the one header stating both states them with the same vector, so this changes no answer, only what it rests on. imgCIF's frame differs from the internal one by a half turn about x, a rotation and not a mirror, as writer/HDF5NXmx.cpp already records from the other side. Where a header carries no table, a "+SLOW" on the Oscillation_axis line still says the spindle runs along the image's slow direction. That is the only thing one of the three affected sets says about it. The axis NAME on that line stays unusable - the header that carries both says "X.CW" where its own table says Y - but the direction token is not: where both are present they agree, which is what makes reading it evidence rather than a guess. Also: naming a frame with no directory at all now finds its sweep. parent_path() of a bare filename is empty and iterating an empty path finds nothing, so running from inside the data directory reported that no images were found. Measured, with nothing on the command line. The vertical-spindle protein set goes from no usable lattice to 100% indexed, P 6(3) 2 2 with a cell 0.43% from deposited, 87846 reflections at 86.3% completeness and CC(1/2) 0.995. Its companion from the same detector, which has no table and only the +SLOW token, goes from a spurious monoclinic cell at 2.3% completeness and I/sigma 0.21 to the right orthorhombic lattice, 97.7% indexed, 59.7% complete, CC(1/2) 0.996. The quarter-turned set's three sweeps, at three arm positions, now all index without the hand-passed quarter turn they needed and agree on one cell to 0.03 A. Six miniCBF sets that state no table and no +SLOW - including one whose Oscillation_axis line names an axis in a third dialect - are byte-identical in .hkl, .mtz, .cif and the image statistics, as are two NXmx sets, which is the shared orientation matcher moving nothing on that path either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:03:32 +02:00
leonarski_fandClaude Opus 5	dc71cb7299	rugnux: keep the P1 merge, so a wrong space group is recoverable ... A de-novo rotation run merges in P1 to search for the symmetry, adopts a group, and then overwrites that merge with the in-symmetry one. If the adopted group is wrong the user has no route back: every file the run wrote, and every statistic in them, is computed in the group that was assumed, so nothing in the output says the choice was wrong and the only way to a different answer is to process the images again. Merge the same integration once more in P1 after the run's own files are written, and put it beside them as <prefix>_P1.mtz. From it the space group can be re-determined and the data re-merged, re-solved or re-refined in any subgroup - measured end to end on three crystals: POINTLESS reads the file, recovers the group, and AIMLESS re-merges it, reproducing rugnux's own merged intensities at CC 0.9965 where the two agree. On one of the three it recovered the deposited/XDS group where this run had under-called the screw axis. The merge is the full one - correction surfaces fitted, ice rings and near-tangential observations kept, whole resolution range - not the deliberately degraded merge the space-group search itself runs on, and it is what `rugnux --mode scale -S P1` produces from a _process.h5. That route already existed but needs a _process.h5, which a merging run does not write, so it only helped a user who had foreseen the problem. Every de-novo rotation run writes the file, including one whose search concluded P1 and where it therefore repeats the merged output byte for byte. Whether a file exists must not depend on what the pipeline decided: a script harvesting results would otherwise have to reproduce the search's decision to know whether to expect it, and a missing file would not separate "the run chose P1" from "the run failed". A user-fixed -S writes nothing, and that condition is not a pipeline decision. With a group fixed, prediction rejects that group's centring absences (IndexAndRefine.cpp:499-506), so those reflections are never integrated; a P1 merge built from such a run would be missing whole centring classes and would mislead rather than merely be smaller. Cost, median of five paired runs read off the log timestamps (the box is shared, so end-to-end wall time is noise): +0.32 s of 4.2 s, +2.18 s of 31.4 s, +0.19 s of 17.9 s, +0.66 s of 15.9 s - 1 to 8% of a run. The file is 1.6x the merged MTZ on a monoclinic crystal and 30x on a cubic one, where the merged MTZ is tiny; it is well under the unmerged export in every case measured, and 0.02 to 0.6% of the raw dataset. Rotation only for now: the stills merge re-fits per-image scales and per-reflection partialities onto the integration outcomes, and _unmerged.mtz is written from those afterwards, so on stills this extra merge alters a file that is the run's own output. Fixing that means running the cross-check below the unmerged export, which needs the merge lambda hoisted out of its block; deferred, since rotation is what the online pipeline processes. On rotation the merged .mtz, .cif, .hkl, _image.dat, _unmerged.mtz and _unmerged_partials.mtz are all byte-identical with and without this change, including on a run whose search returns P1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 12:36:49 +02:00
leonarski_fandClaude Opus 5	5c44e544dc	reader: place a detector swung out on a 2theta arm where the file says it stands ... Chemical crystallography reaches high angle by swinging the detector out on a 2theta arm. Both readers had the number and neither used it: the miniCBF header's Detector_2theta was parsed into a struct member nothing ever read, and on the NXmx side the rotation was in the depends_on chain, which was not followed at all. A sweep taken at 30 degrees was therefore processed with its detector plane 30 degrees from where it stood, and nothing indexed. The geometry could already express it, and needed no change: the arm turns the detector about the sample, so the distance is still measured along the detector normal and the beam centre is still the point of normal incidence - which is exactly the PONI convention, and a swung detector is one PONI rotation. What moves is the direct beam, by distance*tan(2theta), off the beam centre and often off the detector. NXmx is the harder half, because the swing has no field of its own: it is one rotation in the chain the detector's position depends on, and "two_theta" is only one beamline's name for that dataset. So the chain is followed and its rotations composed, rather than a field of one name being looked for - each transformation states its vector in the frame of the one it depends on, which is why the product is the whole placement. Translations are skipped; they are the distance and the beam centre, which the file states separately in the square-on frame. Vectors come from McStas through the same 180-degree turn about z the module directions already use, a proper rotation, so an axis carried through it turns the same way. The three rotations a file this system writes ARE that chain, and are also read as the PONI angles - so those three paths are skipped, or every tilted file we have ever written would come back tilted twice. That is the one way this change could have broken existing data, and the test for it writes a tilted file and reads it back. For miniCBF the arm turns about the base spindle axis: on the four-circle geometry those headers describe the two are one axis, and the imgCIF axis table such a header carries states them with the same vector. Both now come from one constant, so a later correction to the frame moves them together. Measured. On a swung NXmx sweep the chain gives rot2 = -0.34907 rad for the 20 degrees it states, and the sweep goes from "nothing was integrated" to 25000 reflections at 82.2% completeness and CC(1/2) 0.9993, in the same space group and the same cell to 0.03 A as the square-on sweep of that crystal; the opposite sign indexes nothing. A miniCBF sweep at 30 degrees goes the same way, to 0.585 A, and a second sweep of that crystal at 55 degrees reaches 0.476 A and reproduces the cell again - with a low-resolution limit of 2.36 A rather than 13 A, which is what a detector swung that far records. On all of them post-refinement recovers the header's own beam centre and distance, and the beam stop shadow sits within four pixels of where the swung geometry puts the direct beam, 417 and 537 pixels from where the unswung one does. Seven sets whose detector is square to the beam, three of them carrying a chain whose 2theta is zero, are byte-identical in .hkl, .mtz, .cif and the image statistics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 12:36:46 +02:00
leonarski_fandClaude Opus 5	5f47d73cc1	rugnux: write the unmerged MTZ by default ... The unmerged export is how a run's observations reach the rest of the field. It is what aimless, pointless and careless read, it is what a head-to-head against another program's answer runs through, and handing several of these files to pointless is the only way to merge sweeps rugnux does not combine itself. Behind a flag it reached only the people who already knew the flag existed, which is the shape of defect the "rugnux <file> with nothing else" direction asks to design out. --no-export-unmerged turns it off, and the regression batteries now pass it: the objection to defaulting it on was their disk and time cost, not the product's. Measured on three rotation crystals of 900, 1800 and 3600 frames: the file is 21, 86 and 26 MB - two to five times the merged .mtz, .cif and .hkl put together - and the write costs 0.24, 2.29 and 1.11 s, 5.5 %, 12.2 % and 3.1 % of each run's own wall time. The cost is linear in the number of observations and independent of how long processing took, so it is the largest fraction of the runs rugnux finishes fastest, not of the longest sweeps. A run's .hkl, .mtz and .cif are byte-identical with the export on and off. --export-unmerged-partials stays off: it is a second, larger artefact and a separate question. The viewer's reprocessing dialog ties the export to its merged-output switch, so it appears beside the merged files rather than beside a job that only asked for the per-image _process.h5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 10:51:25 +02:00
leonarski_fandClaude Opus 5	bec2a10a3a	docs: add the eight new public datasets to the non-SLS test list ... Three IUCrData Raw Data Letters (Zenodo, CC-BY-4.0) and five SBGrid Data Bank depositions (CC0) have been added to the data the pipeline is exercised on. Same rules as the rest of the page: the source is the repository and its own citable DOI, every one of which was resolved before it was written down; beamline, resolution, space group and cell are the values deposited with the PDB entry; the detector is read out of the image files. None of the new detectors disagree with their PDB entry. Two of them do not fit the page's one-row-per-sweep shape, so the shape is described rather than flattened. The 6R72 Zenodo record holds two complete 360-degree collections on one crystal - a helical one that produced the deposited structure and a low-dose one that has no PDB entry - and both are listed, sharing a DOI, with the second in the no-PDB-entry table. The three CHESS depositions are 4-11 wedges of 50 degrees per crystal plus a rotation taken with the crystal translated out of the beam, tabulated in a new section. One SBGrid deposition is named but not in the table: its images are 1995 CCD TIFFs, a format the reader does not support, so it is not processed here and saying so is more useful than leaving it out. ACKNOWLEDGEMENT gains the new per-repository counts and a pointer to the Raw Data Letter citations; TESTS points at the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-29 23:40:10 +02:00
leonarski_fandClaude Opus 5	fed077e683	geometry: hold the detector plane as axis vectors, and give the discrete part its own home ... The detector plane was three PONI angles and nothing else, so the two things it cannot express - an image mirrored in Y, and one mounted at a multiple of 90 degrees - had no home at all. They are now the DetectorOrientation carried by the detector setup, composed with the PONI rotation into one orthogonal matrix whose columns ARE the fast axis, the slow axis and the sample->PONI normal: lab = R(rot1, rot2, rot3) * Delta * ( (x-bx)*p , (y-by)*p , distance ) GetFastAxis/GetSlowAxis/GetNormalAxis read those columns and DetectorAxes() sets the plane from them, decomposing back to the angles; PoniRotMatrix and PoniAnglesFromMatrix are the conversion in both directions, exact on the canonical branch (rot2 in [-pi/2, pi/2]) and with a stated convention at gimbal lock. The angles stay stored rather than re-derived, so a geometry given as angles is written back as the same angles, to the bit. Delta is never inferred. In particular an arbitrary rot3 is NOT decomposed into a quarter turn plus a residual: rot3 is a fitted quantity, and a least-squares step must not be able to turn the stored image. It is set only where something states it - the detector setup, --detector-mirror-y / --detector-quarter-turns, or the value a file this system wrote records - and defaults to the identity, which makes the whole change a no-op for every existing detector and every existing file. It is a different setting from DetectorSetup::mirror_y, which flips the MODULE LAYOUT while an image is assembled and so decides what the stored pixels are. Merging the two would apply the mirror twice for every modular detector, or change the pixel content of every file written; both are ruled out. The new one earns its keep exactly where the old one is a no-op: a detector whose image arrives already assembled has no layout to flip. Both generators are signed permutations of the in-plane offset, so they preserve the distance from the PONI. That is why almost nothing downstream changes: everything needing an azimuth already goes through LabCoord, and everything that does not needs only a radius. The two hand-written copies of the rotation - XtalResidual and RingOptimizer - take the discrete part as four constants next to cos_rot3/sin_rot3, since it acts in the detector frame where rot3 acts in the laboratory and cannot be folded into it. RingOptimizer needs it despite being a radial fit: it fits the tilt, and the discrete part changes which way the tilt tips a ring. Carried as two optional CBOR keys and two detectorSpecific datasets, both back-compatible; the NXmx module axis vectors and the translation direction stop being hardcoded and are computed from it, reproducing today's values exactly at the identity. GetPoniRotMatrix is renamed GetDetectorMatrix, because it is no longer only the PONI rotation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5	9d3c2787f8	docs: credit the public diffraction data the pipeline is tested against ... Jungfraujoch is developed at one facility, so the reader and the reduction pipeline are exercised on data collected elsewhere - other detectors, other file formats, other conventions. That data was collected and published by other people, and until now nothing in the repository said so. NON_SLS_TEST_DATA lists all 51 datasets: the DOI to cite for each, the repository it came from, and the experiment as deposited. Beamline, resolution, space group and cell are the values deposited with the corresponding PDB entry, read from the RCSB data API - not results measured here; no quantity produced by this software appears on the page. The detector is read out of the image file instead, because the detector named in a PDB entry is often only approximate, and the twelve cases where the two disagree are listed rather than silently reconciled. ACKNOWLEDGEMENT gains a section for the repositories themselves, with the IRRMC, SBGrid and Zenodo citations and the PDB citation for the metadata. Every DOI on both pages was resolved before it was written down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 21:59:49 +02:00
leonarski_fandClaude Opus 5	84efb0666c	viewer: open a miniCBF sweep, the same way rugnux already does ... The native miniCBF reader added for rugnux is a plain JFJochReader, so the viewer only needed to be told which one to open a file with. JFJochImageReadingWorker held a concrete JFJochHDF5Reader; it now holds both readers and a JFJochReader* pointing at whichever the open file needs, chosen by JFJochCBFReader::CanRead. Naming any frame opens the whole sweep - the reader's own template matcher decides which frames belong to it, so a directory holding two sweeps or XDS auxiliary files is not spliced together. The previous file is closed before the new one is opened, so switching format between HDF5 and CBF in one session leaves nothing behind. Everything the viewer draws already goes through JFJochReader and JFJochReaderDataset, so nothing else had to move. The two HDF5-only features stay on the HDF5 reader: calibration images, and the reprocessing snapshots, which are metadata read back over the same images the reader is serving and have nothing to attach to for a directory of CBFs. A raw sweep therefore shows geometry and pictures with an empty run list, which is the state a live HTTP stream is already in. Reprocessing jobs run on a CBF sweep too - JFJochProcessController opens its own reader the same way, and Rugnux has handled a CBF source since the reader landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 21:49:09 +02:00
leonarski_fandClaude Opus 5	172a845cbb	rotation indexing: refine twelve candidate lattices instead of four ... The first pass refined only min(candidates, 4). The comment at the selection already conceded that the pre-refinement indexed fraction is an unreliable discriminator, and on one dataset degenerate cells occupied three of those four slots - so the right cell, ranked fifth, was never refined. The preceding commit's volume guard frees those slots but does not widen them. This is a one-for-one trade, measured by rebuilding with the old value to confirm the attribution. It gains one dataset: R_meas 27.2% to 23.8%, with mean I over sigma better in EVERY shell at identical shell edges, so it is not a resolution ramp. It costs another: a clean abort becomes a wrong four-fold supercell in P1 at 36% completeness. That is the same shape of trade already accepted for the plane-normal cap search in the preceding commit - an honest failure becoming a wrong answer - on a different dataset. It is committed separately so it can be reverted on its own if the maintainer weighs this instance differently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:35 +02:00
leonarski_fandClaude Opus 5	4e8db41785	indexing: refuse a coplanar candidate, and search the plane normal when the shortlist is flat ... Three related changes to the FFT candidate path, batteried together because they touch the same function. A COPLANAR CANDIDATE REACHED REFINEMENT. ReduceResults filtered triples on lengths and angles only - the 30-150 degree bound admits any flat combination - and there was no volume test. On one dataset 41 of 5535 candidates had |V|/abc below 0.05, with a clean decade gap to the next, and three of them reached the optimizer. UnitCell is float, and for a cell that flat the metric determinant is around 1.5e-7, so float32 gets its sign wrong 19% of the time where float64 never does. The guard against a negative argument to sqrt then CREATES the singularity it was meant to prevent: it puts c in the a-b plane, the reciprocal volume is 1/0, and the residual is 0 times infinity. Ceres reported a not-a-number Jacobian and wrote several hundred lines of solver output per failed solve. VolumeFraction() is |V|/(|a||b||c|), rejected below 0.02 - about 1.1 degrees off flat, ten times below the flattest real candidate observed and a thousand times above where float loses the sign. It is enforced at the producer and at the two optimizer entry points. Note the existing sanity checks use ABSOLUTE volume, which a 320 cubic-angstrom flat cell passes. The same reciprocal-volume division is now guarded at the two remaining sites that share the pattern. A SHORTLIST CONFINED TO ONE PLANE cannot close a cell, and the row it is missing is the plane normal. That is detected from the scatter-matrix eigenvalue ratio - measured, degenerate clouds score 2e-5 to 3.3e-4 against 0.026 or more for every non-degenerate one, a factor of eighty - and one further transform is spent with the same direction count inside a three-degree cap about the normal, so the plan and buffers are untouched. More directions cannot substitute: at the exact true direction the long axis ranks 1422 of 16384 by prominence while the shortlist cut is four times higher. Ranking, not sampling, is the obstacle. A four-fold denser grid was measured and rejected - it reaches the same answer to three decimal places and takes a run from 2.5 to 8 GB of device memory. fft_min_unit_cell_A is reachable as --fft-min-unit-cell and is lowered automatically by -C, mirroring how the maximum is already raised. The default of 10 is unchanged: a lower floor admits spurious sub-cells on protein data, and over 73 protein runs the floor was never lowered while the sibling maximum did fire twice, so the path is live and correctly inert. Corpus of 93 datasets, both arms, one build: 72 bit-identical on report content and p.hkl checksum, 13 failing identically, and the count of working datasets rises by one. The volume guard fires on 58 of 93 and 47 of those stay bit-identical - it fires constantly and almost never changes an answer, which is what it should do. Solver chatter falls from 919 lines across three datasets to none. The cap fires on 4 of 93, none of them in the in-house or private arms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:20 +02:00
leonarski_fandClaude Opus 5	10873eee7a	lattice search: reject an impossible character, and supplement every angle ... Two defects in the Bravais character walk, both of which silently cost symmetry. An impossible character MATCHED. The walk computes acos(cond_F/sqrt(A*B)); when the character is geometrically impossible the argument leaves [-1,1], acos returns NaN, and the acceptance test fabs(NaN - actual) > tol is FALSE - so the character is taken. One corpus crystal matched a monoclinic-C character whose implied cos(gamma) is 1.086. The type-boundary retry covered beta only. All three angles carry the Niggli boundary, and negating two basis vectors supplements the OTHER two, so each angle needs its own flip. A crystal whose reduced gamma sits at 89.900 degrees needs the alpha/beta flip and never got it. Measured on 84369 exact lattices spanning all 14 Bravais classes: nothing is lost in any class, and orthorhombic-I recovery rises from 60.3% to 87.2%. Over 440000 random and perturbed cells the first change only ever demotes a monoclinic-C match to triclinic and the second only ever promotes out of triclinic - it is a strict superset of the beta-only retry. On the 94-dataset corpus, two crystals gain their correct point group and none regresses. Both regression tests fail without the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:41 +02:00
leonarski_fandClaude Opus 5	1ea6ba25be	reader: open a master that states its goniometer axes in one place and their angles in another ... A hybrid master carries an NXmx /entry/sample/transformations group holding one EMPTY SUBGROUP per axis - the direction as a vector attribute, no NX_class, no units, no angles - beside a legacy /entry/sample/goniometer group holding all the angles. Opening it failed outright with "Cannot open HDF5 dataset /entry/sample/transformations/omega": the existing legacy fallback keys off Exists("/entry/sample/transformations"), which is true here, so it never fired, and the axis stub was then opened as if it were the angle dataset. Present is not the same as usable. GoniometerGroup now takes transformations only if it holds at least one DATASET, and ReadAxis asks IsDataSet rather than Exists, so a member that is not a dataset can no longer be read as one. HDF5Object gains that predicate, in the style of the neighbouring Exists. The stub is the load-bearing half, not merely the thing that crashed. In the legacy branch, before falling back to the assumed (-1,0,0), the reader now looks for the NXmx stub and takes its stated vector. With it the axis is (0,-1,0) and the run indexes 60/60 validation frames; with the stub deleted the assumption applies and the same file indexes 0/60 on both schemes and both signs, and the run stops with no lattice. So without this half the fix would have turned "cannot open" into "found no lattice" - a differently shaped failure, not a success. The direction stated here is 90 degrees from the assumption, not merely its negation, which the rotation first pass could have recovered on its own. The vector size check moved out of the first branch so it now covers every path that produces one. Verified: the file processes end to end, 100% of frames indexed, cubic cell 105.87 against a deposited 105.88 (0.009%), space group reported as I23 or I213 - correctly refusing to choose, since the reflections that separate them are extinguished by the I-centring and were never measured. A both-layout master reprocesses unchanged (100% indexed, P212121). [HDF5] passes: 2194 assertions in 91 cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:47:41 +02:00
leonarski_fandClaude Opus 5	6319d5c600	indexing: tell an indexer that failed apart from one that found nothing ... IndexerThreadPool collapsed two different outcomes into the same empty reply. A worker that threw set result = nullptr, and so did a dispatch that never found a free worker; both then became a default-constructed IndexerResult, indistinguishable downstream from an indexer that ran and found no lattice. The first says nothing about the frame at all - the indexer never looked, and it will fail again on the next one - while the second is a real negative result about the crystal. The visible cost was the advice a failed run gave. With the card full, rugnux printed "Indexer thread 0 failed: CUDA (GPU) error" and then ended with "Two-pass rotation indexing found no lattice. Check the beam centre (--beam-x / --beam-y), raise --max-spots ..." - sending the operator to look at geometry that was never wrong, for a machine that was simply out of memory. None of those remedies can help when the frames were not examined. IndexerResult gains an optional error, set by the worker and by the pool's own catch; the remaining nullptr path keeps its meaning of "not attempted" and deliberately carries no error. RotationIndexer records it and exposes GetIndexerError(), and rugnux's first pass branches on it at the throw site, naming the resource failure instead. The error travels as DATA through image_analysis/ rather than as an exception, because RotationIndexer::RunIndexing() is on the online path - IndexAndRefine calls it on a schedule from the broker and the receiver, where dropping a frame is the right failure and killing a live acquisition is not. Only rugnux, which owns the "this run is over" decision, turns it into one. The failure result is byte-identical to the default-constructed one it replaces, so any_executed and the online frame-drop behaviour are unchanged. The new IndexerError category exists because the category is only the display prefix on what() - Category() is read nowhere - and the old line read "Processing failed: Input parameter invalid" for a GPU fault, which is the same defect one layer up. SpotFinderError is the precedent. msg.indexing_result is deliberately left alone: of its three states, "not attempted" is the honest one for a frame the indexer never examined, and asserting false would be the same collapse again. Verified on a squeezed card (246 MB free, -N 4): exit 1, zero dropped frames, the new message, and no mention of --beam-x. On a free card, against the previous binary at -N 6 on the same input, the logs differ only in paths and timing - same space group, cell and merge statistics. Targeted Catch2 cases pass, including three that drive the pool through the online path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:30:28 +02:00
leonarski_fandClaude Opus 5	92ad304f15	rugnux: fail on an exhausted GPU instead of dropping the images it could not process ... A per-image worker caught every std::exception and continued to the next image. That is right for one undecodable image and wrong for a resource fault: a CUDA out-of-memory says nothing about the image that happened to be in flight and everything about the machine, and it recurs on the next one. The run therefore logged an error per image, skipped each of them, and exited 0 with a dataset that was silently short - the merged numbers all moved, and nothing in the exit code or the summary said so. Observed with several processes sharing one card: two arms of the same comparison lost 21 and 15 images, by different amounts, so the arms no longer saw the same data. IsFatalResourceError rethrows for std::bad_alloc and for a JFJochException of category GPUCUDAError or MemAllocFailed, and every per-image catch consults it first. There are ten of them, not the three the obvious grep finds: besides the analyze/integrate/load sites in the two image loops, the first-pass spot read swallows the same way at Warning severity, the pre-scan swallows it in three places, and one geometry-refinement site caught it with no log line at all. The first-pass one matters most - those frames build the reciprocal-space cloud the indexer runs on, so losing them thins the input that decides whether the crystal indexes, and it does not change IMAGES_PROCESSED, so a frame-count audit cannot see it. Every one of these loops joins with future::get() or ParallelFor's RunTasks, both of which hold the first exception and rethrow it at the join, so the throw propagates and nothing terminates. JFJochException gained a Category() accessor for the test. Verified by squeezing the card to 804 MB free and running at -N 4: the run now exits 1 with zero dropped-image lines, where the same test on the previous binary produced the silent skips above. The broker and receiver paths are deliberately untouched - dropping a frame is the better failure there than killing a live acquisition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 14:58:04 +02:00
leonarski_fandClaude Opus 5	dce8d877d0	Scaling: make Run() idempotent, and re-calibrate the promotion it was hiding ... RotationScaleMerge::Run() resumed its alternating-least-squares loop from wherever the previous call left off instead of restarting from the ingested data. A de-novo run calls it four times, so pass N was scaled at 3N iterations, not 3. Two identical calls on the same object produced merges correlating at 0.847 with ZERO bit-identical intensities; they are now 68010 of 68010 identical. Where that mattered is narrower than it sounds, and worth stating exactly. In a determined point group the fit is stationary from iteration 2 - over the battery the collapsed-scale guard drops 377 frames at three iterations and 397 at twelve, and 30 of 39 crystals drop nothing at any count - so production merges were barely touched, 38 of 39 rows character-identical. But the SEARCH merges in P1, where a reflection has two or three observations, and there the fit never reaches a fixed point at all: the same guard drops 417 frames at one iteration and 1101 at three. A factor 1.05 against a factor 2.6, same code. So the defect lived in the one arm whose output the user cannot recover from downstream - the arm that picks the space group. Fixing it moves the statistics the promotion gates are calibrated on. The population as a whole does not shift (median change 0.9-2.7% over 280 matched candidates) but the tails do, p90 by 8-14%, and that is exactly where a promotion is decided. On the two crystals that decide, the H statistic's two populations SWAP ORDER: a genuine tetragonal 4->422 goes 1.685 -> 1.819 while a trigonal 3->32 twin law goes 1.906 -> 1.665. No H bound exists any more that keeps the genuine case and refuses the twin - the noise the loop was adding had been acting as a brake. Of the four statistics only b and R_meas still order the two correctly. Set max_systematic_b_ratio 1.90 -> 1.78, by a rule written down before the population statistic existed: take the statistic with the largest spurious_min/genuine_max on the population its gate actually sees, put the bound at the geometric midpoint. That is sqrt(1.711 * 1.853) = 1.78. Battery 36/39 space groups matching XDS, the same as before the scaling fix and one better than the scaling fix alone; the three mismatches are the known references where XDS is wrong. The refusal now reaches the user by name. The bound's margin is 4% either side - SMALLER than the p90 shift this very commit produced in that statistic. It separates the two crystals that exist and is not robust to another input shift, which is written into the header rather than left to be discovered. Two hazards go with it: re-measuring it must hold the scaling correction surfaces fixed, since changing the iteration count re-rolls their cross-validated on/off decision on 7 of 39 crystals for ~0.3 in ISa, all-or-nothing and non-monotone; and --scaling-iterations stays at 3 on both arms, confirmed rather than assumed (production 2->3 moves <I/sigma> +1.38% and 3->6 only +0.13%; the search gives the identical space group on all 39 crystals at 2 and at 3, and loses one at 1). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 22:44:21 +02:00
leonarski_fandClaude Opus 5	4951538368	rugnux: measure the spot budget instead of taking it as given ... --max-spots was a fixed 1000, and on a rotation sweep it sets the DENOMINATOR of the per-frame acceptance gate, which admits a frame when at least a fifth of its spots index. Detections are not all reflections: on a strongly diffracting crystal with heavy solvent background there are 2372 a frame, 1416 of them on ice bands, and only 178 index - so a budget that takes essentially the whole list puts the entire frame population on the gate (median indexed fraction 0.271, tenth percentile 0.218) and the run integrates 83% of its images. The same run with a smaller budget gains 211 frames and loses none, and the frames it gains are the ones with the MOST detections. That is how a larger budget integrates fewer images. So measure it: over the first pass's validation frames, with the sweep's lattice known, tally each spot by rank as +1 if it lies on the lattice and take the budget at argmax over N of n_indexed(N) - 0.20 * n_counted(N) which rises exactly while spots at that depth index better than the gate's own floor and falls after. The 0.20 is that floor, not a new constant - it is lifted out of the function-local it already lived in. It means: as deep into the intensity-ordered list as the image is still showing reflections of THIS crystal. The argmax alone would not do. Under the null that spots index at the same rate at every depth the tally is a driftless random walk, whose maximum is positive whatever the data, so a bare argmax shortens every dataset. The budget therefore has to clear the walk's own noise: the quantity it acts on is the fall from the peak to the end of the list, which is that walk read backwards, and the reflection principle gives its null law in closed form - P(fall > z*sqrt(g(1-g)T)) = 2(1-Phi(z)). That already pays for the search over ranks, so nothing further is owed to multiple comparisons. One false cut in a thousand measurements - a twelfth of one over a 39-crystal two-pass corpus - fixes z at 3.29. The level was chosen before the rule was written and was not revisited afterwards. Battery, same build, one changed default: 36/39 space groups in both arms, none lost, and 36 of the 39 crystals BIT-IDENTICAL. Two move materially - the strong crystal by +17.4% observations and +35.9 points of CC1/2 at 1.58 A, its indexing rate 83.4 -> 95.1%, and another by +93.5% observations with completeness 76.6 -> 98.4%. The significance requirement is what makes that list clean: it removed the one crystal the unguarded rule regressed, and with it two of the five gains, whose peaks do not clear 3.29 sigma on 60 frames. The lever for those is the sample and not the threshold - power grows as the root of the frame count while the bar stays where it is. Rotation only, and the first-pass lattice search always sees the full list: a fixed --max-spots 250 would have been much cheaper to write and fails a battery crystal outright, starving the de-novo FFT into a wrong cell that indexes 7 of 60 frames. Stills never reach the code path - verified bit-identical merged reflections on a serial set - and --max-spots N still pins it, as does the library default the broker and the FPGA use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 19:02:02 +02:00
leonarski_fandClaude Opus 5	df98bb8789	Space-group search: ignore the frames the crystal barely diffracted on ... A crystal that repeatedly leaves the beam over a full turn was read as P2(1) where it is P4(3)2(1)2, on data that merge at CC1/2 96-98% once that symmetry is assumed. The cell was right, the indexing was right - 83% of images - and every 422 operator correlation still came out between 0.08 and 0.38 against a gate of 0.30. The per-frame scale enters the intensities as 1/G. On this crystal the fitted scale spans 356x between its 5th and 95th percentiles and 15% of frames sit below a tenth of the run median, so those frames arrive amplified ten- to a hundredfold - and their sigmas are amplified by exactly the same factor, so nothing weighted by sigma can see it. What saves the PRODUCTION merge is multiplicity: in the determined symmetry a reflection is measured ten or twenty times and --reject-outliers throws the amplified observation out. The search merges in P1, where a reflection has two or three observations and no majority exists to call any of them an outlier, so the bad value lands in the merged intensity at full weight and the operator correlations pay for it. So drop those frames from the search merge only, beside the |zeta| filter and through the same snapshot that undoes it before the production merge, which keeps every frame. A tenth of the median because that is where the two populations sit: over the 39-crystal battery 33 crystals have NOT ONE frame below it, so the filter is inert on them by construction; of the six that do, five spend 0.1-3.5% of their frames there against this crystal's 15%. A twentieth leaves it under the gate; a fifth starts costing frames the battery says are real, the smallest legitimate min(G)/median(G) measured being 0.070. Battery 35/39 -> 36/39 space groups matching XDS, with 38 of the 39 rows character-identical: the only crystal that moves is this one. On it the operators go to 0.37-0.62, and every one improves in every resolution shell, most at low resolution - the signature of a scale error removed rather than noise removed. The run then merges to 1.41 A instead of 1.67, at CC1/2 97.7% and completeness 99.7%. Four other explanations were measured and refuted first: the search resolution cut (the correlations are flat in resolution, and a battery crystal with a finer detector and no cut at all gets 422 without trouble), the space group being inherited from the pass-1 geometry (handing the refined geometry in from the start changes nothing), the ordering of the corr snapshot around the collapsed-scale guard (0.02 in correlation, nothing in space group), and the ice-ring flagging (removing it entirely leaves the answer at P2(1)). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 17:20:36 +02:00
leonarski_fandClaude Opus 5	9f2696c1ac	Adaptive spot finding: the ring threshold replaces the floor, not the local test ... The self-calibrating finder was meant to replace the classic finder's FIXED PHOTON FLOOR with a per-resolution-ring threshold read off the image's own noise. As written it replaced the local-box SNR test as well, and that is the defect: a whole-ring threshold is an ABSOLUTE contour with no feedback from a pixel's own surroundings, so the area a spot puts above it grows as sigma^2*ln(peak/threshold) and never saturates. Measured on a strongly diffracting rotation set, the detected footprint grows by +8.05 pixels per e-fold of peak, so the brightest reflections came out as 100-500 pixel blobs and were then discarded for exceeding the size bound - every one of the ten strongest on an image. Intersecting with the local box gives -0.24 pixels per e-fold, the classic finder's own number to two decimals. WHY the local box is the right partner, rather than merely the incumbent: it is a prominence rule whose reference level is a 961-pixel mean. A spot inflates the box's own variance and the peak divides out of the acceptance test, so it cuts at a fixed FRACTION of the spot's own height. Referring that level to fewer pixels makes it inherit their shot noise - at FIXED footprint, estimating the level from 961 pixels, from 25, and from the single maximum gives centroid residuals of 0.524, 0.539 and 0.656 - so flat growth and a stable centroid turn out to be two ends of one dial. A contour on the bare maximum has the flattest growth of anything tried (+0.1) and merges worst. The two arms bind in different regimes, which is why intersecting beats choosing: on serial stills the ring threshold is 0.6x the classic floor, on this rotation sweep 2.3-6.0x. Stills are a strict no-op - 175 components against 175, identical per frame - so the +40% in stills indexing that the adaptive threshold was introduced for is untouched. What it buys, stated as one fact rather than two. Across five geometry pins spanning 1.1 mm it indexes the most frames of any arm tried, 0.831 against 0.803, and integrates 3.04 to 5.76% more observations - but those are the SAME number: regressing observation count on indexing rate over four arms leaves residuals of +/-0.7 percentage points against swings of -7 to +4.5%, so the extra observations ARE the extra indexed frames, not better data per frame. CC1/2, the only statistic here carrying per-observation quality, is +0.66 at one pin and -0.06 at the other: not harmed, not improved. <I/sigma>, ISa and R_meas cannot arbitrate on this data - across those pins each crosses zero as a monotone function of the pin. WHY an absolute contour indexes fewer frames, when its spot list is equal or better on every axis measured - recall, top-1000 recall, centroid, ice fraction, component count - is the interesting part, and it is not a detection effect at all: ITS OWN SIZE BOUND DELETES THE BRIGHTEST REFLECTIONS ON THE FRAME. A component is discarded because it grew past 200 px, and it grew past 200 px because it was bright, so the deletions are drawn from the head of the indexing budget rather than uniformly from it: they are 11x enriched in the top 250 of the thousand spots handed to the indexer, and the bound's own real deletions sit at MEDIAN RANK 12. Turning the bound off recovers 66% and 50% of the deficit at the two pins, against a bar registered at 33% before the run. Three of us dismissed this for most of a day on the grounds that the gates delete only ~4% of what is detected. That arithmetic was right and the denominator was wrong - a rate is not an impact when the thing being lost is selected for the property that makes it matter. Reworking the bound instead was measured and rejected: it recovers half the deficit, and it cannot be done without re-admitting what the bound is for - 68 components past 200 px, of which 8 are real and 60 are junk, where the intersect gets the 8 without the 60. The residual once the bound is off, +1.08%/+1.70%, is the contour itself. Component merging is ruled out separately: geometrically impossible here, 33.9 px minimum reflection separation against components spanning 10 px. So is a ranking effect - the intersect's lead runs +0.06% at --max-spots 250, +3.46% at 1000 and +14.26% at 2000, which is backwards for a selection artefact. Costs 0.48 ms per image in the finder, and 0.044 px of bright-spot centroid precision - measured convention-free, by fitting a line to a reflection's own centroid across five frames, after an XDS-referenced figure proved to be four fifths aperture convention. It also makes the compactness gate above it safe. On the absolute contour that gate is net damage, deleting 37 genuine reflections per ten frames; once the footprint stops growing nothing reaches its threshold at all. Also fixes a real but unexercised defect in PoissonThreshold, where the exact tail handed over to a normal approximation with a step. It changes nothing here: the clipped ring sigma is over-dispersed 1.2-4.9x against sqrt(mu) because it still contains diffraction, so the Gaussian arm wins every ring above mu=50 and none of the 522 thresholds move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 15:30:39 +02:00
leonarski_fandClaude Opus 5	844b461232	Spot finding: say honestly that the compactness gate is inert ... The commit that added it claimed the shape test was what made raising the size bound safe - "with a flat 200 alone, two battery crystals moved; with the shape test both are unchanged to three decimals". That is wrong. Those two runs came from different builds, and the comparison never isolated the gate at all. Isolated properly, gate on against gate off in ONE build: byte-identical .hkl on three rotation crystals, the strongly diffracting one included. Not merely close - identical, every digit of every merged reflection. On that crystal the gate rejects 5 to 22 of the 110 to 440 oversize components a frame, so essentially all of what that commit bought, it bought by raising the bound from 50 to 200. Keep it, but for the reason that survives rather than the one that did not. The raised bound admits components up to 200 pixels, and on data carrying ice arcs, cosmic-ray tracks or a lit detector row those are what arrives; none of the crystals measured here carry that population in quantity, which is exactly why the gate reads inert on all of them. So it bounds the SHAPE of what the larger size bound now lets through, and no experiment on these crystals can validate it - an inert gate scores identically whatever its threshold. The comment also carried a "keeps 97% of the genuine wide reflections and none of the ice arcs" that came from a simulation over component lists, not from the pipeline. Removed rather than restated: the pipeline measurement says no reflection anywhere changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 15:04:05 +02:00
leonarski_fandClaude Opus 5	0af109b838	Spot finding: bound how large a spot may be, not how bright ... A component of more than 50 connected pixels was discarded. Under the self-calibrating threshold that is an INTENSITY CEILING, not a size bound: the threshold is an absolute per-ring contour, so a component's area above it grows as sigma^2*ln(A/T), without bound in the peak amplitude. Measured on the strong rotation set, footprints run 3 px at 30-100 counts to 50 px above 10000 - a slope of 4.4 px per ln(peak) against the fixed local-box test's 0.8, which saturates at 13 px and never reaches the bound at all. So the brighter a reflection, the more certainly it was thrown away: 59% of the box finder's d>3 A spots were missing from the adaptive finder's list, including every one of its ten strongest, at an intensity ratio of 1.085 for those that did match. Indexed spots per image collapsed from 220 to 7. Raise the bound to 200 - CrystFEL peakfinder8's --max-pix-count, the only directly comparable number in the field; XDS has no such parameter and guards on shape instead - and ask a component above 50 pixels to be COMPACT: it must fill a fifth of the square its bounding box fits inside. A Bragg reflection is round and fills about half of that square however bright it is; an ice arc, a cosmic-ray track or a lit detector row fills a fifth or less, and those are what an upper bound was ever protecting against. Below 50 nothing is asked of the shape, so every component accepted before still is. Integer arithmetic on both sides, and on the GPU the bounding side fits in what was padding, so the device struct does not grow. The shape test is what makes the raise safe. With a flat 200 alone, two battery crystals moved: one lost a little I/sigma, and the other's de-novo lattice was NOT MONOTONE in the bound - correct at 50, 100 and 200, wrong at 150 and at 250 and above - so 200 was partly luck. With the shape test both are unchanged to three decimals. De novo at defaults on the strong set: indexing rate 0.574 -> 0.804, completeness 97.3 -> 99.6%, <I/sigma> 3.42 -> 6.07, R_meas 0.299 -> 0.249, CC1/2 0.947 -> 0.961, ISa 3.29 -> 4.13. The full 37-crystal battery, scored per shell, is still owed. Also documents what StrongPixelSet::AddStrongPixel has required since the component search became linear - pixels in raster order - and puts the existing test's insertion order into it. Both callers scan a bitmap in ascending flat index and always satisfied it; the test did not, and was the only thing that did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 11:49:49 +02:00
leonarski_fandClaude Opus 5	6516bc96af	Ice rings: carry the list past 1.5 A, where ice does not stop ... The eleven measured bands end at 1.522 A because their source says so in its own words - "pure hexagonal ice has 11 diffraction rings between 4 and 1.5 A resolution" - and its subject was detecting ice in deposited data, not masking it. On a detector that reaches further, the rings it does not list are the ones left in the data: on the strong rotation set just added to the battery, 44% of every image's spots sit in ice bands, and beyond 1.5 A the spot list is ice and nothing else, which is why the resolution estimate read the ice rather than the crystal. There is nothing measured to copy below 1.522 A, so the eight added bands are calculated. Enumerating hkl is not enough and the code already said so: ice Ih is P6_3/mmc with O on 4f, and most of what enumeration emits is extinguished by the OXYGEN SUBLATTICE rather than by the space group - which is why (004) at 1.830 A and (104) at 1.657 A are missing from the measured list although they sit inside its range and its reflection conditions allow them (for (00l) the structure factor goes as cos(2*pi*l*z), and z ~ 1/16 kills l = 4). So compute structure factors - oxygen only, the hydrogens being half-occupancy disordered and weak to X-rays - and keep the lines reaching 3% of the strongest. That rule REPRODUCES THE MEASURED ELEVEN EXACTLY and every line it drops inside their range computes to zero, which is what makes it trustworthy below 1.522 A. It stops at 1.170 A: below that the real lines fall to 2-3% while the extinct ones rise to about 1%, and an oxygen-only calculation cannot separate them honestly. Every added band was independently confirmed in the data - the spot-count histogram of the strong set peaks at each of them and is empty between - and every line the rule calls extinct is absent there too. Costs, measured. The bands are inert above 1.6 A: on 38 of 39 battery sets the profile ice score does not move at all, and the one that appeared to (a jet set, 1.25 -> 2.60) does not on the peak-excluded profile the score actually uses - that was Bragg peaks in the plain profile, which is what the peak exclusion is for. Where a detector does reach past 1.5 A the bands cover more of reciprocal space: unchanged at 1.6 A, +7.4 points at 1.4 A, +16.5 at 1.18 A. On the strong set that is 17% -> 27% of reflections held out of the scale fit, and it shows: the spot resolution estimate improves from 1.33 to 1.46 A against a truth near 1.42, while CC1/2 falls 98.5 -> 97.6% and ISa 3.58 -> 3.37. Ice handling only runs at all on a run that trips the ice gate, so a clean crystal pays nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 11:27:28 +02:00
leonarski_fandClaude Opus 5	fe1f3ba96e	rugnux: report the geometry in the form the broker takes it in ... A rotation run post-refines the detector distance and the beam centre, and that is usually the best measurement of them anyone has - but the only way back into the instrument was to read two numbers off the report by eye and retype them into a collection. Write them once more, as the object jfjoch_broker takes: the four required properties of dataset_settings in broker/jfjoch_api.yaml, spelled the way the API spells them, on one line of valid JSON that a script can lift with a grep and POST. The existing DETECTOR_DISTANCE and BEAM_CENTRE keys are unchanged and stay the ones a person reads. REPORT_VERSION is not bumped: a new key breaks no consumer of the old ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:49:18 +02:00
leonarski_fandClaude Opus 5	42add7c0e2	Report the strong-pixel count off the FPGA path too ... DataMessage::strong_pixel_count is written all the way to CBOR, to /entry/MX/strongPixels, to the receiver plots and to the frontend - and only the FPGA receiver ever set it. Every EIGER dataset, and every rugnux run, stored zeros. It is the one number that distinguishes an image the spot finder gave up on from an image that did not diffract, and its absence is why a defect that cost 767 of 1800 images their spots looked like a crystal that stopped diffracting for two blocks of the sweep. Fill it on the CPU/GPU path as well: the finders know the count, the GPU extractor already had it on the device, and it rides back with the spot count on the frame's one synchronisation, so it costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:46:47 +02:00
leonarski_fandClaude Opus 5	1388b16d7a	Resolution estimate: predict past the edge of the detector ... The spot-finding resolution estimate was clamped so it could never beat the detector corner. On a crystal that diffracts past the corner that reports where the DETECTOR stops, which is the one thing this number is not for - it is meant to say how far a merge of data like these would reach, a property of the crystal and the exposure. The clamp also hid the interesting case: an estimate finer than what the run actually merged is the statement "this run was detector-limited", and there was no way to make it. The statistic already extrapolates. Its quantile sits in the middle of the fall-off, well inside what the detector records, so it goes on measuring the crystal's own decay when the detector cuts that decay short. Measured by truncating the spot lists of 31 battery crystals at an artificial detector edge and scoring the unclamped answer against each crystal's own measured CC1/2 = 0.30 crossing, it holds its 8-9% floor out to about 1.7x past the cut and only then drifts pessimistic, which is the safe direction. Every genuinely detector-limited crystal in the battery needs between 1.10x and 1.63x. Against a truth corrected for censoring - the six crystals whose merge is cut off by their own detector cannot have a measured crossing, so theirs is extrapolated from multiplicity-corrected <I/sigma> and anchored on the 25 where both exist: symmetric-log RMS 13.5 -> 9.4% over 37 crystals, 25 -> 28 within 0.2 A. On the six detector-limited ones 26.1 -> 9.8% and the bias goes +19 -> -3%; on the 31 that are not, 9.23 -> 9.32%, i.e. it costs them nothing. The 0.30 tail fraction and the 2.25 reach were refit by leave-one-out against that truth and did not move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:41:34 +02:00
leonarski_fandClaude Opus 5	ac202a55a1	Spot finding: the strong-pixel limit follows the detector ... An image with 65535 or more strong pixels was given up on and reported ZERO spots - silently, no log line, indistinguishable from a frame that did not diffract. 65535 is one pixel in 64 of the JUNGFRAU 4M the number was written for; left fixed while the detectors grew it became one in 276 of an 18-megapixel EIGER, which a strongly diffracting crystal passes on its best frames. On the strong rotation set just added to the battery it cost 767 of 1800 images: peakCountUnfiltered 0 and resolutionEstimate NaN across two blocks of the sweep, the two where the crystal diffracts hardest. Make the bar one pixel in 64 everywhere, and never below the value that stood here, so no smaller detector loses ground. It lived in three places - the host extractor, StrongPixelSet, and SpotExtractorGPU's buffer capacity - now one function. The bar was there for a reason and raising it alone would not have been safe. sparseccl walks a sliding window of the last two lines and tests every pixel in it, which is quadratic in how many strong pixels a line pair holds: a handful for the silicon-tracker hits upstream wrote it for, four thousand for a lit detector line, and 76 seconds for a fully lit frame. But the pixels arrive in raster order, so the window need not be walked at all - a pixel's earlier 8-neighbours are the one to its left and the at most three above it, which is what the GPU extractor already finds by binary search. Keeping the previous line's range and a forward-only cursor gives the same edge set and the same unions in the same order, so the labels are identical, and the fully lit frame now takes 0.16 s. Verified bit-identical on real frames, on fully dense frames, across occupancy 1e-5 to 5e-2, and on 4000 randomised images including ones with blank lines; SpotExtractorGPU's host-vs-device parity test passes untouched. ImagePreprocessorBufferGPU's gather staging was sized to the old constant, with a comment tying it to the caller's give-up. Raising that give-up without it would have run the gather off the end of the device buffer, so it follows the same limit now. Byte-identical .hkl on three battery crystals that never reach the bar. On the strong set, with symmetry, cell and geometry pinned so only the spot list moves: <I/sigma> better in every resolution shell, CC1/2 97.8 -> 98.5%, R_meas 30.5 -> 28.4%, ISa 3.36 -> 3.58, indexing rate 0.772 -> 0.824. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:38:10 +02:00
leonarski_fandClaude Opus 5	a6fbd1d195	Viewer: Alt and the wheel step through the images ... The wheel already zooms, and with Ctrl or Shift it moves the foreground; Alt now moves through the dataset, one image per notch, wheel up forward - the direction QAbstractSlider's own wheel handling uses, which is what the toolbar's scrub slider follows. The view does not know which image it is showing, so it emits the step and the navigation toolbar applies it: the same loadImage() every other control ends in, with the same clamp and the same Sum setting, so nothing about loading is duplicated. An empty dataset is left alone, because loadImage(-1) does not mean "before the first image" but "the latest one" when the viewer is following a running collection over HTTP. The signal is on JFJochImage, so the other views emit it too; only the diffraction view is connected, which leaves them as they were. Note for a desktop where Alt+wheel does nothing: many window managers grab Alt-modified mouse events before the application sees them, and that is a window-manager setting, not something the viewer can take back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:18:19 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00