1175 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	d35e8f680f	symmetry: say when the cell metric hosts more symmetry than the group adopted ... Every symmetry under-call in the corpus has the same signature: a lattice whose metric carries rotations the adopted group does not, with nothing in the run saying so. The user is left to notice that a P1 answer sits on a cell whose axes are equal and whose angles are 60 degrees. So a run that determines its own group now compares the two and says what it sees. It decides nothing - no threshold, no promotion, no demotion, no reprocessing - and the message says as much, because a pseudo-symmetric metric is ordinary and only the intensities can settle whether the extra rotations are real. Measured over 83 crystals it fires on 19 and covers 6 of the 7 point-group under-calls; on the crystal whose deposited group is F 4 3 2 and which rugnux reports as P 1, the line reads "the cell metric is cubic - it admits 24 rotations where P 1 has 1". The metric symmetry is Le Page's, taken through GEMMI's implementation, at an obliquity of 1 degree - the middle of the band over which the answer is stable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	493be2cf19	symmetry: enumerate the settings the cell can host, not only the reference ones ... The space-group search offered a candidate only if gemmi calls it the reference setting. A setting is a statement about direction, so that restricted the search to the axes the convention chose: a crystal whose 2-fold lies on c had no rung between P1 and 222 and fell to P1, and one whose screws lie on b and c was reported as the group with a single screw on c - the wrong group, not a lower one, because the candidate that predicts a subset of the real absences and nothing else wins on no evidence at all. Both stages now enumerate more, under refusals rather than thresholds. Stage B offers the non-reference settings of the chosen point group. Their rotation set is equal to the chosen one, not merely contained in it, so this cannot raise the symmetry; what it adds is a screw or a centring on the axis the data show it on. A candidate is offered only if the cell's own metric admits the rotations its setting names, and one predicting exactly the absences another candidate already predicts is dropped as the same hypothesis under a second name. A non-reference candidate whose centring class this merge does not contain is refused outright: the reference path may adopt an untested centring because the caller's centred-lattice re-test backs it, and a non-reference setting has no such backing. Stage A offers the rotation sets no reference setting carries - the a-unique and c-unique monoclinic 2-folds, and the two rhombohedral-axes trigonal groups - and only those, so every point group reachable before is still reached by the same group in the same setting. A rung reached only that way may be ADOPTED but does not judge anything else: it is skipped when the reference chi^2 is formed and when a higher promotion's parents are collected. Without that it made higher promotions strictly harder - a promotion answers to the most damning of its parents, and offering two more order-2 subgroups of 222 refused 222 and 432 on crystals that had them. The run report gains SPACE_GROUP_NAME beside SPACE_GROUP_NUMBER, since the number alone does not say which axes a group's symmetry lies on, and a group that is not a reference setting is printed as its extended Hermann-Mauguin name. The two-arm reconciliation compares point groups by that name rather than by number for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	26fc4b02b3	symmetry: carry the space group as the group, not as its number ... The adopted space group travelled the pipeline as a bare int and was rebuilt downstream with find_spacegroup_by_number, which returns the reference setting. So every setting a number cannot name was destroyed one line after it was determined: P 1 1 2 came back as P 1 2 1, I 1 1 2 as C 1 2 1, R 3:R as R 3:H. DatasetSettings now holds the gemmi::SpaceGroup itself, DiffractionExperiment exposes it as GetGemmiSpaceGroup() / GetSpaceGroupOrP1(), and everything that used to take an int - HKLKeyGenerator (its int constructor is gone, so the compiler finds the callers), the merge, the R-free flags, French-Wilson, the reindexing ambiguity, the completeness enumeration, the MTZ and mmCIF exports, the model validation - takes the group. -S keeps the setting the symbol names rather than reducing it to a number. The end message carries both spellings and a reader prefers the name, since only the name keeps the setting while the number is what a reader written before the name understands. It carries them over CBOR too: the determined group was never serialised at all, so a group rugnux chose reached the master file only when the same process wrote it, and an online writer fell back to whatever the user had supplied at the start. Both keys are optional additions, so an older reader skips them and a newer one reads an older sender. On disk the master's /entry/sample/space_group carries the extended Hermann-Mauguin name and is what the reader takes the group from, so a setting survives a _process.h5 and the --mode scale that re-reads it; the number stays beside it and is the fallback for files written before. Every one of the 230 reference settings the old writer could produce reads back as itself, so older files are unaffected. Stage A and Stage B of the search still enumerate reference settings only, so this determines no group differently today - it is what the enumeration needs before it can be widened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:16:43 +02:00
leonarski_f	6ed4ea541e	reader: a link to a file that is not there is not a dataset that exists ... Every DECTRIS Eiger master links saturation_value, pixel_mask, bit_depth_readout and serial_number into a companion <prefix>_meta.h5, and that file is routinely not kept when a dataset is archived or deposited. The existence test asked only whether the LINK was written, which it is, so every optional-field guard in the reader answered yes and the read that followed threw. A deposited Eiger 16M set could not be opened at all, over values the reader was perfectly prepared to do without. Exists() now asks the second question too - whether the object the link names can be reached - so an orphaned link reads as absent and the fallbacks behind it do their job. The saturation value is then allowed to be missing outright, because on such a file it is: neither the NXmx name nor the DECTRIS one is readable, and there is no third place to look. Left unset, GetSaturationLimit() falls back to the container's own overflow. That is the safe direction - it can only fail to call a pixel saturated, where too LOW a value drops the whole reflection and silently removes the strongest data - and the run says out loud that nothing will be called saturated. The set that could not be opened now processes to 2.17 A against a deposited 2.40 A, in the deposited space group, with a cell agreeing to 0.08%. Output is byte-identical on datasets that already opened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit 0637979f6d95b446406ab70d1f3982841d195b36)	2026-08-30 21:43:54 +02:00
leonarski_fandClaude Opus 5	fb18457e6f	docs: bring the changelog and the method notes up to rc.166 ... The rc.166 changelog was missing fourteen user-visible changes and carried rationale and measurements that belong here instead. Added: the native miniCBF sweep reader, the third-party and firmware-1.x NXmx masters, the plain LZ4 filter, the image orientation taken from the file's module direction vectors, the two beam-centre flags and their rescues, and the FFT reach past 500 A. Trimmed the rest to one line each, moving the numbers out of the user-facing file. RUGNUX.md described the input as a single Jungfraujoch master file, which it has not been since this branch; it now covers the foreign and legacy masters, the accepted compression filters and the miniCBF sweep, including how a sweep is collected from one named frame. Six options existed with no entry in the table - --beam-center-check, --beam-center-search, --fft-min-unit-cell, --min-indexed-spots, --rot3 and --no-p1-crosscheck - and -C now moves both FFT cell bounds, which was not written down anywhere. CPU_DATA_ANALYSIS.md carried two statements this branch made false: 7.5 still said pass 2 reuses pass 1's space group, and 13.1 still said centrings are ranked by net absence count. Both now describe what the code does - the group is determined after pass 2, and centrings are ranked by the same Beta-tail likelihood the screw test uses. Also documents the per-zone screw scoring, the coplanarity volume-fraction guard, the plane-normal transform, the FFT cell bounds and the twelve refined candidates. The miniCBF reader implements the x-CBF_BYTE_OFFSET scheme and reads the imgCIF axis table from the specification alone. No CBF code is vendored or linked, so there is no licence obligation, but reimplementing a published specification carries one of credit: ACKNOWLEDGEMENT.md gains a section and the two algorithms carry a one-line reference each. Both DOIs were resolved before being written. Rottger's initial was wrong where this branch first cited it - K, not A. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 21:15:31 +02:00
leonarski_fandClaude Opus 5	9924dd9fc3	docs: the open test battery, updated - 82 datasets, and not only non-SLS ... The public data the pipeline is exercised on has grown from 59 datasets to 82, 77 of them with a released PDB entry and released structure factors, so the page that credits the depositors and carries the DOI to cite for each is brought up to date with what is actually run. Renamed from NON_SLS_TEST_DATA to EXTERNAL_TEST_DATA, because the old title stopped being true: a few of the sets were collected at SLS beamlines, where the data are still written by someone else's detector and someone else's acquisition system. What the battery tests is foreign files, not a foreign facility. Also rewritten from the current archives rather than the earlier sample: - Multi-collection archives: eleven are not a single continuous rotation, not four. Seven IRRMC archives hold more than one collection; one sweep is kept in six of them, and both are kept in the one whose two sweeps are at different wavelengths. A repository project page is not a reliable guide here - one describes a 900-frame sweep its own tarball does not contain. - Detector labels: 76 rows can be compared against the PDB entry. Seven genuinely conflict, and one of those the file settles outright - pixel count, pixel size, sensor thickness and firmware string agree with an EIGER2 9M against the entry's PILATUS4 4M. A further 29 differ only in how much they state, which is not a conflict. - One archive ships 30 placeholder files named like images that are 64-byte text; named on the page so a reader that globs the directory is not surprised by them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 20:50:10 +02:00
leonarski_fandClaude Opus 5	03456dcddd	calibration: an option to hold the detector tilt fixed ... --mode calibration fits five parameters - beam centre, distance and the two PONI tilts - and a program that cannot express a tilted detector has nowhere to put the last two. Dropping them after the fact is worse than never fitting them: the centre and the distance of a tilted fit have already absorbed the tilt, so the flattened geometry is right nowhere. rugnux --no-refine-tilt, the "Refine detector tilt" tick box on the viewer's Calib page and RingOptimizer's refine_tilt argument hold rot1/rot2 at the value the geometry came in with and fit the remaining three. That is the best flat-detector answer, and the one such a program would refine to itself. Measured on a five-distance calibrant series. At short distance the tilt is real and reproducible - three independent fits agreeing to 0.01 deg, radial rms 1.4 -> 0.4 px - and its direct beam agrees with the background beam-centre estimator to 0.05 px, so the tilted model is the physically right one. The pinned fit then displaces the centre 2.6 px to absorb the tilt and lands within 0.03 px of the same place at every distance. Past ~300 mm, where only two rings reach the detector, the tilt is instead under-determined: it comes out with the opposite sign to every short-distance fit and drags the PONI 28 px while the rms does not move (0.960 against 0.962). The existing degeneracy guard only fires on a strictly single ring, so it does not catch that; declining a tilt that does not pay for itself in rms is left for a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 20:44:12 +02:00
leonarski_f	d5818713cb	rugnux: also merge in P1, so a wrong space group is recoverable ... A rotation run that determines its own space group now also writes <prefix>_P1.mtz: the same observations, the same scaling, merged in P1. If the group was wrong there is no route back today except processing the images again, and the P1 data settle it - re-merge, re-solve or re-refine in any subgroup. Written whether or not the search adopted P1, because a file whose presence depends on what the pipeline decided cannot be harvested by a script. A run given -S writes nothing: its group's centring absences were never integrated, so the P1 merge would be missing whole classes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f # Conflicts: # docs/CHANGELOG.md # rugnux/rugnux_cli.cpp	2026-08-30 14:15:20 +02:00
leonarski_f	123dc22be2	rugnux: write the unmerged MTZ by default ... A run that does not ask for it still has to hand its data to aimless, pointless or careless eventually, and the file it needs is one a user had to know a flag to get. It is the largest a run produces, but it is 3.4 to 5.8 per cent of the images it replaces - and the better it is, the less anyone needs to keep those. The batteries pass --no-export-unmerged: they process the whole corpus and would write a few gigabytes nobody reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f # Conflicts: # docs/CHANGELOG.md	2026-08-30 14:14:28 +02:00
leonarski_fandClaude Opus 5	f75cf72d1a	beam centre: measure it on every run, and try it when the header fails ... The header beam centre is wrong by more than the geometry absorbs on two thirds of foreign depositions, and nothing measured it. The solvent ring already gives it away: the background projection the beam-stop pre-scan builds is enough to fit the centre, so the measurement is a fit over an array the run has already paid for. It is reported on every run and committed on none. The first pass runs again at the measured centre and the two lattices are compared; the measured centre is taken only where the header indexes nothing, and a disagreement is reported rather than resolved, because the only arbiter available at that stage is the frame count and it is inverted. Recovers two depositions whose header is the geometric detector centre. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:07:13 +02:00
leonarski_fandClaude Opus 5	65dfdb1324	reader: read the detector arm and the mounting each file states ... A detector swung out on a 2theta arm was read as if it stood square on, and a miniCBF written with a vertical spindle or a quarter-turned image was read with the standard mounting assumed. Both are stated in the file and both were ignored: the NXmx depends_on chain was never followed, and Detector_2theta was parsed into a field nothing read. Recovers three datasets that produced no usable lattice, and the small-molecule sweeps at 30 and 55 degrees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:07:04 +02:00
leonarski_fandClaude Opus 5	7528bd8761	docs: write down the three directions this work is heading in ... Three preferences that have been decided in conversation and applied to several changes already, but were nowhere a new reader would find them. They are directional, not absolute. Spending compute to buy quality is now affordable where it once was not, so the question to ask of a slower design is what it buys. Deciding late beats deciding at a threshold, because a gate calibrated on one population refuses another - measured repeatedly in the lattice and symmetry code. And the target is a bare invocation with no flags, because a capability nobody knows to switch on has not solved the problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:06:54 +02:00
leonarski_fandClaude Opus 5	a27c4cf26f	reader: take a miniCBF's mounting from the imgCIF axis table its header states ... A miniCBF header states three things about how the instrument is put together that the reader was assuming instead: which laboratory direction the image's columns run along, which its rows run along, and which the spindle turns about. Some beamlines append a CBF template block holding the full imgCIF axis table, which says all three outright. Two instruments in the corpus are not what was assumed, in two different ways. One mounts its detector a quarter turn round, so the image's columns run vertically. Another turns its spindle about the VERTICAL, with the image mounted the usual way; its table says so, and its "# Oscillation_axis" line says so a second way, by naming the image direction the spindle runs along rather than a vector. Either error leaves the spindle 90 degrees from the image. That is not a sign, so the run's axis-sign rescue cannot reach it, and no refinement recovers it: all three affected sweeps indexed nothing usable. So the table is read. The element axes give the image orientation, matched against the eight discrete mountings exactly as the NXmx module directions already are - the match itself moves to DetectorOrientation, so both readers share one definition rather than two copies. The goniometer axis with no parent gives the spindle DIRECTION; its sign stays the rescue's business, which is the part a convention can legitimately differ on. The detector axis with no parent gives the 2theta arm, replacing the assumption that the arm shares the spindle's axis - the one header stating both states them with the same vector, so this changes no answer, only what it rests on. imgCIF's frame differs from the internal one by a half turn about x, a rotation and not a mirror, as writer/HDF5NXmx.cpp already records from the other side. Where a header carries no table, a "+SLOW" on the Oscillation_axis line still says the spindle runs along the image's slow direction. That is the only thing one of the three affected sets says about it. The axis NAME on that line stays unusable - the header that carries both says "X.CW" where its own table says Y - but the direction token is not: where both are present they agree, which is what makes reading it evidence rather than a guess. Also: naming a frame with no directory at all now finds its sweep. parent_path() of a bare filename is empty and iterating an empty path finds nothing, so running from inside the data directory reported that no images were found. Measured, with nothing on the command line. The vertical-spindle protein set goes from no usable lattice to 100% indexed, P 6(3) 2 2 with a cell 0.43% from deposited, 87846 reflections at 86.3% completeness and CC(1/2) 0.995. Its companion from the same detector, which has no table and only the +SLOW token, goes from a spurious monoclinic cell at 2.3% completeness and I/sigma 0.21 to the right orthorhombic lattice, 97.7% indexed, 59.7% complete, CC(1/2) 0.996. The quarter-turned set's three sweeps, at three arm positions, now all index without the hand-passed quarter turn they needed and agree on one cell to 0.03 A. Six miniCBF sets that state no table and no +SLOW - including one whose Oscillation_axis line names an axis in a third dialect - are byte-identical in .hkl, .mtz, .cif and the image statistics, as are two NXmx sets, which is the shared orientation matcher moving nothing on that path either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:03:32 +02:00
leonarski_fandClaude Opus 5	dc71cb7299	rugnux: keep the P1 merge, so a wrong space group is recoverable ... A de-novo rotation run merges in P1 to search for the symmetry, adopts a group, and then overwrites that merge with the in-symmetry one. If the adopted group is wrong the user has no route back: every file the run wrote, and every statistic in them, is computed in the group that was assumed, so nothing in the output says the choice was wrong and the only way to a different answer is to process the images again. Merge the same integration once more in P1 after the run's own files are written, and put it beside them as <prefix>_P1.mtz. From it the space group can be re-determined and the data re-merged, re-solved or re-refined in any subgroup - measured end to end on three crystals: POINTLESS reads the file, recovers the group, and AIMLESS re-merges it, reproducing rugnux's own merged intensities at CC 0.9965 where the two agree. On one of the three it recovered the deposited/XDS group where this run had under-called the screw axis. The merge is the full one - correction surfaces fitted, ice rings and near-tangential observations kept, whole resolution range - not the deliberately degraded merge the space-group search itself runs on, and it is what `rugnux --mode scale -S P1` produces from a _process.h5. That route already existed but needs a _process.h5, which a merging run does not write, so it only helped a user who had foreseen the problem. Every de-novo rotation run writes the file, including one whose search concluded P1 and where it therefore repeats the merged output byte for byte. Whether a file exists must not depend on what the pipeline decided: a script harvesting results would otherwise have to reproduce the search's decision to know whether to expect it, and a missing file would not separate "the run chose P1" from "the run failed". A user-fixed -S writes nothing, and that condition is not a pipeline decision. With a group fixed, prediction rejects that group's centring absences (IndexAndRefine.cpp:499-506), so those reflections are never integrated; a P1 merge built from such a run would be missing whole centring classes and would mislead rather than merely be smaller. Cost, median of five paired runs read off the log timestamps (the box is shared, so end-to-end wall time is noise): +0.32 s of 4.2 s, +2.18 s of 31.4 s, +0.19 s of 17.9 s, +0.66 s of 15.9 s - 1 to 8% of a run. The file is 1.6x the merged MTZ on a monoclinic crystal and 30x on a cubic one, where the merged MTZ is tiny; it is well under the unmerged export in every case measured, and 0.02 to 0.6% of the raw dataset. Rotation only for now: the stills merge re-fits per-image scales and per-reflection partialities onto the integration outcomes, and _unmerged.mtz is written from those afterwards, so on stills this extra merge alters a file that is the run's own output. Fixing that means running the cross-check below the unmerged export, which needs the merge lambda hoisted out of its block; deferred, since rotation is what the online pipeline processes. On rotation the merged .mtz, .cif, .hkl, _image.dat, _unmerged.mtz and _unmerged_partials.mtz are all byte-identical with and without this change, including on a run whose search returns P1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 12:36:49 +02:00
leonarski_fandClaude Opus 5	5c44e544dc	reader: place a detector swung out on a 2theta arm where the file says it stands ... Chemical crystallography reaches high angle by swinging the detector out on a 2theta arm. Both readers had the number and neither used it: the miniCBF header's Detector_2theta was parsed into a struct member nothing ever read, and on the NXmx side the rotation was in the depends_on chain, which was not followed at all. A sweep taken at 30 degrees was therefore processed with its detector plane 30 degrees from where it stood, and nothing indexed. The geometry could already express it, and needed no change: the arm turns the detector about the sample, so the distance is still measured along the detector normal and the beam centre is still the point of normal incidence - which is exactly the PONI convention, and a swung detector is one PONI rotation. What moves is the direct beam, by distance*tan(2theta), off the beam centre and often off the detector. NXmx is the harder half, because the swing has no field of its own: it is one rotation in the chain the detector's position depends on, and "two_theta" is only one beamline's name for that dataset. So the chain is followed and its rotations composed, rather than a field of one name being looked for - each transformation states its vector in the frame of the one it depends on, which is why the product is the whole placement. Translations are skipped; they are the distance and the beam centre, which the file states separately in the square-on frame. Vectors come from McStas through the same 180-degree turn about z the module directions already use, a proper rotation, so an axis carried through it turns the same way. The three rotations a file this system writes ARE that chain, and are also read as the PONI angles - so those three paths are skipped, or every tilted file we have ever written would come back tilted twice. That is the one way this change could have broken existing data, and the test for it writes a tilted file and reads it back. For miniCBF the arm turns about the base spindle axis: on the four-circle geometry those headers describe the two are one axis, and the imgCIF axis table such a header carries states them with the same vector. Both now come from one constant, so a later correction to the frame moves them together. Measured. On a swung NXmx sweep the chain gives rot2 = -0.34907 rad for the 20 degrees it states, and the sweep goes from "nothing was integrated" to 25000 reflections at 82.2% completeness and CC(1/2) 0.9993, in the same space group and the same cell to 0.03 A as the square-on sweep of that crystal; the opposite sign indexes nothing. A miniCBF sweep at 30 degrees goes the same way, to 0.585 A, and a second sweep of that crystal at 55 degrees reaches 0.476 A and reproduces the cell again - with a low-resolution limit of 2.36 A rather than 13 A, which is what a detector swung that far records. On all of them post-refinement recovers the header's own beam centre and distance, and the beam stop shadow sits within four pixels of where the swung geometry puts the direct beam, 417 and 537 pixels from where the unswung one does. Seven sets whose detector is square to the beam, three of them carrying a chain whose 2theta is zero, are byte-identical in .hkl, .mtz, .cif and the image statistics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 12:36:46 +02:00
leonarski_fandClaude Opus 5	5f47d73cc1	rugnux: write the unmerged MTZ by default ... The unmerged export is how a run's observations reach the rest of the field. It is what aimless, pointless and careless read, it is what a head-to-head against another program's answer runs through, and handing several of these files to pointless is the only way to merge sweeps rugnux does not combine itself. Behind a flag it reached only the people who already knew the flag existed, which is the shape of defect the "rugnux <file> with nothing else" direction asks to design out. --no-export-unmerged turns it off, and the regression batteries now pass it: the objection to defaulting it on was their disk and time cost, not the product's. Measured on three rotation crystals of 900, 1800 and 3600 frames: the file is 21, 86 and 26 MB - two to five times the merged .mtz, .cif and .hkl put together - and the write costs 0.24, 2.29 and 1.11 s, 5.5 %, 12.2 % and 3.1 % of each run's own wall time. The cost is linear in the number of observations and independent of how long processing took, so it is the largest fraction of the runs rugnux finishes fastest, not of the longest sweeps. A run's .hkl, .mtz and .cif are byte-identical with the export on and off. --export-unmerged-partials stays off: it is a second, larger artefact and a separate question. The viewer's reprocessing dialog ties the export to its merged-output switch, so it appears beside the merged files rather than beside a job that only asked for the per-image _process.h5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 10:51:25 +02:00
leonarski_fandClaude Opus 5	bf41b1fab1	rugnux: index both beam centres always, without a significance gate first ... The second first pass was skipped where the measured centre sat within three times the fit's own sigma of the file's - a quarter of the non-SLS corpus, eight of thirty-two runs. That is a decision taken at a threshold before the evidence is in, and the evidence here costs a median 0.55 s: the spots are already found and cached, and their positions do not depend on the centre, so a second first pass is a median 20 % of what the first one costs. Worse, the gate removed exactly the case worth asking about. A centre error ALONG the spindle does not fail - it holds 96-100 % of frames indexed and quietly returns an axis harmonic - and the scale it turns on is a fraction of a pixel: measured on real data, 0.12 px of centre is the whole difference between the deposited cell and a halved axis. No sigma small enough to gate on would notice that, so the gate could only ever hide the question, never answer it. The three-sigma test stays where it belongs, in the report: it says whether the difference is a measurement or noise, which is what a reader wants to know. It no longer decides whether to look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 10:28:35 +02:00
leonarski_fandClaude Opus 5	d8816169e1	rugnux: ask the rotation-axis sign at the measured centre, and say when only the symmetry differs ... Two corrections to the check, both from measuring it on the non-SLS corpus. Where the file's centre indexes nothing and the measured one does not either, the rotation-axis sign is now asked again AT the measured centre. The two unknowns are discrete and coupled: on the two public depositions in the corpus whose header beam centre is the geometric centre of the detector, the file is 73 px out AND its axis sign is the opposite of the one that indexes. The sign rescue therefore asks its question at a centre 73 px wrong (0/60) and puts the sign back, and the centre is then asked at the wrong sign (0/60). Each error hides the other and the run produces nothing. Asking the pair takes both runs to 100 % indexed on the deposited lattice - 199.40 67.11 against a deposited 199.54 67.15, and 208.77 67.20 against 208.77 67.22 - at the cost of one more first pass on a run that has already failed twice. Asked here rather than by moving the check ahead of the sign rescue, which was tried: that also works, but it moves the beam centre of three datasets whose only fault is the axis sign, for no benefit. Where nothing works the file's centre is put back, so a run that fails for another reason fails at the geometry it was given - 6yqf, whose spindle is along the detector's slow axis, and 7atg both come out exactly as they did before. And the beam centre decides the primitive VOLUME, not the Bravais class: a centre error along the spindle makes the FFT take an axis harmonic, which changes the volume by an integer factor or by sqrt(3), while a class differs for a reason that has nothing to do with the centre. The two are now reported separately. Both "disagreements" in the corpus are of the second kind - primitive volumes agreeing to 0.16 % and 0.57 % while one pass reads R-centred trigonal or P orthorhombic and the other stays triclinic - and in both the file's centre is the one that finds the symmetry and the run's cell matches the deposited one to 0.23 % or better. The cell was never in question, and the strong warning now fires zero times in 32 datasets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 09:59:41 +02:00
leonarski_fandClaude Opus 5	60f6a46b25	rugnux: index the measured beam centre too, and say when the two answers differ ... The estimate the previous commit makes is free, and a first pass re-uses spots it has already found, so running the first pass a second time at the measured centre costs about what one rung of --beam-center-search costs. That buys the one comparison an on-failure trigger structurally cannot make. A centre wrong ACROSS the spindle announces itself - the indexed fraction collapses - but a centre wrong ALONG it does not: the run holds 96-100 % of frames indexed and quietly returns a 2x, 3x or sqrt(3) axis harmonic. Nothing fails, so nothing fires. Indexing both centres and comparing the two lattices is what can see that at all. What must not arbitrate the two is the indexed frame count. Acceptance is a fractional-Miller test, so a cell twice as long has to place every spot twice as accurately to score the same; measured on real data, two centres 0.12 px apart gave the deposited cell at 99.23 % and a halved axis at 100.00 %, and the wrong answer indexed better. A rule of the form "take the centre that indexes more" picks wrong in exactly the case the comparison exists for. So only what needs no arbiter is decided. The file's centre indexing nothing where the measured one indexes a majority is not a comparison, it is a run that produced nothing and now does: take the measured centre. The two agreeing on the lattice is reported and nothing else - it is a free statement that the header is good enough for this crystal, which is most runs and is worth saying. A disagreement between two passes that both worked is reported with both cells, their primitive volumes and the ratio, flagged when that ratio is an axis harmonic, and left undecided. Runs before the blind ladder, so a measured hypothesis is tried before any grid, and only where the move exceeds three times the fit's own sigma - below that the two centres are one measurement twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 09:00:55 +02:00
leonarski_fandClaude Opus 5	0644404286	rugnux: say where the scattered background puts the beam, on every run ... A beam centre in a file is the metadata field everyone jokes about and nobody measures. rugnux already has an estimator that can measure it - the isotropy of the scattered background, whose leverage is the curvature of the water ring - and it already builds, on every run, the projection that estimator needs: --detect-beam-stop makes one to find the shadow. So the measurement is a fit over a mean image that already exists, no frames of its own, and it can simply be done always. --beam-center-check, on by default, does that and reports it: what the file claims, what the background says, how far apart the two are, how well the fit knows its own answer, and how right this particular geometry needs the centre to be. That last term is what makes the line a verdict rather than a pixel count - 3 px is nothing at 100 mm and a lost lattice at 500 mm. On the two public depositions in the test corpus whose header is the geometric centre of the detector, the line reads 73.49 px and 70.96 px against a 1.57 px and 1.82 px tolerance, on runs that until now said nothing at all about it. Nothing is committed. The run keeps the centre it was given, and where the fit cannot place the centre - a flat background has nothing to separate a radial shift from an amplitude - it says so rather than guessing. Measured on a 0.3 Mpx detector the fit takes 8 ms. The fall-through in --estimate-beam-center now reads this same result instead of repeating the fit; it is the same call on the same projection with the same mask, so that flag's answer is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:53:05 +02:00
leonarski_fandClaude Opus 5	a2f451cf12	rugnux: try the beam centre as an indexing hypothesis after a failed first pass ... A beam-centre error is not repairable downstream. It is fixed in the lab frame, so accumulating a sweep smears every reciprocal-lattice point around a circle and the FFT amplitude at an axis of length a is multiplied by J0(2 pi delta p a/(D lambda)); past the first zero the true axis is gone and its harmonic wins. But it is decidable from the data, on exactly the count the scheme choice and the axis-sign rescue already use: the right centre indexes and the wrong one does not. So after a first pass that indexes fewer than half the validation frames, step the centre a pixel at a time and keep the first rung that clears the same majority the other guards test. It runs only after a pass that has already failed, so a run that works never pays for it, and spot finding is not repeated - the cache is keyed by image and the spot positions do not depend on the centre. On the prototype this rescued 2 of 12 failing non-SLS datasets and declined cleanly on the other 10. BOTH detector directions are searched, and that is the part to keep. Measured by injection on two crystals: across the spindle the cell stays right and the indexed fraction collapses, 99 % to 25 %, so that direction announces itself; along the spindle - the one the J0 derivation calls free, because the FFT amplitude is translation-invariant - the run holds 96-100 % indexed and quietly adopts a 2x, 3x or sqrt(3) supercell. Searching only the perpendicular direction would search the failure that already shows. The step is one pixel, flat. Deriving it from the J0 law was tried and is wrong: the only cell available at that point is the one the FAILED pass returned, and on a dataset whose failed cell was a small spurious sub-cell the formula asked for a 6 px step, which steps over the lobe it is looking for. Taking any improvement rather than requiring a majority was also tried and is wrong: where no centre works, the ladder wandered to the far end of its range on 10/60 against 5/60. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	4d2bb98a13	rugnux: name the beam centre when two candidate cells are axis harmonics ... The first pass already has the evidence and says nothing about what it means. When two schemes come back with primitive volumes related by a small integer - or by sqrt(3), which is the hexagonal harmonic and is not a whole number - one of them is the other's axis harmonic, and what decides between them is the beam centre to a fraction of a pixel. The mechanism is that the J0 law is about the FFT AMPLITUDE, which is translation-invariant. A centre error along the spindle translates the derotated cloud rigidly, so the transform cannot see it, the peaks stay sharp, and the lattice fit that follows - whose origin is the beam, and which is not translation-invariant - commits with confidence to a sub-multiple. Measured on a deposited dataset: 0.12 px of centre, 0.09 px of it across the spindle against a 2.78 px need, is the whole difference between the deposited cell and a halved axis, and the halved one is the answer that indexed MORE frames - 100.00 % against 99.23 %. So the message is deliberately not conditioned on any indexing-quality signal, and it is taken over the whole pair rather than only over the tie: on this failure the frame count points the wrong way, which is exactly why it cannot arbitrate. Nothing is decided differently; the integer-subcell tie-break below is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	52e9e9da2b	beam centre: gate on what the geometry needs, and on the move being significant ... The accept bound was one flat 1.0 px constant applied at three points. It was calibrated on a 244 mm / 0.95 A / 0.15 mm geometry, and the displacement the J0 smearing law actually allows runs from 0.6 px to 9 px over the in-house and non-SLS corpora - a fifteen-fold spread - so on a loose geometry the constant is nine times too tight and throws away answers that are perfectly usable. The bound is now the LARGER of the constant and what the geometry asks, which is one-sided by construction: nothing today's gate accepts can be lost. What the geometry asks is printed and never tested against. It says how wrong the header may be; the estimator's sigma says how well the estimator knows its own answer, and gating one on the other rejects a centre correct to 0.03 px because its sigma was 0.98. It is not a relevance floor either - measured on real data, a 0.12 px change of centre, 0.03x of what the law asks across the spindle, is the difference between the deposited cell and a halved axis - so the printed line says so. Added alongside: the move has to be worth making. Under three times the estimator's own sigma it is not a measurement of anything, and a centre that is not moved cannot push the two-pass loop off its fixed point. On the 39 rotation regression crystals this adopts 38 and keeps the header on one. Two things that were happening silently now say so: the move resolved across the spindle, which is the only component the law is about, and the fall-through itself. Below about 220 deg of sweep the spot symmetry never clears the bound and the background answers instead - correctly, to within 0.23 px of the full-sweep centre at every span from 120 to 220 deg - and until now the log did not say the spot arm had even been tried. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	d1818b195e	rugnux: say what the file's beam centre looks like, and when no beam stop is found ... Four things a run can say about the beam centre before it reads a single frame. All of them are free, so none of them is behind a flag. Two are worth acting on. A centre that lands on a masked pixel is wrong about something - a real beam does not sit on a dead pixel or in a module gap - and over the 29 non-SLS datasets whose outcome is on record it fires on 4, of which 3 went wrong, against 2 false alarms in 42 in-house masters. And a beam-stop pre-scan that finds exactly zero pixels is not a beamline without a beam stop: the flood starts from seeds within 4 px of the ASSUMED centre, so a centre far enough out puts every seed in a module gap. That one fires on 3 of 29 foreign datasets, all 3 of which had a bad outcome, with no false alarm in 39 in-house sweeps. The other two are provenance and are said rather than warned about: a centre equal to the geometric detector centre, or a whole number of pixels in both coordinates, was typed rather than measured. They are true of 29 of 42 in-house masters and those are out by a median 3.5 px, but as predictors of harm they sit at or below the base rate, so they gate nothing. They earn their line because the spot-symmetry search reaches only about +-55 px: a placeholder tens of pixels out is unreachable by construction, all five of its searches agree, and the sigma comes back small. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	97f57a430d	beam centre: run the four consistency restarts at once ... FindBeamCenterFromSpotSymmetry calls Estimate() five times - once for the answer and once from each of four starts 25 px away - and three quarters of each of those is an 861-point brute-force grid over the spindle. So making the uncertainty gate live was paid for by multiplying the estimator by five, which is the whole of the pre-scan's cost. The four restarts share nothing: each takes its own copy of the geometry and only reads the spots. What is wanted from them is a max, which is order-independent, so running them concurrently gives the same number. Measured on three rotation crystals, the committed centre, the reported sigma and the fitted spindle angles are unchanged to every printed digit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	3ddfbb2da9	tests: give both beam-centre estimators a case with the detector tilted ... No test anywhere set a detector tilt, so PoniRot1/2 were zero in every one of them and the PONI and the direct beam sat on top of each other. That matters because the conversion between the two is used three times in the spot estimator - to centre the vote, to start each tooth's refinement, and to turn the answer back out of the spindle frame - and with the two centres coincident it is the identity, so its sign was unobservable. Verified by flipping it: with DirectBeamOffset negated, the ten pre-existing beam-centre cases all still pass and only the new one fails. The tilt used here puts the direct beam about 12 px from the PONI, twenty-four times the tolerance asserted, and the case also pins that the tilt is not read as a spindle azimuth - what the fit sees of the detector belongs to the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	8e9ca1f6d2	tests: open a third-party NXmx master, in the shape a Diamond-written one takes ... Covers all five ways such a master differed from a DECTRIS one - lengths in millimetres, no detectorSpecific, the distance one level up in NXinstrument, a pixel_mask linked into a file not holding it, and per-file links naming a plain /data - and the reader's refusal to report a master whose data files cannot be read as a dataset with no images. Verified to fail on the parent commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:27:53 +02:00
leonarski_fandClaude Opus 5	c9ca7e424a	reader: read a third-party NXmx master, and stop a broken one reading as empty ... A valid NXmx master written outside the DECTRIS toolchain could not be opened. Measured on a Diamond-written master of a 360 deg EIGER 16M sweep, where the images and the meta file are pure DECTRIS and only the master is third-party - which is why the two sides disagree on units at all. Five independent things, of which two were silent: * The image size came from detectorSpecific/x_pixels_in_detector, a DECTRIS extension rather than NXmx, so a third-party writer has no reason to emit it. It now comes from the image array's own shape, as it already did for a VDS master. * Lengths were assumed to be metres and the units attribute was never read. A pixel size, sensor thickness or distance stated in millimetres - correct NXmx - was silently a factor of a thousand out. The unit is now read; an undeclared one still means metres, an unknown one is refused. * The detector distance can sit in NXinstrument rather than in NXdetector; that is now the last fallback after the NXmx and the firmware-1.x spellings. * A pixel mask that is an external link into a file not holding it passed the Exists() check and then threw on the open. Whether the array is there is now decided by opening it. * Each data file was re-opened and searched for /entry/data/data, ignoring the path the master's own link names. A master linking to a plain /data therefore found no images at all - and that was a warning and exit code 0 over a sweep sitting right there, not an error. The link is now taken at its word, and a master that links to data files but yields no images is an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:27:53 +02:00
leonarski_fandClaude Opus 5	bec2a10a3a	docs: add the eight new public datasets to the non-SLS test list ... Three IUCrData Raw Data Letters (Zenodo, CC-BY-4.0) and five SBGrid Data Bank depositions (CC0) have been added to the data the pipeline is exercised on. Same rules as the rest of the page: the source is the repository and its own citable DOI, every one of which was resolved before it was written down; beamline, resolution, space group and cell are the values deposited with the PDB entry; the detector is read out of the image files. None of the new detectors disagree with their PDB entry. Two of them do not fit the page's one-row-per-sweep shape, so the shape is described rather than flattened. The 6R72 Zenodo record holds two complete 360-degree collections on one crystal - a helical one that produced the deposited structure and a low-dose one that has no PDB entry - and both are listed, sharing a DOI, with the second in the no-PDB-entry table. The three CHESS depositions are 4-11 wedges of 50 degrees per crystal plus a rotation taken with the crystal translated out of the beam, tabulated in a new section. One SBGrid deposition is named but not in the table: its images are 1995 CCD TIFFs, a format the reader does not support, so it is not processed here and saying so is more useful than leaving it out. ACKNOWLEDGEMENT gains the new per-repository counts and a pointer to the Raw Data Letter citations; TESTS points at the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-29 23:40:10 +02:00
leonarski_fandClaude Opus 5	08f57235d6	reader: take the image orientation from the module directions the NXmx file states ... NXmx says how the stored image sits in the detector plane, in the NXdetector_module fast_pixel_direction and slow_pixel_direction vectors. rugnux read neither, and assumed every detector was mounted the way this system mounts its own. A facility that bolts its detector a quarter turn round therefore produced a geometry that was wrong by 90 degrees, which no amount of refinement recovers: the run indexed nothing and reported that it could find no lattice. The two vectors are read, taken from McStas into the internal frame, and matched against the eight discrete image orientations. An exact match is adopted; anything else is left alone, because an orientation that is not discrete is a continuous rotation of the detector in its own plane and cannot be told apart from the tilt by looking at the module. Every file examined here is exactly discrete. Measured over the 94-dataset battery: 78 masters state no module vectors at all and 12 state the standard ones, so the change can reach exactly 4. Three of those go from "Nothing was integrated" to a complete merged result with no flags - 43/93/92% indexed, CC(1/2) 0.957/0.984/0.998 - and are the sets that until now needed a hand-passed --rot3 pi/2. The fourth states a half turn, which the existing rotation-axis sign rescue already absorbed, and is unchanged to every digit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5	fed077e683	geometry: hold the detector plane as axis vectors, and give the discrete part its own home ... The detector plane was three PONI angles and nothing else, so the two things it cannot express - an image mirrored in Y, and one mounted at a multiple of 90 degrees - had no home at all. They are now the DetectorOrientation carried by the detector setup, composed with the PONI rotation into one orthogonal matrix whose columns ARE the fast axis, the slow axis and the sample->PONI normal: lab = R(rot1, rot2, rot3) * Delta * ( (x-bx)*p , (y-by)*p , distance ) GetFastAxis/GetSlowAxis/GetNormalAxis read those columns and DetectorAxes() sets the plane from them, decomposing back to the angles; PoniRotMatrix and PoniAnglesFromMatrix are the conversion in both directions, exact on the canonical branch (rot2 in [-pi/2, pi/2]) and with a stated convention at gimbal lock. The angles stay stored rather than re-derived, so a geometry given as angles is written back as the same angles, to the bit. Delta is never inferred. In particular an arbitrary rot3 is NOT decomposed into a quarter turn plus a residual: rot3 is a fitted quantity, and a least-squares step must not be able to turn the stored image. It is set only where something states it - the detector setup, --detector-mirror-y / --detector-quarter-turns, or the value a file this system wrote records - and defaults to the identity, which makes the whole change a no-op for every existing detector and every existing file. It is a different setting from DetectorSetup::mirror_y, which flips the MODULE LAYOUT while an image is assembled and so decides what the stored pixels are. Merging the two would apply the mirror twice for every modular detector, or change the pixel content of every file written; both are ruled out. The new one earns its keep exactly where the old one is a no-op: a detector whose image arrives already assembled has no layout to flip. Both generators are signed permutations of the in-plane offset, so they preserve the distance from the PONI. That is why almost nothing downstream changes: everything needing an azimuth already goes through LabCoord, and everything that does not needs only a radius. The two hand-written copies of the rotation - XtalResidual and RingOptimizer - take the discrete part as four constants next to cos_rot3/sin_rot3, since it acts in the detector frame where rot3 acts in the laboratory and cannot be folded into it. RingOptimizer needs it despite being a radial fit: it fits the tilt, and the discrete part changes which way the tilt tips a ring. Carried as two optional CBOR keys and two detectorSpecific datasets, both back-compatible; the NXmx module axis vectors and the translation direction stop being hardcoded and are computed from it, reproducing today's values exactly at the identity. GetPoniRotMatrix is renamed GetDetectorMatrix, because it is no longer only the PONI rotation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5	9d3c2787f8	docs: credit the public diffraction data the pipeline is tested against ... Jungfraujoch is developed at one facility, so the reader and the reduction pipeline are exercised on data collected elsewhere - other detectors, other file formats, other conventions. That data was collected and published by other people, and until now nothing in the repository said so. NON_SLS_TEST_DATA lists all 51 datasets: the DOI to cite for each, the repository it came from, and the experiment as deposited. Beamline, resolution, space group and cell are the values deposited with the corresponding PDB entry, read from the RCSB data API - not results measured here; no quantity produced by this software appears on the page. The detector is read out of the image file instead, because the detector named in a PDB entry is often only approximate, and the twelve cases where the two disagree are listed rather than silently reconciled. ACKNOWLEDGEMENT gains a section for the repositories themselves, with the IRRMC, SBGrid and Zenodo citations and the PDB citation for the metadata. Every DOI on both pages was resolved before it was written down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 21:59:49 +02:00
leonarski_fandClaude Opus 5	84efb0666c	viewer: open a miniCBF sweep, the same way rugnux already does ... The native miniCBF reader added for rugnux is a plain JFJochReader, so the viewer only needed to be told which one to open a file with. JFJochImageReadingWorker held a concrete JFJochHDF5Reader; it now holds both readers and a JFJochReader* pointing at whichever the open file needs, chosen by JFJochCBFReader::CanRead. Naming any frame opens the whole sweep - the reader's own template matcher decides which frames belong to it, so a directory holding two sweeps or XDS auxiliary files is not spliced together. The previous file is closed before the new one is opened, so switching format between HDF5 and CBF in one session leaves nothing behind. Everything the viewer draws already goes through JFJochReader and JFJochReaderDataset, so nothing else had to move. The two HDF5-only features stay on the HDF5 reader: calibration images, and the reprocessing snapshots, which are metadata read back over the same images the reader is serving and have nothing to attach to for a directory of CBFs. A raw sweep therefore shows geometry and pictures with an empty run list, which is the state a live HTTP stream is already in. Reprocessing jobs run on a CBF sweep too - JFJochProcessController opens its own reader the same way, and Rugnux has handled a CBF source since the reader landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 21:49:09 +02:00
leonarski_fandClaude Opus 5	172a845cbb	rotation indexing: refine twelve candidate lattices instead of four ... The first pass refined only min(candidates, 4). The comment at the selection already conceded that the pre-refinement indexed fraction is an unreliable discriminator, and on one dataset degenerate cells occupied three of those four slots - so the right cell, ranked fifth, was never refined. The preceding commit's volume guard frees those slots but does not widen them. This is a one-for-one trade, measured by rebuilding with the old value to confirm the attribution. It gains one dataset: R_meas 27.2% to 23.8%, with mean I over sigma better in EVERY shell at identical shell edges, so it is not a resolution ramp. It costs another: a clean abort becomes a wrong four-fold supercell in P1 at 36% completeness. That is the same shape of trade already accepted for the plane-normal cap search in the preceding commit - an honest failure becoming a wrong answer - on a different dataset. It is committed separately so it can be reverted on its own if the maintainer weighs this instance differently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:35 +02:00
leonarski_fandClaude Opus 5	4e8db41785	indexing: refuse a coplanar candidate, and search the plane normal when the shortlist is flat ... Three related changes to the FFT candidate path, batteried together because they touch the same function. A COPLANAR CANDIDATE REACHED REFINEMENT. ReduceResults filtered triples on lengths and angles only - the 30-150 degree bound admits any flat combination - and there was no volume test. On one dataset 41 of 5535 candidates had |V|/abc below 0.05, with a clean decade gap to the next, and three of them reached the optimizer. UnitCell is float, and for a cell that flat the metric determinant is around 1.5e-7, so float32 gets its sign wrong 19% of the time where float64 never does. The guard against a negative argument to sqrt then CREATES the singularity it was meant to prevent: it puts c in the a-b plane, the reciprocal volume is 1/0, and the residual is 0 times infinity. Ceres reported a not-a-number Jacobian and wrote several hundred lines of solver output per failed solve. VolumeFraction() is |V|/(|a||b||c|), rejected below 0.02 - about 1.1 degrees off flat, ten times below the flattest real candidate observed and a thousand times above where float loses the sign. It is enforced at the producer and at the two optimizer entry points. Note the existing sanity checks use ABSOLUTE volume, which a 320 cubic-angstrom flat cell passes. The same reciprocal-volume division is now guarded at the two remaining sites that share the pattern. A SHORTLIST CONFINED TO ONE PLANE cannot close a cell, and the row it is missing is the plane normal. That is detected from the scatter-matrix eigenvalue ratio - measured, degenerate clouds score 2e-5 to 3.3e-4 against 0.026 or more for every non-degenerate one, a factor of eighty - and one further transform is spent with the same direction count inside a three-degree cap about the normal, so the plan and buffers are untouched. More directions cannot substitute: at the exact true direction the long axis ranks 1422 of 16384 by prominence while the shortlist cut is four times higher. Ranking, not sampling, is the obstacle. A four-fold denser grid was measured and rejected - it reaches the same answer to three decimal places and takes a run from 2.5 to 8 GB of device memory. fft_min_unit_cell_A is reachable as --fft-min-unit-cell and is lowered automatically by -C, mirroring how the maximum is already raised. The default of 10 is unchanged: a lower floor admits spurious sub-cells on protein data, and over 73 protein runs the floor was never lowered while the sibling maximum did fire twice, so the path is live and correctly inert. Corpus of 93 datasets, both arms, one build: 72 bit-identical on report content and p.hkl checksum, 13 failing identically, and the count of working datasets rises by one. The volume guard fires on 58 of 93 and 47 of those stay bit-identical - it fires constantly and almost never changes an answer, which is what it should do. Solver chatter falls from 919 lines across three datasets to none. The cap fires on 4 of 93, none of them in the in-house or private arms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:20 +02:00
leonarski_fandClaude Opus 5	9a66a21a10	rugnux: expose the indexed-spot gate and the third detector rotation ... --min-indexed-spots sets how many spots must lie on a candidate lattice for a frame to count as indexed. It is what the rotation first pass scores candidate lattices on, and it was reachable only by editing the source. The default of 9 is unchanged; on a sparse pattern, lowering it to 6 takes the first pass from 31 of 60 validation frames to 38. --rot3 completes the set beside --rot1 and --rot2. rugnux --mode calibration WRITES a .poni file containing all three, and without this flag a refined third rotation could not be given back - it had to be transcribed by hand and its convention undone. With it, a detector mounted rotated in its own plane is expressible from the command line. Also corrects the description of fft_high_resolution_A in the API, which claimed to be the highest resolution of spots USED. It filters nothing - every spot is projected whatever its resolution. It sizes the projection histogram, and with it the transform. Implementing the documented behaviour was tried separately and makes its own motivating crystal strictly worse, so the code is right and the description was wrong. (The generated Python client doc carries the same text and will pick this up at the next regeneration.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:59:03 +02:00
leonarski_fandClaude Opus 5	367eba55b1	reader: take a miniCBF rotation axis from the goniometer the header states ... Every miniCBF sweep was handed the same hardcoded axis regardless of what its header said, and Chi/Kappa/Phi/Omega were not parsed at all - there were no members for them. A sweep collected on a tilted chi cradle therefore ran with an axis that is 54.7 degrees wrong. The header angles and their increments are now read, the scanned axis is identified from the non-zero increment (the name is consulted only when no increment is stated, which is what absorbs the five different spellings the corpus contains, including one file that states no axis name at all), and a phi scan composes the head chain. An omega scan returns the base axis untouched, because a fixed chi cannot tilt the axis it hangs from. -9999 is a sentinel meaning "not set", not an angle. It is treated as absent, so it can never reach the geometry. The direction and sense are not invented: these files append an imgCIF _axis loop stating their own vectors, and SOURCE with GRAVITY fix the imgCIF-to-internal transform, which independently reproduces the transform this repository already documents for NXmx. Under it the file's own stated phi axis is exactly the composed one, to four decimals. Driving the real reader over all 39 corpus sweeps, 37 return the previous axis bit-identically - including every sweep carrying a large fixed chi, every sentinel header and every axis-name spelling. Only the two genuine phi scans move, and an unrelated rotation dataset is unchanged end to end. This is necessary but not sufficient for the one dataset that motivates it: with the axis corrected it still does not index, because that detector is also mounted rotated 90 degrees in its own plane, which the reader does not yet read. Compensating both takes its phi sweep from no indexed validation frames to 90.89% indexed and a complete merge, which is what shows this half is load-bearing. The detector mount and the two-theta swing belong to the detector-frame work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:50 +02:00
leonarski_fandClaude Opus 5	60c15d6931	reader: open a master whose beamline writes standards-correct NXmx differently ... Four defects, hit in sequence, that between them stopped eight masters from one beamline before any geometry question was reached. The files are correct NeXus; the reader was assuming one writer's conventions. ReadScalar demanded rank 0. NXmx puts no rank on distance, saturation_value, two_theta or det_z, and these files write them as shape (1,). Any dataspace holding exactly one element is now accepted; a genuine vector is still refused. frame_time was read unconditionally and NXmx does not require it. A virtual-dataset source filename of "." was resolved as a relative path, giving <dir>/. - but "." is HDF5's spelling for THIS file, and these masters compose /entry/data/data as a virtual dataset over datasets in themselves that are external links to the data files. The virtual source's DATASET PATH was parsed and then ignored in favour of a hardcoded /entry/data/data, while these data files keep their images at the root. Fixing that exposed a fifth: the positional-read fast path used the master's own path, but a dataset reached through an external link lives in another file and a chunk address is an offset into THAT file. An audit of all 38 masters in the non-SLS corpus finds exactly these eight need the change and the other 30 need nothing. Corpus A/B over 73 dataset pairs, base and patched back to back: 64 byte-identical results, 9 identical failures, none differing. The blast radius is bounded by construction - JFJochReader is linked only by rugnux and jfjoch_viewer, and the one shared header this touches only ever accepts more, so nothing that opened before can read differently. With it the eight files read completely: 12850 images, no decode errors, seven of them spanning two source datasets through the master's own external links. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:33 +02:00
leonarski_fandClaude Opus 5	4ef6bd9952	rugnux: decide the space group after the second pass, not the first ... Pass 1 searched for the space group on the geometry the run started with, stashed the answer, and reinstated it before pass 2's merge - so the decision was made on the worse of the two passes and then fed forward as a constraint. Four guards existed to reconcile it with what pass 2 later found. Now pass 1 does not search at all. It keeps its first merge, which is where mosaicity smoothing and geometry post-refinement already happen, and skips the in-symmetry re-merge that existed only to feed the search. prepass_merge_sg_, prepass_promoted_point_group_, the reinstatement, the primitive-to-conventional reindex it needed, and lattice_conflicts_with_prepass_sg all go. The obstacle was the pass-adoption guard, which compared completeness and CC1/2 across what could now be two different space groups. It compares each pass's FIRST merge instead - P1 on both sides, full range, before correction surfaces - which pass 2 produces anyway. One guard had to be replaced rather than removed. The index-time veto is re-keyed from pass 1's GROUP to its LATTICE and made one-directional: a centred pass-1 lattice against a primitive pass-2 one. Without it a C2 crystal fell to P1, and the obvious narrower fix - exempting a pass-1 P1 - would not have caught it, because that crystal's pass-2 lattice is monoclinic-P rather than triclinic. A centred lattice and its primitive sub-cell share a primitive volume, so the supercell arm cannot see that demotion. Corpus of 93 datasets, both arms at -N 6, every log validated: 78 comparable, 65 with a byte-identical p.hkl. Two crystals gain their reference point group, one gains its screw axis, one is lost. Per shell, over the 72 unchanged crystals, the median move is +0.25 points of R_meas and zero in <I/sigma>, CC1/2 and completeness. Load-matched compute is +0.4%, and the header-geometry fallback fires three times in the old arm against once in the new. What is lost is one crystal whose 222 the operator correlations still confirm: the merge chi-squared ratio moves 3% across a fixed bound at the refined geometry. A second crystal in the XDS harness crosses the H bound the same way. Both bounds are in SearchSpaceGroup and are recorded elsewhere as miscalibrated; recalibrating them is deliberately left to its own change. The comment at the long-axis rescue is corrected while here: the resolution setting only sizes the FFT histogram, and coarsening moves the true axis DOWN the direction ranking - 1996 to 16361 of 16384 - rather than making it more robust as the comment claimed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:17 +02:00
leonarski_fandClaude Opus 5	8634e9a72b	symmetry: score screw absences per zone, and choose centring by likelihood ... Two places where the absence tests threw away information they already had. The screw evidence was POOLED across axial rows, so one unmeasured row vetoed a confirmed one. A crystal reading h: +18/+24 and k: -6/+8 answered a silent P222 rather than "2_1 along a confirmed, b undetermined". Scoring each zone separately, and letting a zone abstain, is what POINTLESS has always done; the conditions are independent, so their log-likelihoods add. Centring was decided on a COUNT of net absences while screws used a Beta-tail likelihood. On one crystal that made F222 (1662 absent, 356 violations, absent class at 34% of present) beat C222_1 (1098 absent, 15 violations, 0.4%) purely because 1306 > 1083. Giving centring the same likelihood separates them by 1717 nats - F222 -357.9, C222_1 +1359.3, I222 -787.6 - and needs no bound: min_absent_observed and max_absent_present_ratio are untouched, and nothing was tuned to this case. The deposited structure for that crystal is C222_1. Corpus of 94 datasets, both arms at -N 6: 78 comparable, 73 bit-identical, point group correct on 68 against 66, space-group number on 54 against 51, nothing regressed. The 73 bit-identical results are also the determinism control - two binaries cannot agree byte-for-byte that often by chance. An earlier ranking key summed each group's single gating number and demoted a P2_12_12_1 to P2_12_12, because pooling three genuine screws reads weaker than two when the third row is shallower. Summing the per-zone log-likelihoods is what fixes that, and the whole corpus was re-run on the corrected binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:57 +02:00
leonarski_fandClaude Opus 5	10873eee7a	lattice search: reject an impossible character, and supplement every angle ... Two defects in the Bravais character walk, both of which silently cost symmetry. An impossible character MATCHED. The walk computes acos(cond_F/sqrt(A*B)); when the character is geometrically impossible the argument leaves [-1,1], acos returns NaN, and the acceptance test fabs(NaN - actual) > tol is FALSE - so the character is taken. One corpus crystal matched a monoclinic-C character whose implied cos(gamma) is 1.086. The type-boundary retry covered beta only. All three angles carry the Niggli boundary, and negating two basis vectors supplements the OTHER two, so each angle needs its own flip. A crystal whose reduced gamma sits at 89.900 degrees needs the alpha/beta flip and never got it. Measured on 84369 exact lattices spanning all 14 Bravais classes: nothing is lost in any class, and orthorhombic-I recovery rises from 60.3% to 87.2%. Over 440000 random and perturbed cells the first change only ever demotes a monoclinic-C match to triclinic and the second only ever promotes out of triclinic - it is a strict superset of the beta-only retry. On the 94-dataset corpus, two crystals gain their correct point group and none regresses. Both regression tests fail without the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:41 +02:00
leonarski_fandClaude Opus 5	1ea6ba25be	reader: open a master that states its goniometer axes in one place and their angles in another ... A hybrid master carries an NXmx /entry/sample/transformations group holding one EMPTY SUBGROUP per axis - the direction as a vector attribute, no NX_class, no units, no angles - beside a legacy /entry/sample/goniometer group holding all the angles. Opening it failed outright with "Cannot open HDF5 dataset /entry/sample/transformations/omega": the existing legacy fallback keys off Exists("/entry/sample/transformations"), which is true here, so it never fired, and the axis stub was then opened as if it were the angle dataset. Present is not the same as usable. GoniometerGroup now takes transformations only if it holds at least one DATASET, and ReadAxis asks IsDataSet rather than Exists, so a member that is not a dataset can no longer be read as one. HDF5Object gains that predicate, in the style of the neighbouring Exists. The stub is the load-bearing half, not merely the thing that crashed. In the legacy branch, before falling back to the assumed (-1,0,0), the reader now looks for the NXmx stub and takes its stated vector. With it the axis is (0,-1,0) and the run indexes 60/60 validation frames; with the stub deleted the assumption applies and the same file indexes 0/60 on both schemes and both signs, and the run stops with no lattice. So without this half the fix would have turned "cannot open" into "found no lattice" - a differently shaped failure, not a success. The direction stated here is 90 degrees from the assumption, not merely its negation, which the rotation first pass could have recovered on its own. The vector size check moved out of the first branch so it now covers every path that produces one. Verified: the file processes end to end, 100% of frames indexed, cubic cell 105.87 against a deposited 105.88 (0.009%), space group reported as I23 or I213 - correctly refusing to choose, since the reflections that separate them are extinguished by the I-centring and were never measured. A both-layout master reprocesses unchanged (100% indexed, P212121). [HDF5] passes: 2194 assertions in 91 cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:47:41 +02:00
leonarski_fandClaude Opus 5	6319d5c600	indexing: tell an indexer that failed apart from one that found nothing ... IndexerThreadPool collapsed two different outcomes into the same empty reply. A worker that threw set result = nullptr, and so did a dispatch that never found a free worker; both then became a default-constructed IndexerResult, indistinguishable downstream from an indexer that ran and found no lattice. The first says nothing about the frame at all - the indexer never looked, and it will fail again on the next one - while the second is a real negative result about the crystal. The visible cost was the advice a failed run gave. With the card full, rugnux printed "Indexer thread 0 failed: CUDA (GPU) error" and then ended with "Two-pass rotation indexing found no lattice. Check the beam centre (--beam-x / --beam-y), raise --max-spots ..." - sending the operator to look at geometry that was never wrong, for a machine that was simply out of memory. None of those remedies can help when the frames were not examined. IndexerResult gains an optional error, set by the worker and by the pool's own catch; the remaining nullptr path keeps its meaning of "not attempted" and deliberately carries no error. RotationIndexer records it and exposes GetIndexerError(), and rugnux's first pass branches on it at the throw site, naming the resource failure instead. The error travels as DATA through image_analysis/ rather than as an exception, because RotationIndexer::RunIndexing() is on the online path - IndexAndRefine calls it on a schedule from the broker and the receiver, where dropping a frame is the right failure and killing a live acquisition is not. Only rugnux, which owns the "this run is over" decision, turns it into one. The failure result is byte-identical to the default-constructed one it replaces, so any_executed and the online frame-drop behaviour are unchanged. The new IndexerError category exists because the category is only the display prefix on what() - Category() is read nowhere - and the old line read "Processing failed: Input parameter invalid" for a GPU fault, which is the same defect one layer up. SpotFinderError is the precedent. msg.indexing_result is deliberately left alone: of its three states, "not attempted" is the honest one for a frame the indexer never examined, and asserting false would be the same collapse again. Verified on a squeezed card (246 MB free, -N 4): exit 1, zero dropped frames, the new message, and no mention of --beam-x. On a free card, against the previous binary at -N 6 on the same input, the logs differ only in paths and timing - same space group, cell and merge statistics. Targeted Catch2 cases pass, including three that drive the pool through the online path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:30:28 +02:00
leonarski_fandClaude Opus 5	92ad304f15	rugnux: fail on an exhausted GPU instead of dropping the images it could not process ... A per-image worker caught every std::exception and continued to the next image. That is right for one undecodable image and wrong for a resource fault: a CUDA out-of-memory says nothing about the image that happened to be in flight and everything about the machine, and it recurs on the next one. The run therefore logged an error per image, skipped each of them, and exited 0 with a dataset that was silently short - the merged numbers all moved, and nothing in the exit code or the summary said so. Observed with several processes sharing one card: two arms of the same comparison lost 21 and 15 images, by different amounts, so the arms no longer saw the same data. IsFatalResourceError rethrows for std::bad_alloc and for a JFJochException of category GPUCUDAError or MemAllocFailed, and every per-image catch consults it first. There are ten of them, not the three the obvious grep finds: besides the analyze/integrate/load sites in the two image loops, the first-pass spot read swallows the same way at Warning severity, the pre-scan swallows it in three places, and one geometry-refinement site caught it with no log line at all. The first-pass one matters most - those frames build the reciprocal-space cloud the indexer runs on, so losing them thins the input that decides whether the crystal indexes, and it does not change IMAGES_PROCESSED, so a frame-count audit cannot see it. Every one of these loops joins with future::get() or ParallelFor's RunTasks, both of which hold the first exception and rethrow it at the join, so the throw propagates and nothing terminates. JFJochException gained a Category() accessor for the test. Verified by squeezing the card to 804 MB free and running at -N 4: the run now exits 1 with zero dropped-image lines, where the same test on the previous binary produced the silent skips above. The broker and receiver paths are deliberately untouched - dropping a frame is the better failure there than killing a live acquisition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 14:58:04 +02:00
leonarski_fandClaude Opus 5	dce8d877d0	Scaling: make Run() idempotent, and re-calibrate the promotion it was hiding ... RotationScaleMerge::Run() resumed its alternating-least-squares loop from wherever the previous call left off instead of restarting from the ingested data. A de-novo run calls it four times, so pass N was scaled at 3N iterations, not 3. Two identical calls on the same object produced merges correlating at 0.847 with ZERO bit-identical intensities; they are now 68010 of 68010 identical. Where that mattered is narrower than it sounds, and worth stating exactly. In a determined point group the fit is stationary from iteration 2 - over the battery the collapsed-scale guard drops 377 frames at three iterations and 397 at twelve, and 30 of 39 crystals drop nothing at any count - so production merges were barely touched, 38 of 39 rows character-identical. But the SEARCH merges in P1, where a reflection has two or three observations, and there the fit never reaches a fixed point at all: the same guard drops 417 frames at one iteration and 1101 at three. A factor 1.05 against a factor 2.6, same code. So the defect lived in the one arm whose output the user cannot recover from downstream - the arm that picks the space group. Fixing it moves the statistics the promotion gates are calibrated on. The population as a whole does not shift (median change 0.9-2.7% over 280 matched candidates) but the tails do, p90 by 8-14%, and that is exactly where a promotion is decided. On the two crystals that decide, the H statistic's two populations SWAP ORDER: a genuine tetragonal 4->422 goes 1.685 -> 1.819 while a trigonal 3->32 twin law goes 1.906 -> 1.665. No H bound exists any more that keeps the genuine case and refuses the twin - the noise the loop was adding had been acting as a brake. Of the four statistics only b and R_meas still order the two correctly. Set max_systematic_b_ratio 1.90 -> 1.78, by a rule written down before the population statistic existed: take the statistic with the largest spurious_min/genuine_max on the population its gate actually sees, put the bound at the geometric midpoint. That is sqrt(1.711 * 1.853) = 1.78. Battery 36/39 space groups matching XDS, the same as before the scaling fix and one better than the scaling fix alone; the three mismatches are the known references where XDS is wrong. The refusal now reaches the user by name. The bound's margin is 4% either side - SMALLER than the p90 shift this very commit produced in that statistic. It separates the two crystals that exist and is not robust to another input shift, which is written into the header rather than left to be discovered. Two hazards go with it: re-measuring it must hold the scaling correction surfaces fixed, since changing the iteration count re-rolls their cross-validated on/off decision on 7 of 39 crystals for ~0.3 in ISa, all-or-nothing and non-monotone; and --scaling-iterations stays at 3 on both arms, confirmed rather than assumed (production 2->3 moves <I/sigma> +1.38% and 3->6 only +0.13%; the search gives the identical space group on all 39 crystals at 2 and at 3, and loses one at 1). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 22:44:21 +02:00
leonarski_fandClaude Opus 5	fa4aa30a1e	reader: match the legacy companion suffixes exactly, not any underscore ... The legacy DECTRIS layout stores five companion scalars beside each goniometer axis - AXIS_end, _start, _increment, _range_average, _range_total - and only the bare name is the axis itself. The first cut rejected _end and _range, which left _start and _increment to be offered to ReadAxis as axes in their own right. Matching the full suffix set, anchored at the end of the name, also keeps a genuine two_theta axis readable, which a "contains an underscore" test would not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	df3696e6e2	reader: fall back to the pre-NXmx key names, so an Eiger 1.x master opens ... Firmware 1.x writes the same three values under different names. Try the NXmx spelling first and the old one only if it is absent: detector/distance <- detector/detector_distance detector/saturation_value <- detectorSpecific/countrate_correction_count_cutoff sample/transformations <- sample/goniometer This cannot change what a current file reads: every modern Eiger master carries BOTH spellings. Measured on thirteen masters from ten facilities, firmware release-2020.2.1 through release-2024.1.1 - all of them write detector_distance and countrate_correction_count_cutoff beside the NXmx names, and a goniometer group beside the transformations one. The goniometer is the one that matters. A goniometer is only ever set from that one group, so a file whose axes are somewhere else was not an error - it was read as STILLS, silently, and the run completed with the wrong answer. The old layout also tags no axis with transformation_type and gives no vector, both of which ReadAxis required, so absence now means two different things by layout: in a transformations group it still means "not an axis" (that is how AXISNAME_end and the width scalars are skipped), while in the legacy group every leaf IS an axis and the companions are recognised by name instead. A missing direction defaults to the one every DECTRIS master since has written and says so in a warning rather than assuming it silently; a wrong guess there does not index, so it is visible, and the rotation first pass tries the opposite sign anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	a7615c8778	compression: read plain LZ4 (HDF5 filter 32004), and name the filter when one is unsupported ... DECTRIS Eiger firmware 1.x compressed its images with the HDF Group's plain LZ4 filter rather than bitshuffle, and files from that era are still what a repository hands you. Read-only: nothing here produces it, and the new enumerator goes last so the existing values do not move, the enum being part of the CBOR stream. The framing is the same as bitshuffle's - a 64-bit big-endian total size, a 32-bit big-endian block size in bytes, then each block prefixed by its 32-bit big-endian compressed size - so only the unshuffle step differs. Verified on a real chunk: the declared total matched width*height*4 exactly and the block walk consumed the chunk to the byte. It cannot reuse the bitshuffle path, which requires the block to be a multiple of BSHUF_BLOCKED_MULT elements: these files put the whole image in ONE block, which is not. The unsupported-filter message now names the filter it found and the ones it knows. Before, an Eiger 1.x file did not report a codec problem at all - it reached spot finding with nothing decoded and failed as "0 spots from 60 images" then "found no lattice", which reads as a crystallography failure rather than a format one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	84d14696b1	rugnux: accept a CBF sweep, decide the rotation axis sign, and search as far as a given cell needs ... Three changes to the rotation path, all in the same first-pass block. Input. Rugnux now holds a JFJochReader rather than the concrete HDF5 reader, so a sweep can come from a directory of miniCBF as well as from a master file; the CLI picks by looking at the file. Only the _process.h5 image links need the concrete reader and they ask for it by dynamic_cast - there is nothing for a virtual dataset to point at in a directory of CBFs, so that file writes its images instead of linking to them. --mode scale refuses a CBF sweep, there being no stored reflections to re-scale. Search bound. A reference cell longer than the FFT bound cannot be represented by the search meant to find it, so the run reported "found no lattice" - strictly worse than not giving the cell at all. Raise the bound to cover a given cell (margin 1.3, as DIALS uses when a cell is known), and raise it in the long-axis rescue too, where it costs nothing: that pass already coarsens the resolution to 3.5 A, which shrinks the transform by 1.75x, so spending it back on reach leaves the rescue at 1.14x the standard pass. The rescue's constrained re-index gets a pool wide enough to hold what the coarse pass recovered, which it did not before. ROTATION ONLY - stills fire the FFT once per image across every worker and routinely run with a known cell, so widening the transform there would cost the whole serial run for nothing. Clamped rather than throwing: the rescue can recover an implausible axis, and a bad candidate must not take the run down with it. Axis sign. The rotation-axis sign is a convention the input often cannot settle - a miniCBF header names the axis but gives no direction, and an NXmx vector is only meaningful together with the detector orientation, which nothing here reads. It is decidable from the data, though: the wrong sign does not index at all (measured on one sweep, changing nothing else: every frame indexed one way, none the other). So after a poor first pass, try the other sign and keep whichever indexes more validation frames - the same count the scheme choice already uses, so no new metric and no new threshold. It flips the AXIS, not the angles, because prediction reads the axis too; the decision then holds for the rest of the run. Runs before the long-axis rescue, since with the sign wrong every candidate lattice is wrong. Only after a poor pass, so a correctly-signed file costs nothing, and spot finding is not repeated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:41 +02:00
leonarski_fandClaude Opus 5	dc16a00271	reader: read a PILATUS miniCBF sweep natively, without libcbf ... Most facilities still archive rotation data as a directory of miniCBF frames, which until now had to be converted to HDF5 before rugnux could see it. Nothing in that format needs a CIF parser or a library: it is an ASCII header, four separator bytes, then one byte-offset compressed image, and every value the reader wants sits on a "# " comment line or a MIME line. MiniCBF holds the format itself - header parse and the byte-offset decoder, which is a running value with deltas stored smallest-container-first. Verified byte-exact against dxtbx on PILATUS 6M, 6M-F, 300K, silicon and CdTe sensors, and three sensor thicknesses. JFJochCBFReader is a sibling of JFJochHDF5Reader under the JFJochReader base. NAMING ANY FRAME READS ITS WHOLE SWEEP: the sweep is identified by the template (prefix + digit count) the named frame belongs to, not by "every .cbf in the directory", so a directory holding two sweeps does not splice two crystals together. Naming a directory takes the sweep with the most frames in it. Images decode on demand, one per call, so any number of workers can read at once - there is no global lock as there is on the HDF5 path, HDF5 not being thread-safe. A raw CBF carries no analysis results, so the dataset it builds is the geometry, the mask and nothing else, exactly as a plain DECTRIS file with no /entry/MX gives. Two header quirks are handled because real files have them: the sensor material is written "Silicon" where the rest of the code compares against "CdTe", and the thickness unit is sometimes omitted. Headers are not a fixed size either - one set carries 6335 bytes - so the parse runs to the binary separator rather than over a fixed prefix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:36 +02:00