1150 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	3ddfbb2da9	tests: give both beam-centre estimators a case with the detector tilted ... No test anywhere set a detector tilt, so PoniRot1/2 were zero in every one of them and the PONI and the direct beam sat on top of each other. That matters because the conversion between the two is used three times in the spot estimator - to centre the vote, to start each tooth's refinement, and to turn the answer back out of the spindle frame - and with the two centres coincident it is the identity, so its sign was unobservable. Verified by flipping it: with DirectBeamOffset negated, the ten pre-existing beam-centre cases all still pass and only the new one fails. The tilt used here puts the direct beam about 12 px from the PONI, twenty-four times the tolerance asserted, and the case also pins that the tilt is not read as a spindle azimuth - what the fit sees of the detector belongs to the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	8e9ca1f6d2	tests: open a third-party NXmx master, in the shape a Diamond-written one takes ... Covers all five ways such a master differed from a DECTRIS one - lengths in millimetres, no detectorSpecific, the distance one level up in NXinstrument, a pixel_mask linked into a file not holding it, and per-file links naming a plain /data - and the reader's refusal to report a master whose data files cannot be read as a dataset with no images. Verified to fail on the parent commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:27:53 +02:00
leonarski_fandClaude Opus 5	c9ca7e424a	reader: read a third-party NXmx master, and stop a broken one reading as empty ... A valid NXmx master written outside the DECTRIS toolchain could not be opened. Measured on a Diamond-written master of a 360 deg EIGER 16M sweep, where the images and the meta file are pure DECTRIS and only the master is third-party - which is why the two sides disagree on units at all. Five independent things, of which two were silent: * The image size came from detectorSpecific/x_pixels_in_detector, a DECTRIS extension rather than NXmx, so a third-party writer has no reason to emit it. It now comes from the image array's own shape, as it already did for a VDS master. * Lengths were assumed to be metres and the units attribute was never read. A pixel size, sensor thickness or distance stated in millimetres - correct NXmx - was silently a factor of a thousand out. The unit is now read; an undeclared one still means metres, an unknown one is refused. * The detector distance can sit in NXinstrument rather than in NXdetector; that is now the last fallback after the NXmx and the firmware-1.x spellings. * A pixel mask that is an external link into a file not holding it passed the Exists() check and then threw on the open. Whether the array is there is now decided by opening it. * Each data file was re-opened and searched for /entry/data/data, ignoring the path the master's own link names. A master linking to a plain /data therefore found no images at all - and that was a warning and exit code 0 over a sweep sitting right there, not an error. The link is now taken at its word, and a master that links to data files but yields no images is an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:27:53 +02:00
leonarski_fandClaude Opus 5	bec2a10a3a	docs: add the eight new public datasets to the non-SLS test list ... Three IUCrData Raw Data Letters (Zenodo, CC-BY-4.0) and five SBGrid Data Bank depositions (CC0) have been added to the data the pipeline is exercised on. Same rules as the rest of the page: the source is the repository and its own citable DOI, every one of which was resolved before it was written down; beamline, resolution, space group and cell are the values deposited with the PDB entry; the detector is read out of the image files. None of the new detectors disagree with their PDB entry. Two of them do not fit the page's one-row-per-sweep shape, so the shape is described rather than flattened. The 6R72 Zenodo record holds two complete 360-degree collections on one crystal - a helical one that produced the deposited structure and a low-dose one that has no PDB entry - and both are listed, sharing a DOI, with the second in the no-PDB-entry table. The three CHESS depositions are 4-11 wedges of 50 degrees per crystal plus a rotation taken with the crystal translated out of the beam, tabulated in a new section. One SBGrid deposition is named but not in the table: its images are 1995 CCD TIFFs, a format the reader does not support, so it is not processed here and saying so is more useful than leaving it out. ACKNOWLEDGEMENT gains the new per-repository counts and a pointer to the Raw Data Letter citations; TESTS points at the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-29 23:40:10 +02:00
leonarski_fandClaude Opus 5	08f57235d6	reader: take the image orientation from the module directions the NXmx file states ... NXmx says how the stored image sits in the detector plane, in the NXdetector_module fast_pixel_direction and slow_pixel_direction vectors. rugnux read neither, and assumed every detector was mounted the way this system mounts its own. A facility that bolts its detector a quarter turn round therefore produced a geometry that was wrong by 90 degrees, which no amount of refinement recovers: the run indexed nothing and reported that it could find no lattice. The two vectors are read, taken from McStas into the internal frame, and matched against the eight discrete image orientations. An exact match is adopted; anything else is left alone, because an orientation that is not discrete is a continuous rotation of the detector in its own plane and cannot be told apart from the tilt by looking at the module. Every file examined here is exactly discrete. Measured over the 94-dataset battery: 78 masters state no module vectors at all and 12 state the standard ones, so the change can reach exactly 4. Three of those go from "Nothing was integrated" to a complete merged result with no flags - 43/93/92% indexed, CC(1/2) 0.957/0.984/0.998 - and are the sets that until now needed a hand-passed --rot3 pi/2. The fourth states a half turn, which the existing rotation-axis sign rescue already absorbed, and is unchanged to every digit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5	fed077e683	geometry: hold the detector plane as axis vectors, and give the discrete part its own home ... The detector plane was three PONI angles and nothing else, so the two things it cannot express - an image mirrored in Y, and one mounted at a multiple of 90 degrees - had no home at all. They are now the DetectorOrientation carried by the detector setup, composed with the PONI rotation into one orthogonal matrix whose columns ARE the fast axis, the slow axis and the sample->PONI normal: lab = R(rot1, rot2, rot3) * Delta * ( (x-bx)*p , (y-by)*p , distance ) GetFastAxis/GetSlowAxis/GetNormalAxis read those columns and DetectorAxes() sets the plane from them, decomposing back to the angles; PoniRotMatrix and PoniAnglesFromMatrix are the conversion in both directions, exact on the canonical branch (rot2 in [-pi/2, pi/2]) and with a stated convention at gimbal lock. The angles stay stored rather than re-derived, so a geometry given as angles is written back as the same angles, to the bit. Delta is never inferred. In particular an arbitrary rot3 is NOT decomposed into a quarter turn plus a residual: rot3 is a fitted quantity, and a least-squares step must not be able to turn the stored image. It is set only where something states it - the detector setup, --detector-mirror-y / --detector-quarter-turns, or the value a file this system wrote records - and defaults to the identity, which makes the whole change a no-op for every existing detector and every existing file. It is a different setting from DetectorSetup::mirror_y, which flips the MODULE LAYOUT while an image is assembled and so decides what the stored pixels are. Merging the two would apply the mirror twice for every modular detector, or change the pixel content of every file written; both are ruled out. The new one earns its keep exactly where the old one is a no-op: a detector whose image arrives already assembled has no layout to flip. Both generators are signed permutations of the in-plane offset, so they preserve the distance from the PONI. That is why almost nothing downstream changes: everything needing an azimuth already goes through LabCoord, and everything that does not needs only a radius. The two hand-written copies of the rotation - XtalResidual and RingOptimizer - take the discrete part as four constants next to cos_rot3/sin_rot3, since it acts in the detector frame where rot3 acts in the laboratory and cannot be folded into it. RingOptimizer needs it despite being a radial fit: it fits the tilt, and the discrete part changes which way the tilt tips a ring. Carried as two optional CBOR keys and two detectorSpecific datasets, both back-compatible; the NXmx module axis vectors and the translation direction stop being hardcoded and are computed from it, reproducing today's values exactly at the identity. GetPoniRotMatrix is renamed GetDetectorMatrix, because it is no longer only the PONI rotation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5	9d3c2787f8	docs: credit the public diffraction data the pipeline is tested against ... Jungfraujoch is developed at one facility, so the reader and the reduction pipeline are exercised on data collected elsewhere - other detectors, other file formats, other conventions. That data was collected and published by other people, and until now nothing in the repository said so. NON_SLS_TEST_DATA lists all 51 datasets: the DOI to cite for each, the repository it came from, and the experiment as deposited. Beamline, resolution, space group and cell are the values deposited with the corresponding PDB entry, read from the RCSB data API - not results measured here; no quantity produced by this software appears on the page. The detector is read out of the image file instead, because the detector named in a PDB entry is often only approximate, and the twelve cases where the two disagree are listed rather than silently reconciled. ACKNOWLEDGEMENT gains a section for the repositories themselves, with the IRRMC, SBGrid and Zenodo citations and the PDB citation for the metadata. Every DOI on both pages was resolved before it was written down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 21:59:49 +02:00
leonarski_fandClaude Opus 5	84efb0666c	viewer: open a miniCBF sweep, the same way rugnux already does ... The native miniCBF reader added for rugnux is a plain JFJochReader, so the viewer only needed to be told which one to open a file with. JFJochImageReadingWorker held a concrete JFJochHDF5Reader; it now holds both readers and a JFJochReader* pointing at whichever the open file needs, chosen by JFJochCBFReader::CanRead. Naming any frame opens the whole sweep - the reader's own template matcher decides which frames belong to it, so a directory holding two sweeps or XDS auxiliary files is not spliced together. The previous file is closed before the new one is opened, so switching format between HDF5 and CBF in one session leaves nothing behind. Everything the viewer draws already goes through JFJochReader and JFJochReaderDataset, so nothing else had to move. The two HDF5-only features stay on the HDF5 reader: calibration images, and the reprocessing snapshots, which are metadata read back over the same images the reader is serving and have nothing to attach to for a directory of CBFs. A raw sweep therefore shows geometry and pictures with an empty run list, which is the state a live HTTP stream is already in. Reprocessing jobs run on a CBF sweep too - JFJochProcessController opens its own reader the same way, and Rugnux has handled a CBF source since the reader landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 21:49:09 +02:00
leonarski_fandClaude Opus 5	172a845cbb	rotation indexing: refine twelve candidate lattices instead of four ... The first pass refined only min(candidates, 4). The comment at the selection already conceded that the pre-refinement indexed fraction is an unreliable discriminator, and on one dataset degenerate cells occupied three of those four slots - so the right cell, ranked fifth, was never refined. The preceding commit's volume guard frees those slots but does not widen them. This is a one-for-one trade, measured by rebuilding with the old value to confirm the attribution. It gains one dataset: R_meas 27.2% to 23.8%, with mean I over sigma better in EVERY shell at identical shell edges, so it is not a resolution ramp. It costs another: a clean abort becomes a wrong four-fold supercell in P1 at 36% completeness. That is the same shape of trade already accepted for the plane-normal cap search in the preceding commit - an honest failure becoming a wrong answer - on a different dataset. It is committed separately so it can be reverted on its own if the maintainer weighs this instance differently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:35 +02:00
leonarski_fandClaude Opus 5	4e8db41785	indexing: refuse a coplanar candidate, and search the plane normal when the shortlist is flat ... Three related changes to the FFT candidate path, batteried together because they touch the same function. A COPLANAR CANDIDATE REACHED REFINEMENT. ReduceResults filtered triples on lengths and angles only - the 30-150 degree bound admits any flat combination - and there was no volume test. On one dataset 41 of 5535 candidates had |V|/abc below 0.05, with a clean decade gap to the next, and three of them reached the optimizer. UnitCell is float, and for a cell that flat the metric determinant is around 1.5e-7, so float32 gets its sign wrong 19% of the time where float64 never does. The guard against a negative argument to sqrt then CREATES the singularity it was meant to prevent: it puts c in the a-b plane, the reciprocal volume is 1/0, and the residual is 0 times infinity. Ceres reported a not-a-number Jacobian and wrote several hundred lines of solver output per failed solve. VolumeFraction() is |V|/(|a||b||c|), rejected below 0.02 - about 1.1 degrees off flat, ten times below the flattest real candidate observed and a thousand times above where float loses the sign. It is enforced at the producer and at the two optimizer entry points. Note the existing sanity checks use ABSOLUTE volume, which a 320 cubic-angstrom flat cell passes. The same reciprocal-volume division is now guarded at the two remaining sites that share the pattern. A SHORTLIST CONFINED TO ONE PLANE cannot close a cell, and the row it is missing is the plane normal. That is detected from the scatter-matrix eigenvalue ratio - measured, degenerate clouds score 2e-5 to 3.3e-4 against 0.026 or more for every non-degenerate one, a factor of eighty - and one further transform is spent with the same direction count inside a three-degree cap about the normal, so the plan and buffers are untouched. More directions cannot substitute: at the exact true direction the long axis ranks 1422 of 16384 by prominence while the shortlist cut is four times higher. Ranking, not sampling, is the obstacle. A four-fold denser grid was measured and rejected - it reaches the same answer to three decimal places and takes a run from 2.5 to 8 GB of device memory. fft_min_unit_cell_A is reachable as --fft-min-unit-cell and is lowered automatically by -C, mirroring how the maximum is already raised. The default of 10 is unchanged: a lower floor admits spurious sub-cells on protein data, and over 73 protein runs the floor was never lowered while the sibling maximum did fire twice, so the path is live and correctly inert. Corpus of 93 datasets, both arms, one build: 72 bit-identical on report content and p.hkl checksum, 13 failing identically, and the count of working datasets rises by one. The volume guard fires on 58 of 93 and 47 of those stay bit-identical - it fires constantly and almost never changes an answer, which is what it should do. Solver chatter falls from 919 lines across three datasets to none. The cap fires on 4 of 93, none of them in the in-house or private arms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:20 +02:00
leonarski_fandClaude Opus 5	9a66a21a10	rugnux: expose the indexed-spot gate and the third detector rotation ... --min-indexed-spots sets how many spots must lie on a candidate lattice for a frame to count as indexed. It is what the rotation first pass scores candidate lattices on, and it was reachable only by editing the source. The default of 9 is unchanged; on a sparse pattern, lowering it to 6 takes the first pass from 31 of 60 validation frames to 38. --rot3 completes the set beside --rot1 and --rot2. rugnux --mode calibration WRITES a .poni file containing all three, and without this flag a refined third rotation could not be given back - it had to be transcribed by hand and its convention undone. With it, a detector mounted rotated in its own plane is expressible from the command line. Also corrects the description of fft_high_resolution_A in the API, which claimed to be the highest resolution of spots USED. It filters nothing - every spot is projected whatever its resolution. It sizes the projection histogram, and with it the transform. Implementing the documented behaviour was tried separately and makes its own motivating crystal strictly worse, so the code is right and the description was wrong. (The generated Python client doc carries the same text and will pick this up at the next regeneration.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:59:03 +02:00
leonarski_fandClaude Opus 5	367eba55b1	reader: take a miniCBF rotation axis from the goniometer the header states ... Every miniCBF sweep was handed the same hardcoded axis regardless of what its header said, and Chi/Kappa/Phi/Omega were not parsed at all - there were no members for them. A sweep collected on a tilted chi cradle therefore ran with an axis that is 54.7 degrees wrong. The header angles and their increments are now read, the scanned axis is identified from the non-zero increment (the name is consulted only when no increment is stated, which is what absorbs the five different spellings the corpus contains, including one file that states no axis name at all), and a phi scan composes the head chain. An omega scan returns the base axis untouched, because a fixed chi cannot tilt the axis it hangs from. -9999 is a sentinel meaning "not set", not an angle. It is treated as absent, so it can never reach the geometry. The direction and sense are not invented: these files append an imgCIF _axis loop stating their own vectors, and SOURCE with GRAVITY fix the imgCIF-to-internal transform, which independently reproduces the transform this repository already documents for NXmx. Under it the file's own stated phi axis is exactly the composed one, to four decimals. Driving the real reader over all 39 corpus sweeps, 37 return the previous axis bit-identically - including every sweep carrying a large fixed chi, every sentinel header and every axis-name spelling. Only the two genuine phi scans move, and an unrelated rotation dataset is unchanged end to end. This is necessary but not sufficient for the one dataset that motivates it: with the axis corrected it still does not index, because that detector is also mounted rotated 90 degrees in its own plane, which the reader does not yet read. Compensating both takes its phi sweep from no indexed validation frames to 90.89% indexed and a complete merge, which is what shows this half is load-bearing. The detector mount and the two-theta swing belong to the detector-frame work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:50 +02:00
leonarski_fandClaude Opus 5	60c15d6931	reader: open a master whose beamline writes standards-correct NXmx differently ... Four defects, hit in sequence, that between them stopped eight masters from one beamline before any geometry question was reached. The files are correct NeXus; the reader was assuming one writer's conventions. ReadScalar demanded rank 0. NXmx puts no rank on distance, saturation_value, two_theta or det_z, and these files write them as shape (1,). Any dataspace holding exactly one element is now accepted; a genuine vector is still refused. frame_time was read unconditionally and NXmx does not require it. A virtual-dataset source filename of "." was resolved as a relative path, giving <dir>/. - but "." is HDF5's spelling for THIS file, and these masters compose /entry/data/data as a virtual dataset over datasets in themselves that are external links to the data files. The virtual source's DATASET PATH was parsed and then ignored in favour of a hardcoded /entry/data/data, while these data files keep their images at the root. Fixing that exposed a fifth: the positional-read fast path used the master's own path, but a dataset reached through an external link lives in another file and a chunk address is an offset into THAT file. An audit of all 38 masters in the non-SLS corpus finds exactly these eight need the change and the other 30 need nothing. Corpus A/B over 73 dataset pairs, base and patched back to back: 64 byte-identical results, 9 identical failures, none differing. The blast radius is bounded by construction - JFJochReader is linked only by rugnux and jfjoch_viewer, and the one shared header this touches only ever accepts more, so nothing that opened before can read differently. With it the eight files read completely: 12850 images, no decode errors, seven of them spanning two source datasets through the master's own external links. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:33 +02:00
leonarski_fandClaude Opus 5	4ef6bd9952	rugnux: decide the space group after the second pass, not the first ... Pass 1 searched for the space group on the geometry the run started with, stashed the answer, and reinstated it before pass 2's merge - so the decision was made on the worse of the two passes and then fed forward as a constraint. Four guards existed to reconcile it with what pass 2 later found. Now pass 1 does not search at all. It keeps its first merge, which is where mosaicity smoothing and geometry post-refinement already happen, and skips the in-symmetry re-merge that existed only to feed the search. prepass_merge_sg_, prepass_promoted_point_group_, the reinstatement, the primitive-to-conventional reindex it needed, and lattice_conflicts_with_prepass_sg all go. The obstacle was the pass-adoption guard, which compared completeness and CC1/2 across what could now be two different space groups. It compares each pass's FIRST merge instead - P1 on both sides, full range, before correction surfaces - which pass 2 produces anyway. One guard had to be replaced rather than removed. The index-time veto is re-keyed from pass 1's GROUP to its LATTICE and made one-directional: a centred pass-1 lattice against a primitive pass-2 one. Without it a C2 crystal fell to P1, and the obvious narrower fix - exempting a pass-1 P1 - would not have caught it, because that crystal's pass-2 lattice is monoclinic-P rather than triclinic. A centred lattice and its primitive sub-cell share a primitive volume, so the supercell arm cannot see that demotion. Corpus of 93 datasets, both arms at -N 6, every log validated: 78 comparable, 65 with a byte-identical p.hkl. Two crystals gain their reference point group, one gains its screw axis, one is lost. Per shell, over the 72 unchanged crystals, the median move is +0.25 points of R_meas and zero in <I/sigma>, CC1/2 and completeness. Load-matched compute is +0.4%, and the header-geometry fallback fires three times in the old arm against once in the new. What is lost is one crystal whose 222 the operator correlations still confirm: the merge chi-squared ratio moves 3% across a fixed bound at the refined geometry. A second crystal in the XDS harness crosses the H bound the same way. Both bounds are in SearchSpaceGroup and are recorded elsewhere as miscalibrated; recalibrating them is deliberately left to its own change. The comment at the long-axis rescue is corrected while here: the resolution setting only sizes the FFT histogram, and coarsening moves the true axis DOWN the direction ranking - 1996 to 16361 of 16384 - rather than making it more robust as the comment claimed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:17 +02:00
leonarski_fandClaude Opus 5	8634e9a72b	symmetry: score screw absences per zone, and choose centring by likelihood ... Two places where the absence tests threw away information they already had. The screw evidence was POOLED across axial rows, so one unmeasured row vetoed a confirmed one. A crystal reading h: +18/+24 and k: -6/+8 answered a silent P222 rather than "2_1 along a confirmed, b undetermined". Scoring each zone separately, and letting a zone abstain, is what POINTLESS has always done; the conditions are independent, so their log-likelihoods add. Centring was decided on a COUNT of net absences while screws used a Beta-tail likelihood. On one crystal that made F222 (1662 absent, 356 violations, absent class at 34% of present) beat C222_1 (1098 absent, 15 violations, 0.4%) purely because 1306 > 1083. Giving centring the same likelihood separates them by 1717 nats - F222 -357.9, C222_1 +1359.3, I222 -787.6 - and needs no bound: min_absent_observed and max_absent_present_ratio are untouched, and nothing was tuned to this case. The deposited structure for that crystal is C222_1. Corpus of 94 datasets, both arms at -N 6: 78 comparable, 73 bit-identical, point group correct on 68 against 66, space-group number on 54 against 51, nothing regressed. The 73 bit-identical results are also the determinism control - two binaries cannot agree byte-for-byte that often by chance. An earlier ranking key summed each group's single gating number and demoted a P2_12_12_1 to P2_12_12, because pooling three genuine screws reads weaker than two when the third row is shallower. Summing the per-zone log-likelihoods is what fixes that, and the whole corpus was re-run on the corrected binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:57 +02:00
leonarski_fandClaude Opus 5	10873eee7a	lattice search: reject an impossible character, and supplement every angle ... Two defects in the Bravais character walk, both of which silently cost symmetry. An impossible character MATCHED. The walk computes acos(cond_F/sqrt(A*B)); when the character is geometrically impossible the argument leaves [-1,1], acos returns NaN, and the acceptance test fabs(NaN - actual) > tol is FALSE - so the character is taken. One corpus crystal matched a monoclinic-C character whose implied cos(gamma) is 1.086. The type-boundary retry covered beta only. All three angles carry the Niggli boundary, and negating two basis vectors supplements the OTHER two, so each angle needs its own flip. A crystal whose reduced gamma sits at 89.900 degrees needs the alpha/beta flip and never got it. Measured on 84369 exact lattices spanning all 14 Bravais classes: nothing is lost in any class, and orthorhombic-I recovery rises from 60.3% to 87.2%. Over 440000 random and perturbed cells the first change only ever demotes a monoclinic-C match to triclinic and the second only ever promotes out of triclinic - it is a strict superset of the beta-only retry. On the 94-dataset corpus, two crystals gain their correct point group and none regresses. Both regression tests fail without the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:41 +02:00
leonarski_fandClaude Opus 5	1ea6ba25be	reader: open a master that states its goniometer axes in one place and their angles in another ... A hybrid master carries an NXmx /entry/sample/transformations group holding one EMPTY SUBGROUP per axis - the direction as a vector attribute, no NX_class, no units, no angles - beside a legacy /entry/sample/goniometer group holding all the angles. Opening it failed outright with "Cannot open HDF5 dataset /entry/sample/transformations/omega": the existing legacy fallback keys off Exists("/entry/sample/transformations"), which is true here, so it never fired, and the axis stub was then opened as if it were the angle dataset. Present is not the same as usable. GoniometerGroup now takes transformations only if it holds at least one DATASET, and ReadAxis asks IsDataSet rather than Exists, so a member that is not a dataset can no longer be read as one. HDF5Object gains that predicate, in the style of the neighbouring Exists. The stub is the load-bearing half, not merely the thing that crashed. In the legacy branch, before falling back to the assumed (-1,0,0), the reader now looks for the NXmx stub and takes its stated vector. With it the axis is (0,-1,0) and the run indexes 60/60 validation frames; with the stub deleted the assumption applies and the same file indexes 0/60 on both schemes and both signs, and the run stops with no lattice. So without this half the fix would have turned "cannot open" into "found no lattice" - a differently shaped failure, not a success. The direction stated here is 90 degrees from the assumption, not merely its negation, which the rotation first pass could have recovered on its own. The vector size check moved out of the first branch so it now covers every path that produces one. Verified: the file processes end to end, 100% of frames indexed, cubic cell 105.87 against a deposited 105.88 (0.009%), space group reported as I23 or I213 - correctly refusing to choose, since the reflections that separate them are extinguished by the I-centring and were never measured. A both-layout master reprocesses unchanged (100% indexed, P212121). [HDF5] passes: 2194 assertions in 91 cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:47:41 +02:00
leonarski_fandClaude Opus 5	6319d5c600	indexing: tell an indexer that failed apart from one that found nothing ... IndexerThreadPool collapsed two different outcomes into the same empty reply. A worker that threw set result = nullptr, and so did a dispatch that never found a free worker; both then became a default-constructed IndexerResult, indistinguishable downstream from an indexer that ran and found no lattice. The first says nothing about the frame at all - the indexer never looked, and it will fail again on the next one - while the second is a real negative result about the crystal. The visible cost was the advice a failed run gave. With the card full, rugnux printed "Indexer thread 0 failed: CUDA (GPU) error" and then ended with "Two-pass rotation indexing found no lattice. Check the beam centre (--beam-x / --beam-y), raise --max-spots ..." - sending the operator to look at geometry that was never wrong, for a machine that was simply out of memory. None of those remedies can help when the frames were not examined. IndexerResult gains an optional error, set by the worker and by the pool's own catch; the remaining nullptr path keeps its meaning of "not attempted" and deliberately carries no error. RotationIndexer records it and exposes GetIndexerError(), and rugnux's first pass branches on it at the throw site, naming the resource failure instead. The error travels as DATA through image_analysis/ rather than as an exception, because RotationIndexer::RunIndexing() is on the online path - IndexAndRefine calls it on a schedule from the broker and the receiver, where dropping a frame is the right failure and killing a live acquisition is not. Only rugnux, which owns the "this run is over" decision, turns it into one. The failure result is byte-identical to the default-constructed one it replaces, so any_executed and the online frame-drop behaviour are unchanged. The new IndexerError category exists because the category is only the display prefix on what() - Category() is read nowhere - and the old line read "Processing failed: Input parameter invalid" for a GPU fault, which is the same defect one layer up. SpotFinderError is the precedent. msg.indexing_result is deliberately left alone: of its three states, "not attempted" is the honest one for a frame the indexer never examined, and asserting false would be the same collapse again. Verified on a squeezed card (246 MB free, -N 4): exit 1, zero dropped frames, the new message, and no mention of --beam-x. On a free card, against the previous binary at -N 6 on the same input, the logs differ only in paths and timing - same space group, cell and merge statistics. Targeted Catch2 cases pass, including three that drive the pool through the online path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:30:28 +02:00
leonarski_fandClaude Opus 5	92ad304f15	rugnux: fail on an exhausted GPU instead of dropping the images it could not process ... A per-image worker caught every std::exception and continued to the next image. That is right for one undecodable image and wrong for a resource fault: a CUDA out-of-memory says nothing about the image that happened to be in flight and everything about the machine, and it recurs on the next one. The run therefore logged an error per image, skipped each of them, and exited 0 with a dataset that was silently short - the merged numbers all moved, and nothing in the exit code or the summary said so. Observed with several processes sharing one card: two arms of the same comparison lost 21 and 15 images, by different amounts, so the arms no longer saw the same data. IsFatalResourceError rethrows for std::bad_alloc and for a JFJochException of category GPUCUDAError or MemAllocFailed, and every per-image catch consults it first. There are ten of them, not the three the obvious grep finds: besides the analyze/integrate/load sites in the two image loops, the first-pass spot read swallows the same way at Warning severity, the pre-scan swallows it in three places, and one geometry-refinement site caught it with no log line at all. The first-pass one matters most - those frames build the reciprocal-space cloud the indexer runs on, so losing them thins the input that decides whether the crystal indexes, and it does not change IMAGES_PROCESSED, so a frame-count audit cannot see it. Every one of these loops joins with future::get() or ParallelFor's RunTasks, both of which hold the first exception and rethrow it at the join, so the throw propagates and nothing terminates. JFJochException gained a Category() accessor for the test. Verified by squeezing the card to 804 MB free and running at -N 4: the run now exits 1 with zero dropped-image lines, where the same test on the previous binary produced the silent skips above. The broker and receiver paths are deliberately untouched - dropping a frame is the better failure there than killing a live acquisition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 14:58:04 +02:00
leonarski_fandClaude Opus 5	dce8d877d0	Scaling: make Run() idempotent, and re-calibrate the promotion it was hiding ... RotationScaleMerge::Run() resumed its alternating-least-squares loop from wherever the previous call left off instead of restarting from the ingested data. A de-novo run calls it four times, so pass N was scaled at 3N iterations, not 3. Two identical calls on the same object produced merges correlating at 0.847 with ZERO bit-identical intensities; they are now 68010 of 68010 identical. Where that mattered is narrower than it sounds, and worth stating exactly. In a determined point group the fit is stationary from iteration 2 - over the battery the collapsed-scale guard drops 377 frames at three iterations and 397 at twelve, and 30 of 39 crystals drop nothing at any count - so production merges were barely touched, 38 of 39 rows character-identical. But the SEARCH merges in P1, where a reflection has two or three observations, and there the fit never reaches a fixed point at all: the same guard drops 417 frames at one iteration and 1101 at three. A factor 1.05 against a factor 2.6, same code. So the defect lived in the one arm whose output the user cannot recover from downstream - the arm that picks the space group. Fixing it moves the statistics the promotion gates are calibrated on. The population as a whole does not shift (median change 0.9-2.7% over 280 matched candidates) but the tails do, p90 by 8-14%, and that is exactly where a promotion is decided. On the two crystals that decide, the H statistic's two populations SWAP ORDER: a genuine tetragonal 4->422 goes 1.685 -> 1.819 while a trigonal 3->32 twin law goes 1.906 -> 1.665. No H bound exists any more that keeps the genuine case and refuses the twin - the noise the loop was adding had been acting as a brake. Of the four statistics only b and R_meas still order the two correctly. Set max_systematic_b_ratio 1.90 -> 1.78, by a rule written down before the population statistic existed: take the statistic with the largest spurious_min/genuine_max on the population its gate actually sees, put the bound at the geometric midpoint. That is sqrt(1.711 * 1.853) = 1.78. Battery 36/39 space groups matching XDS, the same as before the scaling fix and one better than the scaling fix alone; the three mismatches are the known references where XDS is wrong. The refusal now reaches the user by name. The bound's margin is 4% either side - SMALLER than the p90 shift this very commit produced in that statistic. It separates the two crystals that exist and is not robust to another input shift, which is written into the header rather than left to be discovered. Two hazards go with it: re-measuring it must hold the scaling correction surfaces fixed, since changing the iteration count re-rolls their cross-validated on/off decision on 7 of 39 crystals for ~0.3 in ISa, all-or-nothing and non-monotone; and --scaling-iterations stays at 3 on both arms, confirmed rather than assumed (production 2->3 moves <I/sigma> +1.38% and 3->6 only +0.13%; the search gives the identical space group on all 39 crystals at 2 and at 3, and loses one at 1). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 22:44:21 +02:00
leonarski_fandClaude Opus 5	fa4aa30a1e	reader: match the legacy companion suffixes exactly, not any underscore ... The legacy DECTRIS layout stores five companion scalars beside each goniometer axis - AXIS_end, _start, _increment, _range_average, _range_total - and only the bare name is the axis itself. The first cut rejected _end and _range, which left _start and _increment to be offered to ReadAxis as axes in their own right. Matching the full suffix set, anchored at the end of the name, also keeps a genuine two_theta axis readable, which a "contains an underscore" test would not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	df3696e6e2	reader: fall back to the pre-NXmx key names, so an Eiger 1.x master opens ... Firmware 1.x writes the same three values under different names. Try the NXmx spelling first and the old one only if it is absent: detector/distance <- detector/detector_distance detector/saturation_value <- detectorSpecific/countrate_correction_count_cutoff sample/transformations <- sample/goniometer This cannot change what a current file reads: every modern Eiger master carries BOTH spellings. Measured on thirteen masters from ten facilities, firmware release-2020.2.1 through release-2024.1.1 - all of them write detector_distance and countrate_correction_count_cutoff beside the NXmx names, and a goniometer group beside the transformations one. The goniometer is the one that matters. A goniometer is only ever set from that one group, so a file whose axes are somewhere else was not an error - it was read as STILLS, silently, and the run completed with the wrong answer. The old layout also tags no axis with transformation_type and gives no vector, both of which ReadAxis required, so absence now means two different things by layout: in a transformations group it still means "not an axis" (that is how AXISNAME_end and the width scalars are skipped), while in the legacy group every leaf IS an axis and the companions are recognised by name instead. A missing direction defaults to the one every DECTRIS master since has written and says so in a warning rather than assuming it silently; a wrong guess there does not index, so it is visible, and the rotation first pass tries the opposite sign anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	a7615c8778	compression: read plain LZ4 (HDF5 filter 32004), and name the filter when one is unsupported ... DECTRIS Eiger firmware 1.x compressed its images with the HDF Group's plain LZ4 filter rather than bitshuffle, and files from that era are still what a repository hands you. Read-only: nothing here produces it, and the new enumerator goes last so the existing values do not move, the enum being part of the CBOR stream. The framing is the same as bitshuffle's - a 64-bit big-endian total size, a 32-bit big-endian block size in bytes, then each block prefixed by its 32-bit big-endian compressed size - so only the unshuffle step differs. Verified on a real chunk: the declared total matched width*height*4 exactly and the block walk consumed the chunk to the byte. It cannot reuse the bitshuffle path, which requires the block to be a multiple of BSHUF_BLOCKED_MULT elements: these files put the whole image in ONE block, which is not. The unsupported-filter message now names the filter it found and the ones it knows. Before, an Eiger 1.x file did not report a codec problem at all - it reached spot finding with nothing decoded and failed as "0 spots from 60 images" then "found no lattice", which reads as a crystallography failure rather than a format one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	84d14696b1	rugnux: accept a CBF sweep, decide the rotation axis sign, and search as far as a given cell needs ... Three changes to the rotation path, all in the same first-pass block. Input. Rugnux now holds a JFJochReader rather than the concrete HDF5 reader, so a sweep can come from a directory of miniCBF as well as from a master file; the CLI picks by looking at the file. Only the _process.h5 image links need the concrete reader and they ask for it by dynamic_cast - there is nothing for a virtual dataset to point at in a directory of CBFs, so that file writes its images instead of linking to them. --mode scale refuses a CBF sweep, there being no stored reflections to re-scale. Search bound. A reference cell longer than the FFT bound cannot be represented by the search meant to find it, so the run reported "found no lattice" - strictly worse than not giving the cell at all. Raise the bound to cover a given cell (margin 1.3, as DIALS uses when a cell is known), and raise it in the long-axis rescue too, where it costs nothing: that pass already coarsens the resolution to 3.5 A, which shrinks the transform by 1.75x, so spending it back on reach leaves the rescue at 1.14x the standard pass. The rescue's constrained re-index gets a pool wide enough to hold what the coarse pass recovered, which it did not before. ROTATION ONLY - stills fire the FFT once per image across every worker and routinely run with a known cell, so widening the transform there would cost the whole serial run for nothing. Clamped rather than throwing: the rescue can recover an implausible axis, and a bad candidate must not take the run down with it. Axis sign. The rotation-axis sign is a convention the input often cannot settle - a miniCBF header names the axis but gives no direction, and an NXmx vector is only meaningful together with the detector orientation, which nothing here reads. It is decidable from the data, though: the wrong sign does not index at all (measured on one sweep, changing nothing else: every frame indexed one way, none the other). So after a poor first pass, try the other sign and keep whichever indexes more validation frames - the same count the scheme choice already uses, so no new metric and no new threshold. It flips the AXIS, not the angles, because prediction reads the axis too; the decision then holds for the rest of the run. Runs before the long-axis rescue, since with the sign wrong every candidate lattice is wrong. Only after a poor pass, so a correctly-signed file costs nothing, and spot finding is not repeated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:41 +02:00
leonarski_fandClaude Opus 5	dc16a00271	reader: read a PILATUS miniCBF sweep natively, without libcbf ... Most facilities still archive rotation data as a directory of miniCBF frames, which until now had to be converted to HDF5 before rugnux could see it. Nothing in that format needs a CIF parser or a library: it is an ASCII header, four separator bytes, then one byte-offset compressed image, and every value the reader wants sits on a "# " comment line or a MIME line. MiniCBF holds the format itself - header parse and the byte-offset decoder, which is a running value with deltas stored smallest-container-first. Verified byte-exact against dxtbx on PILATUS 6M, 6M-F, 300K, silicon and CdTe sensors, and three sensor thicknesses. JFJochCBFReader is a sibling of JFJochHDF5Reader under the JFJochReader base. NAMING ANY FRAME READS ITS WHOLE SWEEP: the sweep is identified by the template (prefix + digit count) the named frame belongs to, not by "every .cbf in the directory", so a directory holding two sweeps does not splice two crystals together. Naming a directory takes the sweep with the most frames in it. Images decode on demand, one per call, so any number of workers can read at once - there is no global lock as there is on the HDF5 path, HDF5 not being thread-safe. A raw CBF carries no analysis results, so the dataset it builds is the geometry, the mask and nothing else, exactly as a plain DECTRIS file with no /entry/MX gives. Two header quirks are handled because real files have them: the sensor material is written "Silicon" where the rest of the code compares against "CdTe", and the thickness unit is sometimes omitted. Headers are not a fixed size either - one set carries 6335 bytes - so the parse runs to the binary separator rather than over a fixed prefix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:36 +02:00
leonarski_fandClaude Opus 5	e78fbd55b7	Indexing: let the FFT search reach past 500 A, and stop clipping the peak background ... fft_max_unit_cell_A was both the default and an enforced check_max, so 500 A was the longest basis vector the FFT could ever return: FFTIndexer sizes its projected histogram from that value and the transform's last usable bin IS that length. Of the PDB's 206950 X-ray entries, 1091 (0.53%) have an axis longer than that and were unindexable by construction. The accepted range now goes to 1200 A, which leaves 8. The DEFAULT is unchanged at 500 - the histogram is sized from the value in use, so nothing pays for the wider range unless a caller asks for it. The peak picker's running-mean background was truncated at the ends of the spectrum rather than slid inward, so a peak within bg_half (~15 A) of either end - which is exactly where the longest cells sit - was judged on a one-sided background, biasing its prominence by however much the spectrum sloped there. Keep the window a constant width and slide it. Both bounds stay monotonically non-decreasing in j, so the GPU kernel's running sum is still valid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:36 +02:00
leonarski_fandClaude Opus 5	4951538368	rugnux: measure the spot budget instead of taking it as given ... --max-spots was a fixed 1000, and on a rotation sweep it sets the DENOMINATOR of the per-frame acceptance gate, which admits a frame when at least a fifth of its spots index. Detections are not all reflections: on a strongly diffracting crystal with heavy solvent background there are 2372 a frame, 1416 of them on ice bands, and only 178 index - so a budget that takes essentially the whole list puts the entire frame population on the gate (median indexed fraction 0.271, tenth percentile 0.218) and the run integrates 83% of its images. The same run with a smaller budget gains 211 frames and loses none, and the frames it gains are the ones with the MOST detections. That is how a larger budget integrates fewer images. So measure it: over the first pass's validation frames, with the sweep's lattice known, tally each spot by rank as +1 if it lies on the lattice and take the budget at argmax over N of n_indexed(N) - 0.20 * n_counted(N) which rises exactly while spots at that depth index better than the gate's own floor and falls after. The 0.20 is that floor, not a new constant - it is lifted out of the function-local it already lived in. It means: as deep into the intensity-ordered list as the image is still showing reflections of THIS crystal. The argmax alone would not do. Under the null that spots index at the same rate at every depth the tally is a driftless random walk, whose maximum is positive whatever the data, so a bare argmax shortens every dataset. The budget therefore has to clear the walk's own noise: the quantity it acts on is the fall from the peak to the end of the list, which is that walk read backwards, and the reflection principle gives its null law in closed form - P(fall > z*sqrt(g(1-g)T)) = 2(1-Phi(z)). That already pays for the search over ranks, so nothing further is owed to multiple comparisons. One false cut in a thousand measurements - a twelfth of one over a 39-crystal two-pass corpus - fixes z at 3.29. The level was chosen before the rule was written and was not revisited afterwards. Battery, same build, one changed default: 36/39 space groups in both arms, none lost, and 36 of the 39 crystals BIT-IDENTICAL. Two move materially - the strong crystal by +17.4% observations and +35.9 points of CC1/2 at 1.58 A, its indexing rate 83.4 -> 95.1%, and another by +93.5% observations with completeness 76.6 -> 98.4%. The significance requirement is what makes that list clean: it removed the one crystal the unguarded rule regressed, and with it two of the five gains, whose peaks do not clear 3.29 sigma on 60 frames. The lever for those is the sample and not the threshold - power grows as the root of the frame count while the bar stays where it is. Rotation only, and the first-pass lattice search always sees the full list: a fixed --max-spots 250 would have been much cheaper to write and fails a battery crystal outright, starving the de-novo FFT into a wrong cell that indexes 7 of 60 frames. Stills never reach the code path - verified bit-identical merged reflections on a serial set - and --max-spots N still pins it, as does the library default the broker and the FPGA use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 19:02:02 +02:00
leonarski_fandClaude Opus 5	df98bb8789	Space-group search: ignore the frames the crystal barely diffracted on ... A crystal that repeatedly leaves the beam over a full turn was read as P2(1) where it is P4(3)2(1)2, on data that merge at CC1/2 96-98% once that symmetry is assumed. The cell was right, the indexing was right - 83% of images - and every 422 operator correlation still came out between 0.08 and 0.38 against a gate of 0.30. The per-frame scale enters the intensities as 1/G. On this crystal the fitted scale spans 356x between its 5th and 95th percentiles and 15% of frames sit below a tenth of the run median, so those frames arrive amplified ten- to a hundredfold - and their sigmas are amplified by exactly the same factor, so nothing weighted by sigma can see it. What saves the PRODUCTION merge is multiplicity: in the determined symmetry a reflection is measured ten or twenty times and --reject-outliers throws the amplified observation out. The search merges in P1, where a reflection has two or three observations and no majority exists to call any of them an outlier, so the bad value lands in the merged intensity at full weight and the operator correlations pay for it. So drop those frames from the search merge only, beside the |zeta| filter and through the same snapshot that undoes it before the production merge, which keeps every frame. A tenth of the median because that is where the two populations sit: over the 39-crystal battery 33 crystals have NOT ONE frame below it, so the filter is inert on them by construction; of the six that do, five spend 0.1-3.5% of their frames there against this crystal's 15%. A twentieth leaves it under the gate; a fifth starts costing frames the battery says are real, the smallest legitimate min(G)/median(G) measured being 0.070. Battery 35/39 -> 36/39 space groups matching XDS, with 38 of the 39 rows character-identical: the only crystal that moves is this one. On it the operators go to 0.37-0.62, and every one improves in every resolution shell, most at low resolution - the signature of a scale error removed rather than noise removed. The run then merges to 1.41 A instead of 1.67, at CC1/2 97.7% and completeness 99.7%. Four other explanations were measured and refuted first: the search resolution cut (the correlations are flat in resolution, and a battery crystal with a finer detector and no cut at all gets 422 without trouble), the space group being inherited from the pass-1 geometry (handing the refined geometry in from the start changes nothing), the ordering of the corr snapshot around the collapsed-scale guard (0.02 in correlation, nothing in space group), and the ice-ring flagging (removing it entirely leaves the answer at P2(1)). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 17:20:36 +02:00
leonarski_fandClaude Opus 5	9f2696c1ac	Adaptive spot finding: the ring threshold replaces the floor, not the local test ... The self-calibrating finder was meant to replace the classic finder's FIXED PHOTON FLOOR with a per-resolution-ring threshold read off the image's own noise. As written it replaced the local-box SNR test as well, and that is the defect: a whole-ring threshold is an ABSOLUTE contour with no feedback from a pixel's own surroundings, so the area a spot puts above it grows as sigma^2*ln(peak/threshold) and never saturates. Measured on a strongly diffracting rotation set, the detected footprint grows by +8.05 pixels per e-fold of peak, so the brightest reflections came out as 100-500 pixel blobs and were then discarded for exceeding the size bound - every one of the ten strongest on an image. Intersecting with the local box gives -0.24 pixels per e-fold, the classic finder's own number to two decimals. WHY the local box is the right partner, rather than merely the incumbent: it is a prominence rule whose reference level is a 961-pixel mean. A spot inflates the box's own variance and the peak divides out of the acceptance test, so it cuts at a fixed FRACTION of the spot's own height. Referring that level to fewer pixels makes it inherit their shot noise - at FIXED footprint, estimating the level from 961 pixels, from 25, and from the single maximum gives centroid residuals of 0.524, 0.539 and 0.656 - so flat growth and a stable centroid turn out to be two ends of one dial. A contour on the bare maximum has the flattest growth of anything tried (+0.1) and merges worst. The two arms bind in different regimes, which is why intersecting beats choosing: on serial stills the ring threshold is 0.6x the classic floor, on this rotation sweep 2.3-6.0x. Stills are a strict no-op - 175 components against 175, identical per frame - so the +40% in stills indexing that the adaptive threshold was introduced for is untouched. What it buys, stated as one fact rather than two. Across five geometry pins spanning 1.1 mm it indexes the most frames of any arm tried, 0.831 against 0.803, and integrates 3.04 to 5.76% more observations - but those are the SAME number: regressing observation count on indexing rate over four arms leaves residuals of +/-0.7 percentage points against swings of -7 to +4.5%, so the extra observations ARE the extra indexed frames, not better data per frame. CC1/2, the only statistic here carrying per-observation quality, is +0.66 at one pin and -0.06 at the other: not harmed, not improved. <I/sigma>, ISa and R_meas cannot arbitrate on this data - across those pins each crosses zero as a monotone function of the pin. WHY an absolute contour indexes fewer frames, when its spot list is equal or better on every axis measured - recall, top-1000 recall, centroid, ice fraction, component count - is the interesting part, and it is not a detection effect at all: ITS OWN SIZE BOUND DELETES THE BRIGHTEST REFLECTIONS ON THE FRAME. A component is discarded because it grew past 200 px, and it grew past 200 px because it was bright, so the deletions are drawn from the head of the indexing budget rather than uniformly from it: they are 11x enriched in the top 250 of the thousand spots handed to the indexer, and the bound's own real deletions sit at MEDIAN RANK 12. Turning the bound off recovers 66% and 50% of the deficit at the two pins, against a bar registered at 33% before the run. Three of us dismissed this for most of a day on the grounds that the gates delete only ~4% of what is detected. That arithmetic was right and the denominator was wrong - a rate is not an impact when the thing being lost is selected for the property that makes it matter. Reworking the bound instead was measured and rejected: it recovers half the deficit, and it cannot be done without re-admitting what the bound is for - 68 components past 200 px, of which 8 are real and 60 are junk, where the intersect gets the 8 without the 60. The residual once the bound is off, +1.08%/+1.70%, is the contour itself. Component merging is ruled out separately: geometrically impossible here, 33.9 px minimum reflection separation against components spanning 10 px. So is a ranking effect - the intersect's lead runs +0.06% at --max-spots 250, +3.46% at 1000 and +14.26% at 2000, which is backwards for a selection artefact. Costs 0.48 ms per image in the finder, and 0.044 px of bright-spot centroid precision - measured convention-free, by fitting a line to a reflection's own centroid across five frames, after an XDS-referenced figure proved to be four fifths aperture convention. It also makes the compactness gate above it safe. On the absolute contour that gate is net damage, deleting 37 genuine reflections per ten frames; once the footprint stops growing nothing reaches its threshold at all. Also fixes a real but unexercised defect in PoissonThreshold, where the exact tail handed over to a normal approximation with a step. It changes nothing here: the clipped ring sigma is over-dispersed 1.2-4.9x against sqrt(mu) because it still contains diffraction, so the Gaussian arm wins every ring above mu=50 and none of the 522 thresholds move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 15:30:39 +02:00
leonarski_fandClaude Opus 5	844b461232	Spot finding: say honestly that the compactness gate is inert ... The commit that added it claimed the shape test was what made raising the size bound safe - "with a flat 200 alone, two battery crystals moved; with the shape test both are unchanged to three decimals". That is wrong. Those two runs came from different builds, and the comparison never isolated the gate at all. Isolated properly, gate on against gate off in ONE build: byte-identical .hkl on three rotation crystals, the strongly diffracting one included. Not merely close - identical, every digit of every merged reflection. On that crystal the gate rejects 5 to 22 of the 110 to 440 oversize components a frame, so essentially all of what that commit bought, it bought by raising the bound from 50 to 200. Keep it, but for the reason that survives rather than the one that did not. The raised bound admits components up to 200 pixels, and on data carrying ice arcs, cosmic-ray tracks or a lit detector row those are what arrives; none of the crystals measured here carry that population in quantity, which is exactly why the gate reads inert on all of them. So it bounds the SHAPE of what the larger size bound now lets through, and no experiment on these crystals can validate it - an inert gate scores identically whatever its threshold. The comment also carried a "keeps 97% of the genuine wide reflections and none of the ice arcs" that came from a simulation over component lists, not from the pipeline. Removed rather than restated: the pipeline measurement says no reflection anywhere changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 15:04:05 +02:00
leonarski_fandClaude Opus 5	0af109b838	Spot finding: bound how large a spot may be, not how bright ... A component of more than 50 connected pixels was discarded. Under the self-calibrating threshold that is an INTENSITY CEILING, not a size bound: the threshold is an absolute per-ring contour, so a component's area above it grows as sigma^2*ln(A/T), without bound in the peak amplitude. Measured on the strong rotation set, footprints run 3 px at 30-100 counts to 50 px above 10000 - a slope of 4.4 px per ln(peak) against the fixed local-box test's 0.8, which saturates at 13 px and never reaches the bound at all. So the brighter a reflection, the more certainly it was thrown away: 59% of the box finder's d>3 A spots were missing from the adaptive finder's list, including every one of its ten strongest, at an intensity ratio of 1.085 for those that did match. Indexed spots per image collapsed from 220 to 7. Raise the bound to 200 - CrystFEL peakfinder8's --max-pix-count, the only directly comparable number in the field; XDS has no such parameter and guards on shape instead - and ask a component above 50 pixels to be COMPACT: it must fill a fifth of the square its bounding box fits inside. A Bragg reflection is round and fills about half of that square however bright it is; an ice arc, a cosmic-ray track or a lit detector row fills a fifth or less, and those are what an upper bound was ever protecting against. Below 50 nothing is asked of the shape, so every component accepted before still is. Integer arithmetic on both sides, and on the GPU the bounding side fits in what was padding, so the device struct does not grow. The shape test is what makes the raise safe. With a flat 200 alone, two battery crystals moved: one lost a little I/sigma, and the other's de-novo lattice was NOT MONOTONE in the bound - correct at 50, 100 and 200, wrong at 150 and at 250 and above - so 200 was partly luck. With the shape test both are unchanged to three decimals. De novo at defaults on the strong set: indexing rate 0.574 -> 0.804, completeness 97.3 -> 99.6%, <I/sigma> 3.42 -> 6.07, R_meas 0.299 -> 0.249, CC1/2 0.947 -> 0.961, ISa 3.29 -> 4.13. The full 37-crystal battery, scored per shell, is still owed. Also documents what StrongPixelSet::AddStrongPixel has required since the component search became linear - pixels in raster order - and puts the existing test's insertion order into it. Both callers scan a bitmap in ascending flat index and always satisfied it; the test did not, and was the only thing that did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 11:49:49 +02:00
leonarski_fandClaude Opus 5	6516bc96af	Ice rings: carry the list past 1.5 A, where ice does not stop ... The eleven measured bands end at 1.522 A because their source says so in its own words - "pure hexagonal ice has 11 diffraction rings between 4 and 1.5 A resolution" - and its subject was detecting ice in deposited data, not masking it. On a detector that reaches further, the rings it does not list are the ones left in the data: on the strong rotation set just added to the battery, 44% of every image's spots sit in ice bands, and beyond 1.5 A the spot list is ice and nothing else, which is why the resolution estimate read the ice rather than the crystal. There is nothing measured to copy below 1.522 A, so the eight added bands are calculated. Enumerating hkl is not enough and the code already said so: ice Ih is P6_3/mmc with O on 4f, and most of what enumeration emits is extinguished by the OXYGEN SUBLATTICE rather than by the space group - which is why (004) at 1.830 A and (104) at 1.657 A are missing from the measured list although they sit inside its range and its reflection conditions allow them (for (00l) the structure factor goes as cos(2*pi*l*z), and z ~ 1/16 kills l = 4). So compute structure factors - oxygen only, the hydrogens being half-occupancy disordered and weak to X-rays - and keep the lines reaching 3% of the strongest. That rule REPRODUCES THE MEASURED ELEVEN EXACTLY and every line it drops inside their range computes to zero, which is what makes it trustworthy below 1.522 A. It stops at 1.170 A: below that the real lines fall to 2-3% while the extinct ones rise to about 1%, and an oxygen-only calculation cannot separate them honestly. Every added band was independently confirmed in the data - the spot-count histogram of the strong set peaks at each of them and is empty between - and every line the rule calls extinct is absent there too. Costs, measured. The bands are inert above 1.6 A: on 38 of 39 battery sets the profile ice score does not move at all, and the one that appeared to (a jet set, 1.25 -> 2.60) does not on the peak-excluded profile the score actually uses - that was Bragg peaks in the plain profile, which is what the peak exclusion is for. Where a detector does reach past 1.5 A the bands cover more of reciprocal space: unchanged at 1.6 A, +7.4 points at 1.4 A, +16.5 at 1.18 A. On the strong set that is 17% -> 27% of reflections held out of the scale fit, and it shows: the spot resolution estimate improves from 1.33 to 1.46 A against a truth near 1.42, while CC1/2 falls 98.5 -> 97.6% and ISa 3.58 -> 3.37. Ice handling only runs at all on a run that trips the ice gate, so a clean crystal pays nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 11:27:28 +02:00
leonarski_fandClaude Opus 5	fe1f3ba96e	rugnux: report the geometry in the form the broker takes it in ... A rotation run post-refines the detector distance and the beam centre, and that is usually the best measurement of them anyone has - but the only way back into the instrument was to read two numbers off the report by eye and retype them into a collection. Write them once more, as the object jfjoch_broker takes: the four required properties of dataset_settings in broker/jfjoch_api.yaml, spelled the way the API spells them, on one line of valid JSON that a script can lift with a grep and POST. The existing DETECTOR_DISTANCE and BEAM_CENTRE keys are unchanged and stay the ones a person reads. REPORT_VERSION is not bumped: a new key breaks no consumer of the old ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:49:18 +02:00
leonarski_fandClaude Opus 5	42add7c0e2	Report the strong-pixel count off the FPGA path too ... DataMessage::strong_pixel_count is written all the way to CBOR, to /entry/MX/strongPixels, to the receiver plots and to the frontend - and only the FPGA receiver ever set it. Every EIGER dataset, and every rugnux run, stored zeros. It is the one number that distinguishes an image the spot finder gave up on from an image that did not diffract, and its absence is why a defect that cost 767 of 1800 images their spots looked like a crystal that stopped diffracting for two blocks of the sweep. Fill it on the CPU/GPU path as well: the finders know the count, the GPU extractor already had it on the device, and it rides back with the spot count on the frame's one synchronisation, so it costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:46:47 +02:00
leonarski_fandClaude Opus 5	1388b16d7a	Resolution estimate: predict past the edge of the detector ... The spot-finding resolution estimate was clamped so it could never beat the detector corner. On a crystal that diffracts past the corner that reports where the DETECTOR stops, which is the one thing this number is not for - it is meant to say how far a merge of data like these would reach, a property of the crystal and the exposure. The clamp also hid the interesting case: an estimate finer than what the run actually merged is the statement "this run was detector-limited", and there was no way to make it. The statistic already extrapolates. Its quantile sits in the middle of the fall-off, well inside what the detector records, so it goes on measuring the crystal's own decay when the detector cuts that decay short. Measured by truncating the spot lists of 31 battery crystals at an artificial detector edge and scoring the unclamped answer against each crystal's own measured CC1/2 = 0.30 crossing, it holds its 8-9% floor out to about 1.7x past the cut and only then drifts pessimistic, which is the safe direction. Every genuinely detector-limited crystal in the battery needs between 1.10x and 1.63x. Against a truth corrected for censoring - the six crystals whose merge is cut off by their own detector cannot have a measured crossing, so theirs is extrapolated from multiplicity-corrected <I/sigma> and anchored on the 25 where both exist: symmetric-log RMS 13.5 -> 9.4% over 37 crystals, 25 -> 28 within 0.2 A. On the six detector-limited ones 26.1 -> 9.8% and the bias goes +19 -> -3%; on the 31 that are not, 9.23 -> 9.32%, i.e. it costs them nothing. The 0.30 tail fraction and the 2.25 reach were refit by leave-one-out against that truth and did not move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:41:34 +02:00
leonarski_fandClaude Opus 5	ac202a55a1	Spot finding: the strong-pixel limit follows the detector ... An image with 65535 or more strong pixels was given up on and reported ZERO spots - silently, no log line, indistinguishable from a frame that did not diffract. 65535 is one pixel in 64 of the JUNGFRAU 4M the number was written for; left fixed while the detectors grew it became one in 276 of an 18-megapixel EIGER, which a strongly diffracting crystal passes on its best frames. On the strong rotation set just added to the battery it cost 767 of 1800 images: peakCountUnfiltered 0 and resolutionEstimate NaN across two blocks of the sweep, the two where the crystal diffracts hardest. Make the bar one pixel in 64 everywhere, and never below the value that stood here, so no smaller detector loses ground. It lived in three places - the host extractor, StrongPixelSet, and SpotExtractorGPU's buffer capacity - now one function. The bar was there for a reason and raising it alone would not have been safe. sparseccl walks a sliding window of the last two lines and tests every pixel in it, which is quadratic in how many strong pixels a line pair holds: a handful for the silicon-tracker hits upstream wrote it for, four thousand for a lit detector line, and 76 seconds for a fully lit frame. But the pixels arrive in raster order, so the window need not be walked at all - a pixel's earlier 8-neighbours are the one to its left and the at most three above it, which is what the GPU extractor already finds by binary search. Keeping the previous line's range and a forward-only cursor gives the same edge set and the same unions in the same order, so the labels are identical, and the fully lit frame now takes 0.16 s. Verified bit-identical on real frames, on fully dense frames, across occupancy 1e-5 to 5e-2, and on 4000 randomised images including ones with blank lines; SpotExtractorGPU's host-vs-device parity test passes untouched. ImagePreprocessorBufferGPU's gather staging was sized to the old constant, with a comment tying it to the caller's give-up. Raising that give-up without it would have run the gather off the end of the device buffer, so it follows the same limit now. Byte-identical .hkl on three battery crystals that never reach the bar. On the strong set, with symmetry, cell and geometry pinned so only the spot list moves: <I/sigma> better in every resolution shell, CC1/2 97.8 -> 98.5%, R_meas 30.5 -> 28.4%, ISa 3.36 -> 3.58, indexing rate 0.772 -> 0.824. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:38:10 +02:00
leonarski_fandClaude Opus 5	a6fbd1d195	Viewer: Alt and the wheel step through the images ... The wheel already zooms, and with Ctrl or Shift it moves the foreground; Alt now moves through the dataset, one image per notch, wheel up forward - the direction QAbstractSlider's own wheel handling uses, which is what the toolbar's scrub slider follows. The view does not know which image it is showing, so it emits the step and the navigation toolbar applies it: the same loadImage() every other control ends in, with the same clamp and the same Sum setting, so nothing about loading is duplicated. An empty dataset is left alone, because loadImage(-1) does not mean "before the first image" but "the latest one" when the viewer is following a running collection over HTTP. The signal is on JFJochImage, so the other views emit it too; only the diffraction view is connected, which leaves them as they were. Note for a desktop where Alt+wheel does nothing: many window managers grab Alt-modified mouse events before the application sees them, and that is a window-manager setting, not something the viewer can take back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:18:19 +02:00
leonarski_fandClaude Opus 5	8999287ba4	Test: a failed calibration now leaves the machine in Error, not Inactive ... JFJochStateMachine_CalibrationFailure was written in rc.164 against the behaviour of the time. rc.165 moved every calibration FAILURE path - CalibrateDetector's catch, TakeDarkMaskInternal's mask-size check and the three ImportPedestal checks - from Inactive to Error, so that /wait_till_done and /wait_until_running answer with the reason instead of the bodiless 502 that a deliberate Deactivate() also leaves; only a CANCELLED calibration still ends in Inactive. The test kept asserting the old state and started failing. Assert Error. The rest of the case is unaffected: the message and severity are what the new SetState writes, and Start() still throws from Error because it requires Idle, so "a data collection is refused on it" keeps its meaning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:01:49 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75 1.0.0-rc.165	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.164	2026-08-26 22:47:00 +02:00
leonarski_f	61c603b274	v1.0.0-rc.163 (#73) ... * Packaging: each package installs its license notices under a directory of its own - `share/doc/jfjoch_broker`, `share/doc/jfjoch_writer`, `share/doc/jfjoch_viewer`, `share/doc/jfjoch_driver_dkms` - instead of all of them into the shared `share/doc/jfjoch`, so `jfjoch` and `jfjoch-writer` can be upgraded one at a time instead of only together. Reviewed-on: #73 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.163	2026-08-25 13:38:29 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.162	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.161	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.160	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.159	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.158	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.157	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.156	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65 1.0.0-rc.155	2026-06-25 22:01:48 +02:00
leonarski_f	6136f858af	v1.0.0-rc.154 (#64) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.154	2026-06-25 18:12:00 +02:00