5 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	a7ed4e2d34	beam stop: a bright ring is not a beam stop ... Inside the stop a whole ring is blocked, so its own median is blocked too and the per-pixel comparison has nothing to work with. The walk that covers that case declared a ring to be inside the stop when its background fell below a third of the LARGEST background of any ring further out - and on a sample whose background peaks in a strong ring away from the beam, the ordinary background inside that ring is legitimately below a third of the peak. The walk then runs out to the ring and returns a filled disk of good detector: on one corpus dataset 16 % of the area, with diffraction rings plainly visible inside the disk it masked, and on another 2.5 %. The defect predates the branch; what changed is that the flood from the beam centre used to discard the disk whenever its seeds landed on invalid pixels, which is how the first of those two datasets came back with an empty mask instead of a wrong one. Dropping that anchor was right, and it made this visible. A ring is now compared against what this detector's background typically is - the median over the rings the walk is willing to judge - which is robust to a bright ring and to a corner ring of a handful of pixels alike, and is less code. Evaluated over 151 corpus datasets: the number returning a disk larger than 0.2 % of the detector falls from 12 to 3, the two pathological cases collapse (radius 974 -> 68 px and 188 -> 12 px), and genuine stops move by a few pixels at most (152 -> 132, 122 -> 120, 70 -> 64). The new test fails on the old walk and passes on this one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-13 07:36:57 +02:00
leonarski_fandClaude Opus 5	99910e187e	beam stop: a ring is only flat once the polarization is divided out ... The shadow test compares a pixel with the median of its ring, which assumes the background is flat around a ring with nothing in the beam. It is not: a polarized source suppresses the background in its own plane by 1 - sin^2(2 theta), a factor of three at 2 theta = 55 degrees and four at 70. That is several times the dip the test is looking for, so on a short-distance geometry the two horizontal lobes of every outer ring read as shadow. Measured over the corpus it cost one dataset 5.1 % of its detector, and another 11.2 %, with nothing visible under either mask. The mean projection is therefore divided by the Kahn factor before the ring comparison, and the Poisson deficit multiplies it back in so the significance is still the significance of the counts that were recorded. The factor is the geometry's own CalcAzIntPolarizationCorr, evaluated about the centre the caller named rather than the one in the file: the azimuth is the whole point of this correction, and the geometry is what knows where the polarization plane lies once the detector is tilted, the stored image quarter-turned or the detector rotated in its own plane. Six corpus datasets are quarter-turned, and a flat-detector formula on the stored image's own axes gets them 90 degrees out of phase - measured, it takes one of them from 16.5 % masked to 22.8 % where the geometry takes it to 4.4 %. Polarization is also the only correction a ring carries that varies along it; solid angle, detector and air absorption are functions of 2 theta and the ring median absorbs them. Measured, mask fraction of the detector: a clean 110 mm sweep 0.82 -> 0.51 %, a clean 200 mm 18 Mpx sweep 0.94 -> 0.80 %, a 150 mm sweep whose header centre is 351 px out 5.05 -> 0.74 %, and the sweep with a third of the detector behind a pin 32.22 -> 32.83 %. Clean detectors lose mask, a real obstruction gains it. Across 151 corpus datasets nothing gains more than a tenth of a percent from the correction and 26 lose some, 6 of them by more than 3 percentage points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-13 07:36:57 +02:00
leonarski_fandClaude Opus 5	dd95704d7b	beam stop: the rings are drawn about the centre the data measures, not the one the file claims ... Every pixel is compared against the ring it sits on, and the rings were drawn about the beam centre in the file. Displacing that centre costs nothing for tens of pixels and a great deal beyond: on clean sweeps 50 px leaves the mask where it was, 150 px returns a tenth of the detector as shadow with nothing blocking it. Header centres are wrong by that much - one sweep in the corpus states one 351 px from where the data put it, on a run that otherwise indexes every frame and merges at CC1/2 0.999 - so a shadow search that trusts the header is a live hazard now that an empty mask is no longer the fail-safe it used to be. The centre is therefore fitted from the projection BEFORE the mask is read, and named to the finder with BeamCenter(). It costs no frame of its own: the fit reads the same per-pixel mean the mask is computed from. Measured on the phase boundary, the pre-scan grows by 0.4-0.6 s on a 2.5 Mpx detector and by 3-4 s on an 18 Mpx one. The other half of the circle is that this fit is itself biased by a shadow it has not yet masked - on a sweep with a third of the detector behind a pin it put the centre 5 px from the truth and called it 0.40 px, ten of its own sigmas out. It does not have to be unbiased here. The ring comparison does not notice tens of pixels and the bias is a few, and the centre the run REPORTS and consumes is not this one: it is the fit that already ran after the mask was loaded, unchanged, with the shadow out of the way. The order is fit, mask, fit, and only the second answer leaves the function. Two rounds are enough, measured rather than assumed: on four clean sweeps the fit before the mask and the fit after it agree to 0.03 px, so a third round would draw the same rings. The centre estimator is not made robust to a shadow either - it already robustifies across azimuthal sectors, three IRLS rounds on the per-sector shifts, and that is what returned 5 px at 0.40 px, because a third of the azimuth is a second population and not an outlier. And no guard refuses to mask when the fitted centre disagrees with the header: a header 351 px out is precisely the case this has to survive. Mask, before -> after (pixels, and per cent of the detector): clean, 2.5 Mpx 20851 (0.82%) -> 20670 (0.82%) clean, 2.5 Mpx 33308 (1.32%) -> 32730 (1.29%) clean, 18 Mpx, low bkg 173804 (0.96%) -> 170031 (0.94%) clean, 18 Mpx 250196 (1.38%) -> 249915 (1.38%) a third behind a pin 818184 (32.32%) -> 815699 (32.22%) header 351 px out 54183 (0.87%) -> 314592 (5.05%) The clean sweeps do not move; on all four the measured centre is 1-9 px from the file's and the mask shrinks by under 3 %. The pinned sweep does not move either - its header centre happens to be right, so the rings were already where they belong - and its merge is unchanged to the third decimal. The last row goes the other way and is not the ordering: with the rings finally about the beam, this geometry reaches 2 theta = 68 deg, where the polarization of the source modulates the background around a ring by a factor of three. That is four times the threshold this test cuts at, so the horizontal lobes read as shadow. The commit that follows removes that confound; with it the same row reads 39 k px (0.63 %), which is the holder arm and nothing else. Even uncorrected the run's answer is unchanged - same space group, same cell to the third decimal, 12658 unique reflections either way, R_meas 9.06 -> 9.10 %, and 1 % of the multiplicity lost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-13 07:36:57 +02:00
leonarski_fandClaude Opus 5	d412306c77	beam stop: a shadow does not have to touch the direct beam ... The shadow finder kept only the low region connected to the beam centre, flooding outward from seeds within 4 px of it. That is written for the beam stop and its holder arm, an object that touches the direct beam. Hardware that stands in the beam further out - a pin, a loop - casts a shadow that begins some way out in radius, with lit detector between it and the stop, and no bridge crosses that gap. On a sweep where such a shadow covers 38 % of the detector the per-ring test found the whole of it and connectivity then discarded 950 k pixels, leaving the beam-stop disk alone; those pixels went into integration as measured-and-near-zero. What makes the test specific instead is size: keep the connected components that hold at least MIN_SHADOW_PIXELS core pixels. A shadow is cast by something physical and is correspondingly large, while the background wanders a pixel or two at a time. Measured on five clean in-house sweeps spanning 0.05 to 9.5 counts/px/frame of background, every one returns exactly one such component - the beam stop - and the largest spurious candidate anywhere is 74 pixels, a 27x margin. MIN_EXPECTED_COUNTS becomes a Poisson significance. A ratio says nothing when the background behind it is a handful of photons, so the deficit is now required to be significant against its own scatter, sqrt(2 (E - N + N ln(N/E))) over the pooled counts. This is what makes the comparison scale-free rather than tuned to one exposure: rebuilt from six frames of a low-background sweep, the same thresholds without it mask three quarters of the detector and with it mask none of it. The index of dispersion, measured over the frames, does NOT do this job - it is 1.0 inside the shadow and 1.0 outside it, because a shadowed pixel is Poisson at a low rate and a lit one is Poisson at a high rate. Only the rate relative to the ring separates them. The rest follows: the core ratio moves 0.35 -> 0.50 and the half-shadow reach 0.72 within 14 px -> 0.75 within 30 px, since a pin's penumbra is much wider than a beam stop edge's; the ring walk that finds the rings lying wholly inside the stop keeps its own threshold so the central disk does not move; and those rings join the region after the size filter rather than being asked to be large themselves. On the shadowed sweep this takes the mask from 16,587 px (0.66 %) to 818,184 px (32.32 %), against 956,705 px (37.79 %) for the reference implementation, at a precision of 0.92 against it. Clean sweeps move from 0.60 % to 0.82 %, the width of the same arm. One thing this costs: an empty mask used to be a statement about the beam centre, because the seeds were placed at it. It no longer is, and a centre wrong by a couple of hundred pixels now draws the rings across the background's own fall-off and returns a large spurious mask instead of an empty one. The log message says so. The fix is ordering - the centre is measured from this same projection a moment later - and is not attempted here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-13 07:36:57 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00