4 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	4951538368	rugnux: measure the spot budget instead of taking it as given ... --max-spots was a fixed 1000, and on a rotation sweep it sets the DENOMINATOR of the per-frame acceptance gate, which admits a frame when at least a fifth of its spots index. Detections are not all reflections: on a strongly diffracting crystal with heavy solvent background there are 2372 a frame, 1416 of them on ice bands, and only 178 index - so a budget that takes essentially the whole list puts the entire frame population on the gate (median indexed fraction 0.271, tenth percentile 0.218) and the run integrates 83% of its images. The same run with a smaller budget gains 211 frames and loses none, and the frames it gains are the ones with the MOST detections. That is how a larger budget integrates fewer images. So measure it: over the first pass's validation frames, with the sweep's lattice known, tally each spot by rank as +1 if it lies on the lattice and take the budget at argmax over N of n_indexed(N) - 0.20 * n_counted(N) which rises exactly while spots at that depth index better than the gate's own floor and falls after. The 0.20 is that floor, not a new constant - it is lifted out of the function-local it already lived in. It means: as deep into the intensity-ordered list as the image is still showing reflections of THIS crystal. The argmax alone would not do. Under the null that spots index at the same rate at every depth the tally is a driftless random walk, whose maximum is positive whatever the data, so a bare argmax shortens every dataset. The budget therefore has to clear the walk's own noise: the quantity it acts on is the fall from the peak to the end of the list, which is that walk read backwards, and the reflection principle gives its null law in closed form - P(fall > z*sqrt(g(1-g)T)) = 2(1-Phi(z)). That already pays for the search over ranks, so nothing further is owed to multiple comparisons. One false cut in a thousand measurements - a twelfth of one over a 39-crystal two-pass corpus - fixes z at 3.29. The level was chosen before the rule was written and was not revisited afterwards. Battery, same build, one changed default: 36/39 space groups in both arms, none lost, and 36 of the 39 crystals BIT-IDENTICAL. Two move materially - the strong crystal by +17.4% observations and +35.9 points of CC1/2 at 1.58 A, its indexing rate 83.4 -> 95.1%, and another by +93.5% observations with completeness 76.6 -> 98.4%. The significance requirement is what makes that list clean: it removed the one crystal the unguarded rule regressed, and with it two of the five gains, whose peaks do not clear 3.29 sigma on 60 frames. The lever for those is the sample and not the threshold - power grows as the root of the frame count while the bar stays where it is. Rotation only, and the first-pass lattice search always sees the full list: a fixed --max-spots 250 would have been much cheaper to write and fails a battery crystal outright, starving the de-novo FFT into a wrong cell that indexes 7 of 60 frames. Stills never reach the code path - verified bit-identical merged reflections on a serial set - and --max-spots N still pins it, as does the library default the broker and the FPGA use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 19:02:02 +02:00
leonarski_fandClaude Opus 5	1388b16d7a	Resolution estimate: predict past the edge of the detector ... The spot-finding resolution estimate was clamped so it could never beat the detector corner. On a crystal that diffracts past the corner that reports where the DETECTOR stops, which is the one thing this number is not for - it is meant to say how far a merge of data like these would reach, a property of the crystal and the exposure. The clamp also hid the interesting case: an estimate finer than what the run actually merged is the statement "this run was detector-limited", and there was no way to make it. The statistic already extrapolates. Its quantile sits in the middle of the fall-off, well inside what the detector records, so it goes on measuring the crystal's own decay when the detector cuts that decay short. Measured by truncating the spot lists of 31 battery crystals at an artificial detector edge and scoring the unclamped answer against each crystal's own measured CC1/2 = 0.30 crossing, it holds its 8-9% floor out to about 1.7x past the cut and only then drifts pessimistic, which is the safe direction. Every genuinely detector-limited crystal in the battery needs between 1.10x and 1.63x. Against a truth corrected for censoring - the six crystals whose merge is cut off by their own detector cannot have a measured crossing, so theirs is extrapolated from multiplicity-corrected <I/sigma> and anchored on the 25 where both exist: symmetric-log RMS 13.5 -> 9.4% over 37 crystals, 25 -> 28 within 0.2 A. On the six detector-limited ones 26.1 -> 9.8% and the bias goes +19 -> -3%; on the 31 that are not, 9.23 -> 9.32%, i.e. it costs them nothing. The 0.30 tail fraction and the 2.25 reach were refit by leave-one-out against that truth and did not move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:41:34 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	061152279c	v1.0.0-rc.91	2025-10-20 20:43:44 +02:00