6 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	b650cda037	Revert "integration: the background is fitted over the ring that survives, not averaged over it" ... This reverts commit 6a3ae9c65. The rotation battery refuses it. The median is clean - zero movement in R_meas and CC1/2, most crystals untouched or moving in the third decimal, which is what was predicted - but the tail is not a cost, it is a failure: one crystal's CC1/2 falls from 91.3% to 6.8% with its reflection and observation counts essentially unchanged, so the intensities are being corrupted rather than the weak ones discarded. Another loses seven points of R_meas and half its ISa; a third has its high-shell CC1/2 go negative; a fourth's ISa doubles, which is an error model measuring something different rather than measuring better. The change was predicted to cost one to two points of high-shell CC1/2 on crowded patterns, and that prediction was made on six geometries. Fifty-one say otherwise. The defect it fixes is real and stays on the record: within the outer background radius of the array edge the ring is truncated inward while the disk is not, so the surviving ring sits where the radial background is higher and the reflection reads low - measured at 1.79 counts per pixel four pixels from a border, and confirmed by a masked gap producing the same bias signed by the direction of its displacement. Fitting the background over the surviving ring is the right shape of answer. Doing it per reflection, from a ring that a neighbour mask has often already truncated, is evidently not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 16:30:22 +02:00
leonarski_fandClaude Opus 5	6a3ae9c658	integration: the background is fitted over the ring that survives, not averaged over it ... The signal disk and the background ring are concentric, which is the whole reason a linear background cancels between them. Within the outer ring radius of the edge of the sensor array that concentricity is gone: the ring loses its outer part while the disk barely loses anything, so what is left of the ring sits further into the detector, where the radial background is higher, and the reflection reads low. Measured at signal-free positions four pixels from a border: the ring reads 162.35 counts per pixel against a true background over the disk of 160.57, which over a hundred disk pixels is 182 counts of deficit, against 216 to 239 observed. <I/sigma> runs -1.83, -2.34 and -1.18 at nought to three, three to six and six to nine pixels from the border, and recovers exactly at the outer ring radius. The same reflection measured at a border reads 179 counts lower than in the interior over seven thousand matched pairs. A masked module gap does the same thing but signed by the direction of the displacement, which is why nothing has caught this: at a gap the two populations cancel in the mean, while at the sensor border the truncation is always inward, so the bias is always negative. The background is now the intercept of a straight line in radial offset over whatever ring pixels survive, read at the reflection's centre. Three extra sums per ring pixel and no extra reads; the radial distance was already computed there. It is exact under any truncation and reduces to the mean when the ring is whole, so it is unconditional rather than a mode: a badly truncated ring pays in sigma, through the variance the fit honestly reports, rather than in a rejection. On the crystal where this surfaced the outermost shell's correlation with a deposited model goes from -0.234 to +0.004, and the shell above it from -0.091 to +0.179. Correcting beats discarding: dropping every observation within fifteen pixels of a border reached only -0.019 and +0.127, because the corrected observations still carry signal. Interior reflections do not move. The cost is one geometry: where a neighbour mask has already truncated the ring almost everywhere, the fit roughly doubles the variance of the background estimate while finding no gradient worth removing, and a crowded small detector loses one to two points of CC1/2 in its finest shells. Also: the MINPK denominator counted only profile mass that lands on the detector, so a reflection whose peak is off the sensor scored a perfect one and no guard could fire. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 15:46:32 +02:00
leonarski_fandClaude Opus 5	db9cc9106f	rugnux: the per-reflection correction factor is named for what it is, not for what it once held ... The factor multiplied into each integrated intensity was called rlp, for reciprocal Lorentz-polarization, and until this week that is all it held. It now also carries the sensor efficiency at the angle the beam arrives, and on the stills path it holds that efficiency and the polarization with no Lorentz term at all - correctly, since the Lorentz factor of a still is one. Three different products under one name that promises exactly one of them, in code where the neighbouring member is the total correction. Rename it prescaling_corr: multiplicative, applied before scaling, therefore not a scale, and silent about its contents - which is the point, since the contents have now grown twice. It is also what DIALS calls the same product. The stills refinement member spelled "1 / rlp" becomes inv_corr, and the comments and usage text that promised "the Lorentz-polarization factor and nothing else" now say what is actually there. The Lorentz term keeps its own name where it is computed, because that name is correct. The two external spellings are untouched: the CBOR key and the reflection dataset are a published format, and a reader that meets an unknown key would take the factor as zero, which both the merge key and the ingest treat as a reflection to drop - so every reflection would vanish and the run would still exit zero. No output changes: the merged and unmerged files of two full runs are byte for byte what the previous binary wrote, four stored files from before the efficiency correction still re-scale identically, and the reflection datasets of the process file are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 13:41:16 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00