6 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	cff5fa0fdb	Merge branch 'shadow-6jgh-dimfield' into r4-integration ... # Conflicts: # image_analysis/beam_stop/ShadowFinder.cpp	2026-09-27 10:10:07 +02:00
leonarski_fandClaude Opus 5.5	7b55a31a3a	ShadowFinder: judge the arm search's dim pixels against the ring's azimuthal harmonic ... The transmitting-piece step added in a1816e905 called a pixel dim when it sat below 0.75 of its ring's median. When the polarization factor the background is divided by is not the beam's, every outer ring is left with a cos 2phi modulation: on one open-arm marCCD sweep (0.75 A, 90 mm, default p = 0.99) the corrected background runs 1.5-1.8x the ring median along x and 0.6-0.75x along y at r = 1600-2200 px. Joined, the y lobes made two pieces of 1.01 M and 0.39 M dim pixels holding only 0.17 % deep pixels (noise and a real ~20 % partial shadow) - past the absolute gate of 200 - and the mask grew from 16,584 to 1,428,328 pixels, covering the top and bottom of the detector. The step now fits, per 64 px radial band, the 24 sector medians with m + p cos 2phi + q sin 2phi (sectors below 0.75 of the fit left out, three times over) and judges dim against min(1, model). Capped at 1 the model only removes dim pixels, so a sweep with no piece keeps its mask bit for bit and a piece can only shrink or go. Shadow pixels (b90b377b8 -> this), same binary otherwise: that sweep 1,428,328 -> 29,821 (the stop's stick extension kept); arm sets 7atg 724,032 -> 723,708, 6oel 223,087 -> 216,126, 7rji 311,951 -> 309,816, 9fhc 173,597 -> 171,372, 5ebi 793,217 -> 785,477, 6zqr 289,324 -> 251,764, 5j23 530,541 -> 526,115, metformin 341,368 -> 231,123 (two speckled pieces in the high-angle lobe dropped, the arm below the beam kept). Merges: 6oel, 7rji, 6zqr, metformin unchanged in space group and resolution, R_meas/ISa within 0.1 %/0.2; 7atg fails either way on this base (the beam-centre defect fixed on atg-arm-centre). The sweep above: same space group, resolution, CC1/2 and ISa (6.72 -> 6.73); R_meas 21.7 -> 22.7 %, because the spurious mask also hid part of a real ~20 % partial shadow at the top right that neither version masks (it is above penumbra_ratio). New test ShadowFinder_AWrongPolarizationFactorIsNotAnArm (unpolarized background, 0.99 stated, small absorbing speck in the dim lobe) masks 4547 px before and 0 after; all *Shadow* tests pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-27 09:59:17 +02:00
leonarski_fandClaude Opus 5.5	a42b73ccfb	Rugnux: fit the beam centre without the transmitting beam-stop pieces ... a1816e905 adds the partly transmitting holder-arm pieces to the beam-stop mask, and the background beam-centre fit read that mask too. A thin arm running into the beam, masked, takes cells out of the rings it crosses; which radial bins stay above the sector-coverage gate then changes as the trial centre moves, and a walk seeded beside the arm (the FFT capture sat 48 px off) limit-cycled on that edge and stopped 44 px short (sigma 0.34 -> 2.27 px). On an open-arm set whose header centre is 171 px out, the measured centre no longer indexed (60/60 -> 0/60), the run kept the file's centre and merged P1 at CC1/2 0.004 instead of P212121 at 0.995. The arm's dimming is multiplicative in azimuth, which the fit's per-sector amplitude already absorbs, so the fit now uses the mask without those pieces (ShadowFinder marks them TRANSMITTING); integration still masks them. The fitted centre is back to its a0518abe6 value on every set checked (the failing one plus four arm sets); the failing set merges P212121, CC1/2 0.995, ISa 22.6. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-27 08:34:36 +02:00
leonarski_fandClaude Opus 5.5	a1816e905e	ShadowFinder: add beam-stop holder arms that let part of the beam through ... After the existing mask is complete, a second step takes the pixels it left out that are significantly dimmer than their ring, joins them through a 6 px bridge and across module gaps of any width, and adds a piece whole when it holds >= 2000 dim pixels of which >= 200 are deep. The existing mask is never touched, so a sweep with no such piece keeps its mask bit for bit. Quick subset tests (battery --only, no model check) against the rc173 a0518abe6 full battery: 152 of 233 masks identical; 10 of 10 controls (including the sets where earlier shadow changes regressed through marginal decisions) give byte-identical merges. On the 21 sets that gain a piece the space group never changes, merge outlier rejections fall on 19, R_meas falls and ISa rises on 17, and the shell-scaled model R improves on 16 of 18; a transmitting arm with an over-subtracted background strip is recovered (R_meas 13.3 -> 12.0 %, ISa 13.4 -> 15.5). Known costs: on one set with background bumps at ring radii the low-resolution agreement with the model falls (CC 0.89 -> 0.85) while its internal statistics improve, and two sets cut slightly coarser (1.50 -> 1.55 A, 1.72 -> 1.80 A) with a better model R. Squashed from branch hq-beamstop (cfa080a59..60b6c6822), whose history records the variants tried and dropped: a straight port of the earlier arm finder moved every mask, a two-shape construction filled almost a whole sweep through hole filling, and a deep-fraction floor rejected a real arm that is dim along its whole length. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 14:01:17 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00