094eee31a2d8e40609fa41b53b67a6fc01959e9f
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  leonarski_fandClaude Opus 5.5
|
f6c948732b |
GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block
The stills geometry refinement (module placement fits, the held-out fits that decide them, the tilt decision) spent most of its time differentiating parameters it never moves and in Ceres' general Schur elimination: - Each spot's residual declared the rotation axis, the cell angles and the (usually held) tilt as blocks. Ceres sizes its dual numbers from the declared blocks, so a placement residual carried 20 derivatives to use 12. They now enter as plain numbers (SpotResidual / TiltedSpotResidual). - A fit that moves only the orientations (the held-out scoring) gets a residual over the orientation alone, with the observed and predicted vectors taken once per spot (OrientationResidual): 3 derivatives instead of 17. - The beam and the distance are one block of three, so every refined block has three parameters and Ceres uses its fixed-size Schur eliminator (1.7x faster linear solve on one thread). A held distance is a SubsetManifold on that block. - One robust loss shared by every spot instead of one allocated per spot. Same arithmetic on the same values: replaying the dumped placement inputs of four multi-panel stills runs (AGIPD 128 ASICs, MPCCD 8, JF4M 8, JF16M 32 modules) on one thread gives identical held-out residuals, beam, distance, cell and every module shift at the printed precision. Single- thread CPU of the nine placement fits: 397 -> 112 s (MPCCD), 179 -> 80 s (JF4M), 131 -> 68 s (JF16M). On AGIPD the concurrent step goes 8.5 -> 6.4 s wall (quiet replay, 32 threads). Test: RefineGlobalGeometry_ModulePlacement now also covers a held distance and an orientation-only fit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Opus 5.5
|
a39e18b80f |
stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree
The stills geometry bundle (RefineGlobalGeometry) can now refine the detector tilt together with the beam, the distance, the cell and the modules. With the tilt free the beam block is the direct beam, from which the PONI is worked out inside the residual: in PONI terms a tilt moves the whole pattern as a moved beam does (D tan(tilt)/pixel), more than the beam's 8 px box holds, while the direct beam does not move with it and the tilt is left with the keystone. The two tilt-like rows of ModuleGauge are gone: within a module a tilt is a gradient no rigid move makes, and the rows cost held-out accuracy. The shift, turn and outward gauges stay. The tilt is a hypothesis (DecideStillsTilt): the strong frames of the last geometry refinement are split in two halves, each fitted with the tilt held and refined (modules placed in both where the run placed them), each fit scored on the other half, and the tilt is kept where the refined fits predict the held-out spots better. No threshold. It is decided after the stills beam-centre trial and its re-refinement, on frames indexed at the settled centre: a centre error on frames indexed at it reads as a tilt. --rot1/--rot2 or --no-refine-tilt hold the tilt on stills. The stills beam-centre trial now compares direct beams. Its hypotheses (the file's centre, the background walk, the point-symmetry capture) are taken as the point the beam lands on and each trial puts the PONI where its distance and the run's tilt land the direct beam there; the walk, fitted through the geometry's tilt, returns a PONI and is converted. The pre-scan beam-centre check log compares the walk's direct beam, not its PONI, with the file's direct beam. Without a tilt nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Opus 5.5
|
96322a8ec9 |
stills fixed cost: geometry refinement and pre-scan do the same work in less wall time
Four changes to the per-run fixed cost of a stills run (Step 2 geometry refinement, Step 3 pre-scan), none of which changes what is computed: - Module placement: the four held-out fits and the placement on every strong frame are independent, and one Ceres fit keeps only ~4.6-6.5 of 32 cores busy (perf, JF16M, 4000 frames / 150888 spots). They now run side by side on a fifth of the threads each; the placement fit is computed speculatively and used only where the held-out residuals call for it, as before. Baseline (quiet): held-out 12.3 s + placement 5.1 s on JF16M, 16.4 + 6.5 s on JF4M, 19.0 + 7.6 s on MPCCD. - Module scan (+-25 px grid per module ahead of the placement) was one thread: every grid shift is now scored on the workers and the grid then walked in its old order, so the first best still wins (3.3 s on JF16M, ~5 s on JF4M/MPCCD before). New test ScanModuleShifts_FindsShiftedModule checks the shift is found and that 1 and 4 threads give the same answer. - Stills geometry trial: each hypothesis (centre, distance, frame sample) is indexed once. The distance grid ran twice on the same centre whenever the file's centre was kept (40 of ~275 hypotheses, ~15% of the trial: 6.2 s on JF16M, 6.7 s on the flat JF8M), and the final here/there check repeated the first trial. Repeats always gave the same count (checked on three logs). - The saturation histogram (40 frames, 0.1-2.2 s) reads only frames and the mask: it is taken beside the geometry refinement and the limit decided where it was, after it. Plus TIMING marks for the sub-steps (geometry refinement sample, module scan, module placement fits). Outputs identical to the baseline binary (bb7c9c) where checked: JF16M int16 15 runs (geometry, all 32 module shifts, held-out residuals, cell), JF4M int16, MPCCD 236 (indexing rate, CC1/2, R_split, R-free with --model identical), PAL 90 and LCLS 222 (identical reports). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Opus 5.5
|
ebce906025 |
Serial XFEL stills inputs: readers from the files themselves, module geometry and its refinement
Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto rc178 with the owner's decisions for this release: - Inputs are files, never a directory; several files of one format are one dataset (the multi-input CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles, ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and xtc::CollectStreams are gone, CanRead* accept a regular file only. - One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds: per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work). - Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib, bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through HDF5's filter under the global lock, one frame at a time. - LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all. - CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path) and optionally the camera length; everything else is ignored with a note in the log. - --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored layout (what a .geom's mask_file used to give). - Gain measured on the background for panel-table and CXI input that states none (--gain overrides). - Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions are mapped through the modules, the image is not re-assembled. - Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table, .geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx NXdetector_module groups), read back with --module-geometry. - A panel-table file's camera length is nominal: with a reference cell it is refined from the cell. - Broker and writer output unchanged: masters keep the single NXdetector_module group, the start message carries no module list. Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation], SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |