417 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	96322a8ec9	stills fixed cost: geometry refinement and pre-scan do the same work in less wall time ... Four changes to the per-run fixed cost of a stills run (Step 2 geometry refinement, Step 3 pre-scan), none of which changes what is computed: - Module placement: the four held-out fits and the placement on every strong frame are independent, and one Ceres fit keeps only ~4.6-6.5 of 32 cores busy (perf, JF16M, 4000 frames / 150888 spots). They now run side by side on a fifth of the threads each; the placement fit is computed speculatively and used only where the held-out residuals call for it, as before. Baseline (quiet): held-out 12.3 s + placement 5.1 s on JF16M, 16.4 + 6.5 s on JF4M, 19.0 + 7.6 s on MPCCD. - Module scan (+-25 px grid per module ahead of the placement) was one thread: every grid shift is now scored on the workers and the grid then walked in its old order, so the first best still wins (3.3 s on JF16M, ~5 s on JF4M/MPCCD before). New test ScanModuleShifts_FindsShiftedModule checks the shift is found and that 1 and 4 threads give the same answer. - Stills geometry trial: each hypothesis (centre, distance, frame sample) is indexed once. The distance grid ran twice on the same centre whenever the file's centre was kept (40 of ~275 hypotheses, ~15% of the trial: 6.2 s on JF16M, 6.7 s on the flat JF8M), and the final here/there check repeated the first trial. Repeats always gave the same count (checked on three logs). - The saturation histogram (40 frames, 0.1-2.2 s) reads only frames and the mask: it is taken beside the geometry refinement and the limit decided where it was, after it. Plus TIMING marks for the sub-steps (geometry refinement sample, module scan, module placement fits). Outputs identical to the baseline binary (bb7c9c) where checked: JF16M int16 15 runs (geometry, all 32 module shifts, held-out residuals, cell), JF4M int16, MPCCD 236 (indexing rate, CC1/2, R_split, R-free with --model identical), PAL 90 and LCLS 222 (identical reports). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 17:48:09 +02:00
leonarski_f	91861e9b08	Merge branch 'geom-mask' into panel-s5	2026-10-21 13:18:11 +02:00
leonarski_fandClaude Opus 5.5	c942dcb584	Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane ... - The NXmx master (and _process.h5) writes module_0..N with the panel table when the start message has one, besides the whole-image module, so a stored-layout file reads back with its panels (JFJochReader_PanelId checks the round trip). - RotationScaleMerge's modulation grid spans PlaneBounds instead of the image rectangle; with no panel table x0 = 0 and the extent is the image, so the cells are unchanged. - The beam-stop / pre-scan memo key includes each panel's box and placement; nothing is added with no table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 12:44:40 +02:00
leonarski_fandClaude Opus 5.5	451fd25857	Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table ... A detector of two or more panels is no longer assembled. The image is the stored frame, and each panel of the geometry (CrystFEL .geom or the file's own panel table) is a DetectorModule over its box of the frame, at its true placement on the detector plane: fractional origin, the panel's own axes, no snapping to an image axis or a whole pixel. Pixels on no panel, edge, bad-region and file-masked pixels are masked in the stored layout. The block scatter (a transpose for the 90-degree-turned JUNGFRAU and MPCCD panels) is gone for these inputs. A single panel keeps the old block path, so single-panel CXI files are unchanged. Signed-integer frames in their own units (gain 1, no per-frame mask) go up with no copy: a bitshuffle/LZ4 chunk still compressed, for the GPU preprocessor's BSLZ4 decoder (int16's smallest value is its error value there, so the type-min rule holds), a deflated one inflated outside the HDF5 lock straight into the caller's buffer. Everything else (float, unsigned, a gain, a per-frame mask) is converted once in stored order. A float NaN/inf now reads as no measurement (INT32_MIN) instead of 0, and values past int32 are clamped before the cast. Also: SpotToSave.panel is filled from DiffractionSpot::Panel(), so AssignModules takes the panel the spot was found on. PreviewImage_PanelLayout expects stage 3's PlaneRasterOf raster (merge fallout). Main pass, 2000 frames, vs the CCD-reader build: JF4M int16 bslz4 465 -> 1860 frames/s, JF16M 105 -> 1111, MPCCD 212 -> 252; flat Jungfraujoch data unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 12:44:40 +02:00
leonarski_fandClaude Opus 5.5	7c23b4befe	CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them ... The .geom parser now takes the masking information the format defines: - mask arrays: mask/mask_file/mask_good/mask_bad (mask 0) and maskN_data/maskN_file/maskN_goodbits/ maskN_badbits for N = 0..7; global keys hold for the panels after them, a panel may override. A mask that is the same on every panel is one whole-frame mask, otherwise one per panel, applied on that panel's pixels only. - a relative mask_file is looked for in the working directory (as CrystFEL does), then beside the .geom. - bad regions (bad*): a lab-plane box (min_x..max_y) or a fs/ss range on a panel (or, old syntax, on the whole frame); mask_edge_pixels and no_index. JFJochXFELReader already had the static pixel mask and the per-frame mask path for these; the masks go through them unchanged. A mask whose path holds % (one per shot group) is applied frame by frame like a 3D one. Where the geometry names mask arrays, the data file's own Cheetah masks (CXI mask_shared and per-shot mask with Cheetah's generic bad bits, /metadata/pixelmask) are not added: the geometry says which bits matter (e.g. an AGIPD mask_bad that includes bit 32). --mask-good / --mask-bad still add their masks to these. Tests: [CrystFELGeometry] (new keys, per-panel override, relative mask file), JFJochXFELReader_* (new GeometryMasks case on a synthetic CXI + .geom), [JFJochMultiReader], JFJochXTCReader*. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 12:42:08 +02:00
leonarski_f	64f6b7a5c4	Merge branch 'rc178-still-results' into panel-s5 ... # Conflicts: # reader/JFJochXFELReader.cpp # tests/XFELReaderTest.cpp	2026-10-21 11:28:17 +02:00
leonarski_f	6a7706b955	Merge branch 'ccd-reader-speed' into panel-s5	2026-10-21 11:27:56 +02:00
leonarski_f	481ddeae24	Merge branch 'panel-s4' into panel-s5 ... # Conflicts: # common/AssembleToPlane.h	2026-10-21 11:27:51 +02:00
leonarski_f	648ae8f0d5	Merge branch 'panel-s3' into panel-s5 ... # Conflicts: # common/CMakeLists.txt # tests/PanelLayoutTest.cpp	2026-10-21 11:27:31 +02:00
leonarski_f	e065011826	Merge branch 'panel-s2' into panel-s5	2026-10-21 11:27:12 +02:00
leonarski_fandClaude Opus 5.5	95e51f868d	XFEL reader: integer frames stay integer, one gain, placement by panel ... DecodeInto no longer turns every stored pixel into a float and back: - the gain is one value (adu_per_photon_), not a float per stored pixel - it always was one value; only --gain or a measured gain set it; - a 16-bit signed frame (Cheetah CXI, SACLA) without a gain is handed up as Int16, in its own type; other integer frames are widened to int32 without a float; only a float frame or a gain other than one goes through lround(v / gain), exactly as before; - placement is per panel (start, size, destination steps) instead of a 64-bit destination per stored pixel; a panel square with the image is copied row by row (PAL: 1440x1440 into 1444x1444 - the two-pixel margin is Place()'s, kept so the geometry does not move); - the decode buffers (chunk, inflated, unshuffled) are reused per thread, the bitshuffle decode is not copied again; - a chunk whose HDF5 filter_mask is not zero (a filter skipped) is read by HDF5 instead of being decoded as if it were not; - the never-set value flags (flag_lessthan/morethan/equal) are gone from the layout and from the per-pixel loop. MeasureGain in the CLI now reads Int16 frames as well, so a SACLA run still has its gain measured. Tests: JFJochXFELReader_SACLA/_CXI updated for the mode, new JFJochXFELReader_CXI_Int16 (deflate without shuffle, a chunk with its filter skipped) and JFJochXTCReader_StoredPixels (synthetic XTC, stored hand-up and the offset path). Measured on two PAL-XFEL Rayonix sets (int16, deflate; RTX 5080, 5950X, page cache warm; reports and .hkl identical to the baseline): main pass, 2000 frames: 296 -> 799 frames/s (pal86), 302 -> 755 frames/s (pal90) wall, 2000 frames: 26.4 -> 13.6 s, 32.1 -> 19.4 s (the geometry-refinement steps, which read frames too, halve) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 10:50:34 +02:00
leonarski_fandClaude Opus 5.5	d6166f6fa5	Panel layout stage 2: prediction and Bragg integration through the panel table ... Everything sparse that took a radial quantity from an image coordinate now takes it from the reflection's (or spot's) position on the detector plane, through its panel (DetectorModules). With an empty module table every changed expression reduces to the old one in the same floating-point order, so flat detectors - every rotation set, EIGER, CCDs, assembled JUNGFRAU - integrate bit for bit as before. - Prediction (CPU, GPU, rotation CPU and GPU): bounds test against PlaneBounds instead of the image rectangle. - IntegrateOnImage: a prediction in a gap or off every panel is dropped (PlaneToPanel), no longer passed through onto an unrelated stored pixel. The panel id stays local: the engine and the way back find it from the image position (the panel boxes tile the image), and a prediction that rounding to float put on another panel's edge is dropped so the two agree. Nothing is added to Reflection. - Bragg engines (CPU, CUDA, Metal) take a BraggPanel per reflection (BraggStencil.h): its plane position, the panel's pixel axes on the plane and its box. The stencil's radial direction is taken on the plane and expressed in the panel's pixel axes; r0, the radial background curve, the Gaussian's elongation, the learned moments, tan(2theta) and Finalize's polarisation use the plane position; r_max / n_rad come from PlaneBounds. The neighbour mask and the owner map are clamped to the reflection's own panel; inside the walked box a pixel off the reflection's panel is read as a gap (unreadable, statically masked - not an overload), which is what the assembled image has there, so a disk cut by a panel edge is partial rather than silently "full". - Risk R4: the ProfileEmpirical grid is learned and applied in the plane's axes (the panel axes taken to the nearest quarter turn, BraggProfileCell), so turned panels pool one shape. Without it the stacked layout read up to 0.3 % off the assembled one in the harness. - SpotWidth / SpotFootprint: aperture bounds against the spot's panel box, beam distance, resolution and the shape's 2theta Jacobian on the plane, moments read in the panel's axes. - GetDetectorMaxQ_recipA, the far-corner radii (first-pass tilt report, metric-symmetry arm, footprint table range) from PlaneBounds; pre-scan spot resolution on the plane; LogHeaderBeamCenter: geometric centre of the panels' plane box, and the masked-pixel check on the panel the beam lands on (none when it is in a gap). Harness: PanelLayout_BraggIntegration is now a real [PanelLayout] check, over six settings (default, bandwidth-elongated ring, radial background, overlap exclude, box sum, empirical profile, all with polarisation), stacked vs assembled to 1e-5; PanelLayout_BraggIntegrationGPU runs the same on the CUDA engine. Its spots avoid exact half pixels: the grid centres on the rounded position, and a half rounds to the other neighbour on a turned panel. The Metal engine (.metal/.mm) mirrors the CPU change exactly but is UNCOMPILED here (no macOS): it needs a build and BraggIntegrationEngineMetal_BitIdenticalToCPU on a Mac. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 10:40:20 +02:00
leonarski_fandClaude Opus 5.5	fd798cd665	Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster ... Spot finding works panel by panel, as the FPGA path does per JUNGFRAU module. PanelBoxes (common/) gives one box per DetectorModule; an empty table gives one box, the whole image, so every clamp reduces to the old image bounds in the same evaluation order (rotation sets, EIGER, CCDs, our JUNGFRAU unchanged). - ImageSpotFinder::SetPanels: the CPU local-box pass (dense, count-threshold and the DetectAt/candidate path of the adaptive finder) slides its windows inside each panel box; host CCL labels panel by panel; pixels on no panel are folded into the resolution mask. - GPU: analyze_pixel tiles each panel (warps numbered panel after panel), analyze_candidates and union_neighbours clip to the pixel's panel, the device spot carries its panel; the GPU list is put in panel order so it equals the host's. - DiffractionSpot carries its panel; SpotAnalyze maps by ImageToPlane(panel, ...) (d_A on the plane). - ImagePreprocessorBuffer::GatherTiles (CPU+GPU) clips each square to the panel of its centre. - MXAnalysisWithoutFPGA and the pre-scan workers hand the module table to finders and buffers. Raster algorithms: new common/AssembleToPlane.h (PlaneRasterOf, AssembleToPlane nearest-pixel gather with gaps, GatherFromPlane back to stored pixels; empty table = copy). With panels, ShadowFinder::GetMask runs on the projection assembled to the plane raster and gathers the mask back per stored pixel (CPU route); FindBeamCenterFFT scores the assembled projection and reports plane coordinates; the detector diagnostic JPEG shows the assembled plane. Without panels all three take their old path. Harness: PanelLayout_SpotsNeverCrossPanels enabled (count threshold and SNR on the edge scene), new PanelLayout_SpotFinderParity (CPU vs GPU, local box and adaptive, on the stacked layout), PanelLayout_AssembleToPlane, PanelLayout_ShadowFinder, PanelLayout_BeamCenterFFT, PanelLayout_DetectorDiagnostic (stacked vs assembled image cut to the raster). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 10:29:07 +02:00
leonarski_fandClaude Opus 5.5	4d90053899	Viewer and preview show a panel detector as it lies on the detector plane ... With a panel table, the viewer and the broker preview gather the stored image onto the detector plane (AssembleToPlane) and draw on that: spots, predictions, rings, beam centre and ROIs are plane coordinates, so they now land on the right pixels for a stored (stacked or moved-module) layout, where they were drawn as image pixels before. The viewer scene is the plane; its image item sits at the raster's origin (JFJochImage::origin_, zero for a flat detector). Mouse readout, pixel labels and the magnifier read the shown raster; top pixels and saturation crosses are mapped from the stored pixel to the plane; an ROI turned into a user mask is tested per stored pixel at its plane position. A flat detector (empty table) takes the old path verbatim: the shown pixels alias the image, origin (0, 0) - a JF/EIGER file renders identically. common/AssembleToPlane.h is a TEMPORARY header-only version so this stage builds on its own; stage 3 owns the file and its version replaces this one at merge (callers use AssembleToPlane(modules, image, width, height, PlaneBounds, gap) -> PlaneRaster{x0, y0, width, height, pixels}). Test: PreviewImage_PanelLayout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 10:28:23 +02:00
leonarski_fandClaude Opus 5.5	dc784c073d	Panel id on spots and the panel table in the start message ... SpotToSave gains int16_t panel (-1 = no panel table). It sits in the padding after the two bools, so sizeof(SpotToSave) stays 80. It is serialised only where it is set: CBOR spot key "panel", HDF5 /entry/MX/peakPanel (data file, plus a virtual dataset in a VDS master when the start message has panels), read back by HDF5MetadataSource. For a flat detector none of these is written, so files and streams for existing data are byte-for-byte unchanged. The start message carries the panel table (detector_modules, CBOR key absent when empty), filled from the experiment; JFJochHttpReader takes it, so a stream viewer knows the panels too. AssignModules takes the spot's panel id where it has one - the stored pixel box it was found in - and falls back to the plane position otherwise (every spot until per-panel spot finding sets the id). Reflection deliberately gets no panel id (owner decision): it is needed only between prediction and integration, and the struct feeds GPU layout and memory budgets. Tests: CBORSerialize_SpotPanelId, CBORSerialize_StartDetectorModules, JFJochReader_PanelId (VDS/legacy/integrated, with and without panels). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 10:28:23 +02:00
leonarski_fandClaude Opus 5.5	dcad4104ce	Dense per-pixel maps through the panel table (panel layout stage 1) ... Every dense map that needs geometry now evaluates each image pixel at its place on the detector plane (DetectorModules), so it is right for a stacked panel layout and picks up refined module placements: - AzimuthalIntegrationMapping: bins, resolution, solid-angle/polarisation corrections built per pixel through the table; a pixel on no panel is left as a masked one. The geometry cache key now includes the panel table, so a mapping built after the stills module placement is rebuilt rather than served from the cache. ResolutionMaskBits follows (it is a lookup). - ROIMap / ExportROIMap: azimuthal ROIs on the plane (box and circle ROIs stay in image pixels). - ROIIntegration CPU+GPU: the weighted sums are taken per ROI and panel in image pixels and placed on the plane at the end (exact; the affine commutes with a weighted mean). - HotPixelFinder: ring/sector keys and the zeta of the persistence rule on the plane. - Background beam-centre walk CPU+GPU: pixels binned at their plane position; host and device share BackgroundPlanePoint so they stay bit-identical. New common/PanelRows.h: ForEachPixelInRows, ForEachPixel cut to a band of rows in index order, for the row-parallel builders. An empty table takes the old arithmetic in the old order: rotation sets and every single-panel input are unchanged. [PanelLayout] AzintMapping and ResolutionMask enabled (+ azimuthal ROI map and ROI centroid on the plane); new BeamCenterFromBackground_BinsOnThePlane and ROIIntegrationGPU_MatchesCPU_Panels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 10:26:30 +02:00
leonarski_fandClaude Opus 5.5	c8fb472f4d	tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout ... A synthetic 2 x 2 panel detector, its panels turned 0/180/90/-90 degrees, recorded twice: assembled with the gaps masked, and stacked one panel under the other with the DetectorModules that place them - the same value in every physical pixel. What is already layout-correct runs as [PanelLayout]: the panel table itself, spot centroids on the plane through SpotAnalyze, and predictions going plane -> panel -> Bragg engine -> plane. The equivalences the next stages have to deliver are hidden cases tagged [PanelLayoutTodo], failing today: the azimuthal-integration map and the resolution mask per stored pixel (stage 1), Bragg integration measuring the same intensity (stage 2), and spot finding that never joins two stacked panels (stage 3). A stage that makes its case pass retags it [PanelLayout]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 09:41:35 +02:00
leonarski_fandClaude Opus 5.5	e4eaeba874	AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets ... tools/battery/still/agipd_to_int16.py converts an AGIPD 1M Cheetah CXI file into the only form rugnux supports for AGIPD: each module's eight 64-row ASICs placed 66 rows apart (the 2-pixel ASIC gaps inserted), one 526 x 128 panel per module; the per-frame mask and mask_shared (bad bits = the geometry's mask_bad) and the geometry's bad regions folded into the data as -32768; values round(value / ADU per photon) as int16 (clipped to +-32767; no pixel clipped in either set); bitshuffle/LZ4, one chunk per frame at /entry_1/data_1/data. It writes the matching CrystFEL geometry (16 panels, each the rigid fit of its ASICs, module coffset kept) and refuses a module whose ASICs are not on one straight line at a whole-pixel pitch within 0.5 px (worst here: 0.16 px for CXIDB 83, 0.002 px for 152). Reader: a stored value equal to the smallest value of its signed integer type (INT16_MIN, INT32_MIN) is no measurement. The reader decodes to float and divides by the gain, so -32768 used to come out as a count; the CrystFEL parser takes only module geometry, so a flag_lessthan in the .geom could not say it. Still battery: euxfel83 / euxfel152 read conv/*_int16.h5 with conv/agipd_int16.geom and --gain 1. Full sets, -N 16, --refine-geometry=off, --model (other jobs on the machine, load 7-23): indexed d_min CC1/2 CC_HALF_SPLIT R_split R_FREE main pass wall CXIDB 83 original 90.9% 1.84 0.641 0.617 0.482 0.280 89.3 s (31/s) 105 s converted 94.1% 1.84 0.686 0.644 0.470 0.277 6.5 s (429/s) 20 s CXIDB 152 original 17.5% 5.37 0.521 0.479 0.590 0.308 131.7 s (29/s) 153 s converted 62.8% 2.92 0.876 0.891 0.354 0.284 13.2 s (287/s) 27 s Files 4.7 -> 1.3 GB and 5.7 -> 1.1 GB. The 152 gain is the mask: the reader took only Cheetah's generic bad bits, while the geometry's mask_bad (0xffff) also marks three broken modules (bit 32) and its bad region a fourth; the converter run with the generic bits instead gives 17.5% / 3.41 A again. With geometry refinement on (16 rigid modules): 83 94.2%, 1.75 A, CC1/2 0.665, R_FREE 0.284, 49 s; 152 65.1%, 2.94 A, CC1/2 0.843, R_FREE 0.278, 27 s - mixed, the manifest keeps it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 09:21:33 +02:00
leonarski_fandClaude Opus 5.5	ebce906025	Serial XFEL stills inputs: readers from the files themselves, module geometry and its refinement ... Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto rc178 with the owner's decisions for this release: - Inputs are files, never a directory; several files of one format are one dataset (the multi-input CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles, ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and xtc::CollectStreams are gone, CanRead* accept a regular file only. - One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds: per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work). - Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib, bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through HDF5's filter under the global lock, one frame at a time. - LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all. - CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path) and optionally the camera length; everything else is ignored with a note in the log. - --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored layout (what a .geom's mask_file used to give). - Gain measured on the background for panel-table and CXI input that states none (--gain overrides). - Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions are mapped through the modules, the image is not re-assembled. - Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table, .geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx NXdetector_module groups), read back with --module-geometry. - A panel-table file's camera length is nominal: with a reference cell it is refined from the cell. - Broker and writer output unchanged: masters keep the single NXdetector_module group, the start message carries no module list. Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation], SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-20 23:30:48 +02:00
leonarski_fandClaude Opus 5.5	6557ba87f4	rugnux: several stills inputs in one run; inputs are files, never a directory ... Every input is a file named on the command line, read in the order given (a shell glob gives them sorted by name); a directory is refused with the glob form to use instead. - Several Jungfraujoch/NXmx masters of stills are read as one run (JFJochMultiReader): their images one after another, the pixel masks combined, the per-image efficiency placed at each input's images. They must share the detector, wavelength (0.1%), distance (0.1 mm) and beam centre (1 px), since stills geometry refinement refines one geometry for all frames; rotation sweeps are refused. The log gives the indexing rate of each input beside the run's (ProcessResult::image_indexed). - SACLA HDF5 / CXI (JFJochCrystFELReader::ReadFiles, ReadCXI, Open) and XTC (JFJochXTCReader::ReadFiles) take the list of files instead of scanning a directory; CanReadCXI and xtc::CanRead accept only a file; ListFiles and xtc::CollectStreams are gone. - Several inputs must be one format, --mode mx, and stored files (no broker, no --simulate-live); one frame of a CBF/marCCD/SMV sweep still names the whole sweep. Validated: two Jungfraujoch lysozyme stills runs together (49,920 frames, known cell) 57.1% indexed (59.3% / 54.9% per run), 1.22 A, CC1/2 0.986, R_split 8.8%, REFMAC R/R-free 0.253/0.269 against 0.257/0.273 for the earlier hand-built combined master. Rotation: 5reo and lyso_x10sa_90deg_1 unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-20 23:30:17 +02:00
leonarski_fandClaude Opus 5.5	093069be29	rugnux stills: a given cell is matched in any setting, and per-image reindexing moves the lattice with the indices ... Three defects that stopped known-cell stills runs: - The consensus cell compared every frame's cell to -C axis by axis. A run whose frames index in another setting of the same lattice (axes in another order, another monoclinic choice) had no frame accepted, no consensus cell and no output ("No crystal lattice was determined"), although it merged fine. A frame whose lattice contains the given cell now counts on the given cell's axes (CellOnReference, the same FindSublatticeBases the merge-stage relabel uses). - The per-image ambiguity resolver relabelled a frame's reflections but left its lattice on the old axes. Everything after it predicts from index and lattice: the sublattice relabel put such a frame through a second transformation, and on a monoclinic cell with a near-symmetry (a+c nearly as long as a) the merge came out at indices finer than anything measured, shells over 100% complete. The lattice now follows the indices (ReindexLattice, B' = R^-1 B). - With -C/-S the stills path replaced the known-cell indexer's lattice, already on the given axes, by LatticeSearch's conventional choice, which on that near-symmetric cell is the other description on half the frames. The indexer's lattice is now kept when it is close to the given cell. Measured (cell + space group given, dimple against the deposited model and free set): - monoclinic MbCO-type still set: was over-complete/broken with --model; now 2.06 A, R/Rfree 0.233/0.314 at 2.11 A (P1 merge before: 0.275/0.306). - C-centred orthorhombic set: no MTZ before; now 3.27 A, R/Rfree 0.298/0.403 at 3.68 A (de novo 0.336/0.448). - monoclinic two-form set, one form: cell P 1 21 1 with beta (was alpha); Rfree 0.310 -> 0.260, 3.05 -> 2.72 A. - No change on seven other stills sets, nor on two rotation battery sets (5reo, lyso_x10sa_90deg_1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 20:56:07 +02:00
leonarski_fandClaude Opus 5.5	990bbaf9a7	GivenCellPopulation: the population nearest the given cell among those holding a real share of the frames ... The peak nearest the given cell could be a single stray frame when the given cell is a few percent off the crystal (a cryo cell given for room-temperature lysozyme stills): the run then merged one frame. Every population is now found from its own peak and valleys, and of those holding at least a tenth of the frames the one centred nearest the given cell is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 18:39:41 +02:00
leonarski_fandClaude Opus 5.5	380cdf3dc7	rugnux stills with a given cell: one crystal form merged, distance and centre searched together, a stated distance kept ... - A given cell keeps only the frames whose refined cell belongs to its population (GivenCellPopulation): the peak of the refined-volume histogram nearest the given cell, out to the valley that separates it from another population, with the bin set by the data's own spread. A sample of two crystal forms indexed against one form's cell otherwise merges both; a run with one population keeps every frame. - The distance grid is run from every centre hypothesis, not only the one decided at the file's distance, and a local grid of centres follows at the distance decided. A wrong header distance and a centre several pixels off (both background estimators misread it on a Rayonix set) otherwise leave each other flat; now the cell alone finds both. - A distance given with --detector-distance is not searched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 18:02:10 +02:00
leonarski_fandClaude Opus 5.5	e4cb3e1b09	rugnux stills: the saturation level read off the frames; Cheetah CXI only for a square frame ... Foreign stills files state their saturation level wrongly or not at all: an MPCCD Cheetah file is dark-subtracted per pixel, so each pixel clips at the ADC limit less its own dark, and its CrystFEL geometry leaves max_adu commented out - the run reported "Overloads: 0" while the inner panels saturated in most frames. saturation_ceiling (image_preprocessing/SaturationCeiling) histograms pixel values on a log scale (64 bins per octave from 256) and returns the lower edge of the lowest bin holding over twice what any of the 8 bins below it held (and at least 50 values): a bright tail only falls, a clipped one piles up below where it stops, as a band where the level differs per pixel. DetectSaturation runs it on 40 frames spread over every stills run and logs it beside the file's limit; for any input but a Jungfraujoch-written file (ProcessConfig::detect_saturation) a level below the file's replaces it. The CrystFEL/CXI reader, where the geometry states no max_adu, now takes the stored integer type's maximum (an int16 Cheetah file clips at 32767). The geometry-file-free CXI path (previous commit) is limited to square frames: a tiled detector's Cheetah slab (CSPAD, AGIPD, MPCCD, ...) read as one flat panel would place every pixel wrongly without a word, and those slabs are not square. Measured: MPCCD stills, 1339 frames: pile-up from 1218 photons (by hand 12000 ADU ~ 1200 ph); 5.5% of reflections coarser than 4 A flagged; lowest-shell CC1/2 0.545 -> 0.674, overall 0.741 -> 0.793; same dimple refinement against the deposited model R-free 0.2586 -> 0.2556 single-panel Rayonix CXI, 1316 frames: no pile-up (20 pixels at the int16 limit in 40 frames); the int16 limit flags 123 reflections; CC1/2 0.737 -> 0.787, rugnux R-free 0.382 -> 0.358 JUNGFRAU stills (Jungfraujoch files: lysozyme and two private sets): no pile-up below the stated limit, results bit-identical Cost 0.5 s per run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 16:37:25 +02:00
leonarski_fandClaude Opus 5.5	6a1d859d49	Stills merge: R_split and a frame-split CC1/2, per shell and overall ... Serial crystallography quotes R_split and a CC1/2 whose half-sets keep each crystal whole (CrystFEL's partialator), and rugnux's stills report had neither, so its numbers could not be set beside CrystFEL's. MergeOnTheFly now also sums every frame's observations into the frame's own half (the existing per-image hash preference), and MergeStats reports from those two means R_split = (1/sqrt 2) sum|I1-I2| / (1/2 sum(I1+I2)) and their CC1/2. They are printed as a "Serial statistics" table under the shell table (also in the --report-resolution table) and as R_SPLIT / CC_HALF_SPLIT and their REFRES_ keys. The balanced per-reflection split behind CC_HALF is unchanged, and so is everything decided on it (resolution cut, space-group search, pass gates). Rotation output is unchanged: RotationScaleMerge does not fill the new fields and the table is not printed. On a 4000-frame jet subset over 30-1.5 A: CC1/2 0.830, R_split 31.8%, against 0.834 / 31.3% computed by hand from two separate even/odd runs and 0.842 / 30.5% from CrystFEL on the same frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 15:40:30 +02:00
leonarski_fandClaude Opus 5.5	2a45eeb831	JFJochXTCReader: unit test with a synthetic XTC stream and a one-panel DIALS .expt ... The test writes a Configure and two L1Accepts, the first with a second, smaller camera beside the diffraction detector, and checks that the reader takes the larger image, reads the geometry, wavelength and polarization from the .expt, and hands out the stored pixels. The reader itself (previous commit) now builds and has processed an LCLS MFX Rayonix stream end to end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 15:10:10 +02:00
leonarski_fandClaude Opus 5.5	acf6b3604f	tests: CrystFEL geometry parser - old-syntax fixture and a multi-panel file with path-valued clen, directions, masks and bad regions ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 15:09:57 +02:00
leonarski_fandClaude Opus 5.5	9e09cd4b3d	broker: initialize drops the dataset's bearer tokens and everything they protected ... Initialize is the fresh start between users. It already cleared the images, plots, scan result and live rugnux status; it now also drops the tokens, resets the dataset settings (name, file prefix, unit cell, ...) to their defaults and forgets the last collection followed through /live/events, so after it nothing of the previous dataset is readable, with or without its token - the broker is as open and as empty as a freshly started one. The auth test checks that after /initialize the data-collection statistics and the /statistics aggregate are open and no longer carry the previous dataset's name. SECURITY.md also lists /rugnux/status among the protected endpoints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 12:16:32 +02:00
leonarski_fandClaude Opus 5.5	6f5c235fd7	Live molecular replacement: warmed up at model upload, searched beside the second pass and carried to the answer's merge ... - WarmUpModelValidation did nothing for a prediction (no crystal cell: "no atoms or no unit cell"), so a broker's first molecular replacement paid every first use on the GPU. A prediction is now warmed up by a search of its own (WarmUpMolecularReplacement): its search model in a P 1 21 1 cell four radii on each edge, that crystal's intensities to 2.5 A, the search, and the validation of what it placed. RunMolecularReplacement waits for a warm-up still running, as ValidateAgainstModel does. --simulate-live warms the model up as the broker does at upload. Result-identical. - fast_live with a prediction (or --mr): the search starts on the geometry pre-pass's merge, averaged into a group from that merge's own space-group search (to 3 A, standard setting), beside the second pass, on card 1 % cards, a quarter of the threads, under GPUWorkBeside. At the answer's merge CarryMolecularReplacement takes it over where it holds - same group and a cell within 1%, every copy refined by the rigid body beside the others against the answer's merge and its R-free within R_FREE_MARGIN of the one it was found at - and the merge is searched itself otherwise (and always where the twin hypothesis' merge is searched too). RESULT-CHANGING only in the refinement end point of the placement. Sim-live (10 ms/frame, RTX 5080): molecular-replacement step thaumatin 0.96 -> 0.31 s, cytochrome C 1.19 -> 0.25 s; merged reflections identical; thaumatin placed model identical (0.000 A), R-free 0.3765 both; cytochrome C a symmetry-equivalent placement, CA rmsd 0.03-0.05 A per copy, R-free 0.4443 -> 0.4431. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 07:51:07 +02:00
leonarski_fandClaude Opus 5.5	147f750c90	broker: rugnux live settings and model need no bearer token ... GET/PUT /config/rugnux and PUT/DELETE /config/rugnux/model prepare the next collection and expose nothing of the current one, so they are open like the rest of the configuration. Protecting them meant the next user needed the previous dataset's token to set up their own run. /rugnux/status, which reports the current collection, stays protected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 07:43:47 +02:00
leonarski_fandClaude Opus 5.5	1949155c12	Rotation: a blank sweep end is trimmed before scaling, and a delta-CC1/2 stretch takes the sliver it leaves at an end (RESULT-CHANGING) ... Shutters that open late or close early, and beams that go off, leave frames at the ends of many sweeps with no exposure at all. Until now those frames went into the scaling like any other: on a CCD their noise, divided by an incident flux of ~0.005, made the partials loop walk their scales toward zero until the 50x collapse guard dropped them - or, if the loop stopped first, merged one as noise, which delta-CC1/2 then removed with its whole 10 deg batch and reported as SWEEP_GAPS (3p85, frames 338-359). On photon-counting detectors the blank frames fitted G = 0, escaped the guard and were reported as merged although nothing of them was (cytc_x10sa, myob_x10sa, thau_micromax_pink, lysoI_micromax_pink, 3p85). TrimBlankSweepEnds (RotationScaleMerge.h) walks in from each end and clears the reflections of every frame that - recorded no diffraction: as many of its reflections stand more than 3 sigma below zero as above it (a sign count, so no sigma calibration is needed and a slight background bias cannot hide the strong reflections), and that excess is 3 sigma clear of the run's typical frame - so a crystal too weak for one frame to show it is never trimmed; and - had no beam: its mean background is nearer to none than to the run's typical frame. That is what keeps a crystal leaving the beam, which also records nothing but still scatters, in the sweep ledger as a finding (insu_I_x06da_ref, 9zm0, 9fhc, 5reo are untouched). No stretch length: one blank frame is as blank as twenty. Partly exposed first/last frames recorded diffraction and stay (7tcd, 7q6j, 9z72, lyso_micromax_pink). It runs on the integration outcome before the overload count, so the merge, post-refinement and unmerged output all see the trimmed sweep, and in --mode scale on the stored reflections. The report gives FRAMES_BLANK_START/END and a prose line - information, no warning; the frames count as rejected. The sweep ledger looks for degraded stretches, and reads its running medians, only between the first and last frame with data. Delta-CC1/2: a convicted stretch that stops short of a sweep end by less than one rocking event takes the rest, where those frames read as the stretch does (5reo: 331-396 rejected, 397-399 were left in as a three-frame weak_diffraction range with its own warning). Battery --only, 31 sets, tol5 f217f180a -> this commit: no space group, d_min or verdict changes; merged statistics identical within 0.001 except 5reo (R_meas 0.236 -> 0.231, ISa 23.0 -> 23.3, R-free 0.2013 -> 0.2004, warnings 3 -> 2). Trimmed: 3p85 0/19 (+3 empty frames; collapse-guard drops 13 -> 0), 6w4h 0/5 (guard 5 -> 0), 7rji 0/4 (delta-CC1/2 no longer removes 3406-3412, two of which diffracted: 7 -> 0), cytc_x10sa 0/2, thau_x10sa_* 0/1-2, myob_x10sa 0/1, lyso_x10sa_strong 0/1, 8a1a 0/1, lyso_micromax_mono 1/0, lysoI_micromax_pink 2/0, thau_micromax_pink 1/0. Controls (11if, 5f6m, 6moj, 8u0i, 9fhc, 9z44, 9z72, 9zm0, 6iu6, insu_I_x06da_ref, myob_x06da_split, lyso_x06da_half_image, thau_bl1a_4p6keV) trim nothing and are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 02:18:19 +02:00
leonarski_fandClaude Opus 5.5	97ef0936b2	CudaFFTPlan: one cuFFT plan per card and shape, shared by its users; rigid body transforms one grid at a time ... Every cuFFT plan has the driver compile cuFFT's PTX kernels: cufftMakePlanMany -> cuModuleLoadData -> the PTX JIT, under cuFFT's process-wide lock. A second plan of the same shape pays again, even while the first is alive. The CUDA JIT cache makes that ~1 ms. Without one (CUDA_CACHE_DISABLE, no writable $HOME) a plan costs 20-250 ms. That holds for every size (64^3 and 1D 256 included), every transform type and batch, on sm_75 and sm_120, with cuFFT 11.2, 12.0 and 12.4. No CUFFT_* setting changes it, so a choice of grid sizes cannot avoid it; only the number of plans made can be cut. Before this change, a plan was lent to one user at a time, so each concurrent user of a shape on a card made its own: 9-24 rigid-body engines, the translation slices and the score engines. Now a plan is made once per card and shape and shared. Each user keeps its own stream and work area, and CudaFFTPlan::R2C/C2R set both and launch under the plan's mutex. Shared plans give the same bits as private ones: 8 threads on one plan with their own work areas, 9 shapes x 160 transforms, 0 mismatches on RTX 5080 and T4. A shared work area, the negative control, gives 2-13 mismatches per shape where the plan has a work area. The rigid-body engine also dropped its batch-3 plan: the Jacobian transforms its three grids one at a time on the batch-1 plan. That gives the same bits (10 zone-grid shapes, sm_75 and sm_120) and one plan less per zone and card. The cytochrome C and thaumatin MR test sets, AF models (MR step, s): 4x T4 cytc cache on 2.22 -> 2.22 cache off 8.41 -> 4.00 plans 86 -> 24 4x T4 thau cache on 1.58 -> 1.61 cache off 11.13 -> 3.08 plans 131 -> 23 1x T4 cytc cache on 2.87 -> 2.81 cache off 8.82 -> 3.84 plans 80 -> 12 1x T4 thau cache on 2.41 -> 2.39 cache off 10.42 -> 3.17 plans 104 -> 11 RTX5080 cytc cache on 1.41 -> 1.12 cache off 6.5-10.7 -> 2.02 RTX5080 thau cache on 0.90 -> 0.85 cache off 8.03 -> 1.60 Outputs are bit-identical before and after, cache on or off: model, merged hkl, maps, and every MR and validation log line. The mtz/cif differ only in the command line they record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 01:51:28 +02:00
leonarski_fandClaude Opus 5.5	02e5bc387f	broker: initialize and start clear the last collection's live rugnux status ... /rugnux/status kept answering with the previous collection's rugnux run after POST /initialize, and also after a /start whose collection does not run rugnux - the latter under the new run's tokens. RugnuxLiveStatus gets a Clear(), called from JFJochReceiverService::ClearResults(), which /start already uses to drop the previous run's status, plots and buffered images before the tokens change. Initialize now calls ClearResults() instead of ClearImageBuffer(), so a fresh start also drops the previous run's receiver status and plots. Authentication: every rugnux endpoint (GET/PUT /config/rugnux, PUT/DELETE /config/rugnux/model, GET /rugnux/status) was already in the broker's bearer-protected set and carries bearerAuth in jfjoch_api.yaml; the HTTP auth test now checks all five. The yaml descriptions of /initialize and rugnux_live_status say so; the C++ model and frontend client are regenerated (comments only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 01:00:54 +02:00
leonarski_fandClaude Opus 5.5	c2e5e4a4e8	broker: initialize clears the image buffer, turns live rugnux off and removes its model ... Initialize is the fresh start between users, so nothing of the previous one's session is left: the image buffer is cleared, live rugnux processing is disabled and the uploaded MR model is dropped. It is done at the start of Initialize, under m then rugnux_mutex (the order the other setters take), so it happens even if configuring the detector then fails. The receiver is always Idle when the state machine is not running, so the buffer clear has no unmet precondition. The dashboard's rugnux panel read the settings only once, on mount, so after an initialize from any client it kept showing the previous run; it now polls them like the Rugnux tab does. The /initialize description in jfjoch_api.yaml says so; the frontend client is regenerated (description only). The python client docs and redoc follow at the release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-19 22:11:51 +02:00
leonarski_fandClaude Opus 5.5	831b05aebd	Live rugnux panel: the timeline spans the collection's set-up length from the first poll ... The broker's rugnux live status carries expected_collection_s = images x image time (DiffractionExperiment::GetImageTime, summation included), set at START. The frontend's timeline uses it as the expected end until a frame rate is measured, and as the axis floor while collecting, so the axis no longer starts at 10 s and rescales as images arrive. An older broker without the field keeps the 10 s floor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-19 20:48:07 +02:00
leonarski_fandClaude Opus 5.5	7349832309	Live rugnux timing: from the puller's receive time, and from the first image ... The live panel showed ~80 Hz on a 100 Hz collection. Its clock started when the receiver had applied and forwarded the START message, so the START -> first image gap counted as collection time; the images were stamped when a forward thread dequeued them, and the last image only after END and the thread joins; and N images were divided by a span holding N-1 intervals. - ImagePullerOutput carries `received`, a steady_clock time set by the TCP and ZeroMQ pullers as the message comes off the socket, before it is queued or deserialised (TestImagePuller stamps it on Put when unset). - RugnuxLiveStatus takes those times: START's for its origin, each image's for the first and last image (min/max, as forward threads report out of order). last_image_s is the last data image's puller time. - Zero of every reported time (elapsed_s, steps' started_s, probe_s, last_image_s, cut_s) is now the FIRST IMAGE, as the frontend already said; before the first image arrives "now" is 0 and earlier steps are negative. The START -> first image time is recorded on its own as start_to_first_image_s (snapshot, OpenAPI rugnux_live_status, regenerated C++ model and frontend client, OpenAPIConvert). - The frontend rate is (images_received - 1) / (last - first image); the panel headers show "START -> first image X.X s". - The broker logs "rugnux live: START -> first image ..., N images in T s from the first to the last (R Hz)"; "done X s after the last image" and last_image_to_written_s run from the puller time of the last image. The deadline timer is unchanged (it still starts once the forward threads have joined). No analysis result changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-19 19:35:54 +02:00
leonarski_fandClaude Opus 5.5	e724f3d26a	Beam-centre gates: decide on the strong-reflection count, not on how a scaling loop stopped ... MeasuredCentreClearlyLoses, different-lattice branch: the measured centre loses on the pre-pass subset when it measures clearly fewer strong reflections (Poisson, 2 sigma) than the file's centre. The subset CC1/2 is no longer read there (still logged). A third of the frames in five blocks merges to a multiplicity of ~1.09 in P1, so its CC1/2 reads the relative scale of the blocks: on 9gdj the file centre's subset CC1/2 moved 0.899 / 0.880 / 0.832 / 0.846 with the scaling path (plain 1e-3 / 1e-5, Anderson 1e-3 / 1e-5), ~50x the Fisher noise the gate allowed, and flipped the early verdict. The count is comparable across lattice classes (taken on the primitive lattice) and stable across paths: 36.6-38.6k against 103.4-103.6k on 9gdj. MeasuredCentreWins, same-lattice branch: the measured centre has to be ahead on the strong count by more than 2 Poisson sigmas; within that the file's centre stays (it adopted at 1874 vs 1873 on one in-house small-molecule set, and at 0.4 sigma on 9zmu). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-19 18:49:24 +02:00
leonarski_fandClaude Opus 5.5	0894230671	Anderson: accelerate a scaling loop only near its fixed point, and never again once its step grows (RESULT-CHANGING) ... The Anderson commit before this one changed three rc176 battery results, which is why it was reverted there. One of them is the acceleration's own doing: on a split myoglobin sweep (P2_1, ISa 2.5) the partials loops never settle - one walks at a step of ~2e-2 for as long as it runs - and the fulls loops walk from their first round (their cross-validated lambda moves every round, so the map itself is not fixed). Restarting the history whenever the step grew only rebuilt a one- or two-round history and extrapolated along the walk: the stalled loops ended somewhere else, the pre-pass fulls smoothing window went from ~7 frames to ~1, and CC1/2 fell 0.701 -> 0.595. Two rules now confine the acceleration to the regime it is for - a map contracting to its fixed point: - a loop is accelerated only once its plain step is below ten times the tolerance (1e-2); - the first round whose weighted residual grew turns the acceleration off for the rest of the loop (AndersonMixing::Stopped), instead of restarting it. A loop that walks therefore stays the plain loop it was in rc176. The growth rule alone gave CC1/2 0.625; with the onset, 0.701, as rc176. Measured against rc176 bc191bb80 (GPU, workstation): - myob_x06da_split: P2_1, CC1/2 0.701, d 1.81 A - as rc176; 481 -> 257 scaling iterations. - 7qij 434 -> 126 iterations, 9v16 273 -> 101, same space group and statistics. - ci tier (29 sets, no null model): 28 unchanged beyond noise; sum of times 580 -> 585 s (9gdj +9 s, below). The two other changes are NOT Anderson's and stay: - insu_I_x06da_weak I23 -> I222: the all-observations search refuses I23 on a merge chi2 ratio 2.08 against a 1.85 bound; plain scaling converged to 1e-5 gives 2.07-2.09 and refuses it too. Only the plain loop stopped at 1e-3 passed - the bound was set on unconverged scaling. - 9gdj +9 s, same answer: the early beam-centre comparison's "loses beyond noise" margin (-0.002 against 2 sigma 0.0016) moves by ~0.07 between scaling paths, so the converged loop no longer takes the shortcut and both centres are judged on all frames; the file's centre is kept either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-19 18:49:24 +02:00
leonarski_fandClaude Opus 5.5	138abafa14	RotationScaleMerge: Anderson-accelerate the per-frame scaling loops (RESULT-CHANGING) ... The partials and fulls scaling loops are alternating least squares that converge linearly: over the rc176 battery's 2527 loops a round shrinks the step by a median 0.86 (partials) and 0.93 (fulls), up to 0.98, so the 1e-3 step tolerance stopped them 0.6% / 1.4% (median) and up to 17% (fulls p90) short of their fixed point, and 313 of them were stopped as "stalled" when they were only slow - a stalled fulls loop also skipped ShrinkToRestrained. Each round is now accelerated in log G by type-II Anderson mixing (Walker & Ni 2011), depth 5, each frame weighted by its observation count as in the step test; frames below the credible floor take the plain round's scale. The history is dropped when the weighted residual grows or the set of fitted frames changes (new AndersonMixing class, unit-tested on a synthetic slow ALS loop). The mixed scales are gauge-pinned again before the rescale, and the stop test is unchanged: it reads the round's residual F(x) - x, so "stalled" now means a loop that does not settle under acceleration either. Measured (CPU, -N 8): 4nwv 311 -> 82 iterations over 9 loops, 9v16 76 iterations over 11 loops, the same counts as the env-gated prototype on tail-solver; no history restart fired on either. Depth 3/5/8 give 92/91/90 iterations on 4nwv. A restart on a change of the fulls loop's cross-validated lambda was tried and gave nothing (83 vs 82). Same space group and merging statistics on 4nwv and 9v16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-19 18:49:24 +02:00
leonarski_f	7ed32b899f	v1.0.0-rc.177 (#87) ... * jfjoch_broker: Faster live Rugnux analysis, with the GPU indexers prepared at the start of a collection and kept between collections; Rugnux no longer writes `.ccp4` maps (the map coefficients are in `<prefix>_maps.mtz`). Reviewed-on: #87 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 16:07:31 +02:00
leonarski_f	8261b7354e	v1.0.0-rc.176 (#86) ... * jfjoch_broker: Optional live Rugnux analysis of DECTRIS rotation collections inside the broker, with merged data, maps and the placed model written beside the master file, the rotation lattice in the scan result, and a "Rugnux live" tab and dashboard row in the web frontend. * Rugnux: Faster processing, with molecular replacement and model validation spread over several GPUs and run on the Apple GPU on macOS. * Rugnux: The polarization stated in miniCBF and d*TREK headers is used, twinned crystals' space groups are decided more reliably, and a run that finds no lattice still reports every image. Reviewed-on: #86 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 07:03:28 +02:00
leonarski_fandFilip Leonarski	19b135186d	v1.0.0-rc.175 (#85) ... * Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`). * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS). * Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead. * Rugnux: On macOS, Bragg integration runs on the Apple GPU. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. * Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`. * Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. * Rugnux: The beam-stop shadow is found correctly on tilted detectors. * Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement. * jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps. * jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images. * Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. --------- Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home> Reviewed-on: #85 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-16 15:34:43 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00