194 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	dda86080e2	BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain ... background_gain::Estimate is the gain of any input that reports detector units: the photon peaks where the histogram of a few frames shows them (PhotonPeak), else the background's variance per unit mean (Gain). The stills reader path (MeasureGain) and the pre-scan's CCD measurement now both call it, so a rotation sweep and a still read the same number from the same frames. On a CCD the photon-peak test now also sees the pixels outside the active area, all at zero: on one Saturn sweep that spike, the background hump far above it and the hump's echo at twice the value passed as photon peaks (gain 2725). A zero-photon peak is the detector's noise and has a width, so a single bin with nothing beside it is refused. Checked on the 39 SMV/marCCD sets of the battery: every pre-scan gain is the same as rc178's (no photon peaks; the variance per unit mean as before). Where the gain is applied stays as it is: a stills file is converted to photons as it is read (the spot finder and the stills measurements of spot width, bandwidth and partiality work in photons), a rotation CCD sweep keeps its raw counts and the gain enters the integrator (pixel_gain, 46da7fb35). Moving the stills to the integrator route too was tried (stills PixelGain, no reader conversion) and lost badly on MPCCD: R-free +0.032 and +0.047 on two SACLA sets, R_split 0.055 -> 0.091 and 0.067 -> 0.144, with the indexing unchanged; the stills rocking width/bandwidth fit, measured on frames in ADU, collapsed (bandwidth 0.0015 -> 0.0000). still_manifest.json: euxfel152 drops --gain 62; the gain is now read off its photon peaks (62.65, on a baseline of 124.2 left in the frames): R-free 0.252, the same as with --gain 62.649 or 61.5 given by hand (0.252, 0.252); --gain 62 gives 0.2495 - the R-free moves by ~0.003 with the gain. 66.4% indexed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-23 04:01:11 +02:00
leonarski_fandClaude Opus 5.5	b6f4973908	CHANGELOG: rc.178 condensed - XFEL reading and stills processing, Mol* view on the live page ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-23 01:17:19 +02:00
leonarski_fandClaude Opus 5.5	e6b0dfafca	Merge rc178 (Mol* viewer, changelog rules) into stills-tilt ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-22 22:13:20 +02:00
leonarski_fandClaude Opus 5.5	4abf498f54	ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI) ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-22 20:24:42 +02:00
leonarski_f	6abc44710d	Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer ... Model validation already made the anomalous difference map's coefficients ((F(+) - F(-)) on the model phases less 90 degrees, where the merge kept the Bijvoet pairs) to read the anomalous sites, then threw them away. They now go into the map coefficients as ANOM/PHANOM (phenix.maps' labels), missing on the reflections without a pair, so _maps.mtz opens as an anomalous map in Coot. Nothing in scaling, merging or the validation's decisions changes; a merge without the Bijvoet split writes the twelve columns as before. The broker serves the map as /rugnux/map.ccp4?type=anom (404 without Bijvoet pairs), and rugnux_live_status.anomalous_map says it is there. The Mol* viewer loads it only then, as a magenta mesh at +4 sigma with its own level slider and switch. Tests: ANOM/PHANOM appear with Bijvoet pairs (as many as the reported pairs) and not without; the served anomalous map of a point peaks at it; the auth test covers type=anom. End to end (myoglobin 2M replay): anomalous_map true, the served map equals gemmi's FFT of ANOM/PHANOM in the written _maps.mtz to 3.4e-6 rms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c4d6ad74c706181cbb183cfcaa2481653cd7cad6)	2026-10-22 20:18:51 +02:00
leonarski_f	ee016d3d13	rugnux live maps: one unit cell centred on the model; the viewer opens on all of it ... /rugnux/map.ccp4 now serves one unit cell of map, centred on the model, instead of the model's box plus 5 A; along an axis where the model reaches further than the cell (a predicted model with long termini in a small cell), the model plus 5 A instead, so that all of it has density. The Mol* viewer opens on the whole cell and the model (map radius 0); a radius still draws the maps only around the centre of view, Coot-style. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 1c1e1d96359f112d6820d972005e9f7eca4ebd50)	2026-10-22 20:18:51 +02:00
leonarski_f	90bbba33bc	writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does ... The placed-model END block lost everything but the atoms, so the writer's _model.cif had no sequence references, authors, software or secondary structure, and gemmi relabelled the subchains. The block now also carries label_asym_id, label_entity_id and label_seq_id per atom and the input model's metadata as the mmCIF categories gemmi writes without coordinates (metadata_mmcif), which the writer parses with gemmi and puts back. The writer's _model.cif is now the same text as the one rugnux writes itself and as /rugnux/model.cif serves. Older blocks without the new fields are read as before. Test: a predicted myoglobin model through END -> CBOR -> writer gives byte-identical _model.cif/_model.pdb to WritePlacedModel and to PlacedModelMmcif, with its _struct_ref, _entity_poly_seq, _audit_author, _software and _struct_conf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 29d752e6a107e3b8bc71ad6e9af0d81064966537)	2026-10-22 20:18:51 +02:00
leonarski_f	15ca4b329f	broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4) ... A rugnux live run that placed a model keeps it, with a copy of its map coefficients, beside the live status (RugnuxStructure), for as long as /rugnux/status keeps the run. Made in memory on the first request and cached, never written to disk: - GET /rugnux/model.cif: the placed model as mmCIF, labelled as WritePlacedModel labels _model.cif (PlacedModelMmcif, the labelling factored out of WritePlacedModel; the file output is unchanged). - GET /rugnux/map.ccp4?type=2fofc|fofc: the 2mFo-DFc / mFo-DFc map as a CCP4 map, FFT of the FWT/PHWT or DELFWT/PHDELWT coefficients (get_f_phi_on_grid + MapFromFPhi, sampled as ModelValidation samples), cut to the box around the model plus 5 A, with the whole cell's mean and RMS in the header. - 404 without a placed model or maps, 400 for an unknown map type; both routes protected by the dataset bearer token like /rugnux/status. rugnux_live_status gets placed_model. C++ model, python client docs, TypeScript client and redoc regenerated. Tests: a point's coefficients give a map peaking at the atom in a cropped box; the mmCIF; the structure is cleared at the next start; the auth test covers the new routes. End to end (myoglobin 2M replay with an AlphaFold model): the maps equal gemmi's FFT of the written _maps.mtz to 2.5e-6 rms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 33124c3288ba112def3f51c0d91f5206886f3495)	2026-10-22 20:18:50 +02:00
leonarski_fandClaude Opus 5.5	be45f99298	stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes ... The absolute distance scan ran at one centre on the 8 strongest of 96 sampled frames, at the run's indexing tolerance. Three ways that found nothing, and the run then stayed at the file's value or, on an XTC, at the 100 mm placeholder: - a sparse I-centred lattice whose file states 180 mm for a detector at ~135 mm indexed at no distance from the background centre (7 px off) - only from the file's centre. Every centre hypothesis is now scanned; - a 92 A cell at ~199 mm indexed on no centre 3 px off, nor at a distance half a scan step off. The scan's frames are now indexed at tolerance 0.2 instead of 0.1; the scan only has to find the peak, and every later step (the peaks on 32 frames, the fine step, the centre grids) counts at the run's own tolerance; - on a run with a hit rate of a few percent the 8 strongest of 96 frames held no crystal. Where nothing indexes at any distance at the run's tolerance, the sample is made ten times larger and the search run again, the scan on 16 frames. With these an LCLS XTC (no distance stated) with two crystal forms finds 198.93 mm against the 199.0 it was given (R-free unpinned 0.236, pinned 0.238); a Rayonix CXI stating 839 mm finds 106.23 mm (0.206 vs 0.204 pinned at 105 mm); another XTC finds 193.41 against 193.43 (0.202 vs 0.200); the 180 mm file still finds 136.15 mm. Sets whose stated distance is right keep it, with unchanged results. The fallback costs ~30 s, only where it fires. RefineStillsGeometry also reports the slope of the indexed spots' signed excitation error against d*^2 at the geometry it starts from: positive where the distance (and the cell) is short, negative where long. Reported only. The usage message and docs no longer describe --detector-distance as the joint fit's starting value (it is held since 45abde17a), and say that an XTC with -C has its distance searched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-22 18:27:40 +02:00
leonarski_fandClaude Opus 5.5	72cc542d78	Stills keep an overloaded observation as a lower bound instead of dropping it ... The stills merge dropped every observation with a saturated pixel. On a still the saturated observations are the fully recorded ones of the strongest low-resolution reflections, so what survived were their partial observations, whose partiality correction then set the merged intensity: on clipped CCD stills the merged low-resolution intensities came out at 0.4-0.6x of the deposited ones, a ramp no Wilson B absorbs, invisible to CC1/2 and R_split. The pixels are really clipped (flat-topped plateaus a few per mille wide, a pile-up in the pixel-value histogram from the per-pixel ADC limit), so the auto ceiling is right; the drop was the defect. A still now keeps the observation: the box sum with every saturated pixel counted at the saturation limit, unflagged and marked clipped, counted in BraggIntegrationCounts and reported as OVERLOADED_REFLECTIONS_KEPT. Rotation data (IsRotationIndexing) are unchanged and keep the flag for the rocking-event drop. Measured on a clipped-CCD stills subset (dimple R-free, deposited free set): dropped 0.247, lower bound 0.230 (same as switching saturation off). A Kabsch fit to the unsaturated wings read lower than the lower bound (0.237), and inflating the lower bound's sigma lost (0.255). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-22 10:36:33 +02:00
leonarski_fandClaude Opus 5.5	82f90259a0	still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good ... The SwissFEL .geom files name their pixel mask with an absolute facility path, so the sets read a copy beside the original (jf-16M_mask.geom, mbco_march_mask.geom) whose mask_file is relative to the copy's directory; only that line differs. EXTERNAL_TEST_DATA_STILLS.md says how to make them. On a 2000-frame subset of the int16 MbCO set, before (geometry without masks + --mask-good) and after (the geometry's mask and bad regions): indexed 98.2% both, R_FREE (--model) 0.244 -> 0.234. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 13:17:34 +02:00
leonarski_fandClaude Opus 5.5	7c23b4befe	CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them ... The .geom parser now takes the masking information the format defines: - mask arrays: mask/mask_file/mask_good/mask_bad (mask 0) and maskN_data/maskN_file/maskN_goodbits/ maskN_badbits for N = 0..7; global keys hold for the panels after them, a panel may override. A mask that is the same on every panel is one whole-frame mask, otherwise one per panel, applied on that panel's pixels only. - a relative mask_file is looked for in the working directory (as CrystFEL does), then beside the .geom. - bad regions (bad*): a lab-plane box (min_x..max_y) or a fs/ss range on a panel (or, old syntax, on the whole frame); mask_edge_pixels and no_index. JFJochXFELReader already had the static pixel mask and the per-frame mask path for these; the masks go through them unchanged. A mask whose path holds % (one per shot group) is applied frame by frame like a 3D one. Where the geometry names mask arrays, the data file's own Cheetah masks (CXI mask_shared and per-shot mask with Cheetah's generic bad bits, /metadata/pixelmask) are not added: the geometry says which bits matter (e.g. an AGIPD mask_bad that includes bit 32). --mask-good / --mask-bad still add their masks to these. Tests: [CrystFELGeometry] (new keys, per-panel override, relative mask file), JFJochXFELReader_* (new GeometryMasks case on a synthetic CXI + .geom), [JFJochMultiReader], JFJochXTCReader*. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 12:42:08 +02:00
leonarski_f	30ea607686	Merge branch 'rc178-still-results' into geom-mask	2026-10-21 11:55:56 +02:00
leonarski_fandClaude Opus 5.5	a2b770fe43	still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed ... The AGIPD sets are run on the files as deposited (Cheetah CXI with the per-ASIC CrystFEL geometry), and their processing speed is reported as it is on those files; only the SwissFEL float32 sets keep an int16 photon copy. The reader rule that a signed integer frame's smallest value is no measurement stays: it is general. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 11:53:30 +02:00
leonarski_fandClaude Opus 5.5	dafb112d93	EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery" ... Each set's row now says how many images we use and, where that is a subset, what the entry or its deposition holds (SACLA 237 ~1.8% of dark_ref, LCLS 222 one of four runs, LCLS 242 runs 256-267, both AGIPD sets and A2A against their deposited frame counts). The section listing what is not used is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 11:44:25 +02:00
leonarski_f	64f6b7a5c4	Merge branch 'rc178-still-results' into panel-s5 ... # Conflicts: # reader/JFJochXFELReader.cpp # tests/XFELReaderTest.cpp	2026-10-21 11:28:17 +02:00
leonarski_fandClaude Opus 5.5	dc784c073d	Panel id on spots and the panel table in the start message ... SpotToSave gains int16_t panel (-1 = no panel table). It sits in the padding after the two bools, so sizeof(SpotToSave) stays 80. It is serialised only where it is set: CBOR spot key "panel", HDF5 /entry/MX/peakPanel (data file, plus a virtual dataset in a VDS master when the start message has panels), read back by HDF5MetadataSource. For a flat detector none of these is written, so files and streams for existing data are byte-for-byte unchanged. The start message carries the panel table (detector_modules, CBOR key absent when empty), filled from the experiment; JFJochHttpReader takes it, so a stream viewer knows the panels too. AssignModules takes the spot's panel id where it has one - the stored pixel box it was found in - and falls back to the plane position otherwise (every spot until per-panel spot finding sets the id). Reflection deliberately gets no panel id (owner decision): it is needed only between prediction and integration, and the struct feeds GPU layout and memory budgets. Tests: CBORSerialize_SpotPanelId, CBORSerialize_StartDetectorModules, JFJochReader_PanelId (VDS/legacy/integrated, with and without panels). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 10:28:23 +02:00
leonarski_fandClaude Opus 5.5	e4eaeba874	AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets ... tools/battery/still/agipd_to_int16.py converts an AGIPD 1M Cheetah CXI file into the only form rugnux supports for AGIPD: each module's eight 64-row ASICs placed 66 rows apart (the 2-pixel ASIC gaps inserted), one 526 x 128 panel per module; the per-frame mask and mask_shared (bad bits = the geometry's mask_bad) and the geometry's bad regions folded into the data as -32768; values round(value / ADU per photon) as int16 (clipped to +-32767; no pixel clipped in either set); bitshuffle/LZ4, one chunk per frame at /entry_1/data_1/data. It writes the matching CrystFEL geometry (16 panels, each the rigid fit of its ASICs, module coffset kept) and refuses a module whose ASICs are not on one straight line at a whole-pixel pitch within 0.5 px (worst here: 0.16 px for CXIDB 83, 0.002 px for 152). Reader: a stored value equal to the smallest value of its signed integer type (INT16_MIN, INT32_MIN) is no measurement. The reader decodes to float and divides by the gain, so -32768 used to come out as a count; the CrystFEL parser takes only module geometry, so a flag_lessthan in the .geom could not say it. Still battery: euxfel83 / euxfel152 read conv/*_int16.h5 with conv/agipd_int16.geom and --gain 1. Full sets, -N 16, --refine-geometry=off, --model (other jobs on the machine, load 7-23): indexed d_min CC1/2 CC_HALF_SPLIT R_split R_FREE main pass wall CXIDB 83 original 90.9% 1.84 0.641 0.617 0.482 0.280 89.3 s (31/s) 105 s converted 94.1% 1.84 0.686 0.644 0.470 0.277 6.5 s (429/s) 20 s CXIDB 152 original 17.5% 5.37 0.521 0.479 0.590 0.308 131.7 s (29/s) 153 s converted 62.8% 2.92 0.876 0.891 0.354 0.284 13.2 s (287/s) 27 s Files 4.7 -> 1.3 GB and 5.7 -> 1.1 GB. The 152 gain is the mask: the reader took only Cheetah's generic bad bits, while the geometry's mask_bad (0xffff) also marks three broken modules (bit 32) and its bad region a fourth; the converter run with the generic bits instead gives 17.5% / 3.41 A again. With geometry refinement on (16 rigid modules): 83 94.2%, 1.75 A, CC1/2 0.665, R_FREE 0.284, 49 s; 152 65.1%, 2.94 A, CC1/2 0.843, R_FREE 0.278, 27 s - mixed, the manifest keeps it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 09:21:33 +02:00
leonarski_fandClaude Opus 5.5	43e4207aaa	still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes ... - still_battery.py run --cold: posix_fadvise(DONTNEED) on a set's input files just before its run, so every timed row is a cold read without root (the page-cache column confirms it). - Alvra sets use the depositors' pixel masks (--mask-good); the MbCO int16 glob now finds the copies, which keep the tarball's directory tree; LCLS 242 states its wavelength (the XTC does not). - EXTERNAL_TEST_DATA_STILLS.md names the mask files and the one MbCO dark run that holds no frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 08:59:36 +02:00
leonarski_fandClaude Opus 5.5	0f560ecddc	docs: the stills workflow figure, overview and methods for rc178's stills ... The stills pipeline figure keeps its layout and adds what rc178 does: the input box lists CXI and XTC, the geometry box is the beam centre decided on the frames and, with a cell, the distance and the module placement; the pre-scan reads the saturation level; the P1 merge keeps one crystal form. RUGNUX_OVERVIEW's stills paragraph says the same in prose. Methods: §1.4 describes the stills centre and distance hypotheses, a new §1.7 the saturation level read off the frames, and a new §7.7 a given cell as a starting point, the crystal-form filter and the module placement with its gauge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 00:15:42 +02:00
leonarski_fandClaude Opus 5.5	2bd3616388	Merge still-battery into rc178: the still battery and the external stills test data ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 00:10:43 +02:00
leonarski_fandClaude Opus 5.5	063a63e55d	Merge rc178-version into rc178: v1.0.0-rc.178 and its CHANGELOG ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 00:10:43 +02:00
leonarski_fandClaude Opus 5.5	26efa8c395	rugnux stills: masks in the data files themselves and by bits, a sub-unity gain left alone, HDF5 probed quietly ... - --mask-good / --mask-bad take [<file>:]<dataset>[@<bits>]: without a file the dataset is read from each data file (a per-frame mask stored beside the frames, as a Cheetah CXI holds it), and <bits> says which bits mark a pixel bad (or good) - what a CrystFEL geometry's mask/mask_good/mask_bad said, now that the geometry file gives module geometry only. - A gain the background shows at or below one detector unit per photon is not applied; such frames stay in detector units, as a CCD's do. Scaling a low-gain Rayonix's frames up by it cost one PAL set most of its indexing (69% -> 33%). - The HDF5 stills readers ask H5Fis_accessible with the error stack silenced before opening a file, so an XTC stream offered to them no longer prints HDF5's diagnostics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-21 00:00:43 +02:00
leonarski_fandClaude Opus 5.5	ebce906025	Serial XFEL stills inputs: readers from the files themselves, module geometry and its refinement ... Combines the stills reader and module work (stills-readers, stills-modgeom, stills-euxfel) onto rc178 with the owner's decisions for this release: - Inputs are files, never a directory; several files of one format are one dataset (the multi-input CLI of the previous commit), so every reader takes an explicit list: JFJochXFELReader::ReadFiles, ReadCXI, ReadPanelTable and JFJochXTCReader::ReadFiles take std::vector<std::string>; ListFiles and xtc::CollectStreams are gone, CanRead* accept a regular file only. - One HDF5 frame reader (JFJochCrystFELReader -> JFJochXFELReader) parameterised by what the file holds: per-shot groups or a 3D stack, a /metadata panel table (posx/posy/angle, pixel size, nominal camera length, pixel mask: as SACLA writes), a Cheetah CXI single panel (square frame; a tiled slab is refused with a message). At most 8 files open; every file checked against the first. Signed 32-bit decode, per-frame 3D masks, panels at their own coffset (from the EuXFEL work). - Frames stored one chunk each are read the way the NXmx reader reads its images: only the chunk's place is asked of HDF5 under its lock (H5Dget_chunk_info_by_coord), the bytes are read through HDF5ImageSource::RawFile (positional, no lock) and decoded outside it - deflate(+shuffle) by zlib, bitshuffle/LZ4 (and zstd, plain LZ4) by CompressedImage's decoders. Bitshuffle/LZ4 used to go through HDF5's filter under the global lock, one frame at a time. - LCLS-I XTC (Rayonix): wavelength from the beam record, pixel size from the Rayonix model and binning, only --detector-distance needed; damaged/compressed frames skipped, a truncated last event dropped; std::ifstream, no POSIX calls: builds with MSVC. No DIALS .expt support at all. - CrystFEL .geom: module geometry only (panel ranges, fs/ss, corners, pixel size, coffset, data path) and optionally the camera length; everything else is ignored with a note in the log. - --mask-good / --mask-bad <file>:<dataset>: a mask array in another HDF5 file, in the frames' stored layout (what a .geom's mask_file used to give). - Gain measured on the background for panel-table and CXI input that states none (--gain overrides). - Detector modules from explicit geometry only (DetectorModules: the reader's panels, an NXmx file with one NXdetector_module per module, or --module-geometry); never from the image. Spots and predictions are mapped through the modules, the image is not re-assembled. - Module refinement in the stills bundle adjustment (Ceres; per-module shift and in-plane turn, gauge on mean shift/turn/scale and on the two tilt-like patterns when the tilt is refined; adopted only where it predicts held-out frames better). Applied for a foreign multi-panel detector (panel table, .geom panels); for a JUNGFRAU measured and reported only. Written to <prefix>_modules.h5 (NXmx NXdetector_module groups), read back with --module-geometry. - A panel-table file's camera length is nominal: with a reference cell it is refined from the cell. - Broker and writer output unchanged: masters keep the single NXdetector_module group, the start message carries no module list. Tests: [CrystFELGeometry], JFJochXFELReader_*, JFJochXTCReader, [JFJochMultiReader], [CellPopulation], SaturationCeiling_*, [DetectorModules], RefineGlobalGeometry_ModulePlacement, [reindex], Sublattice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-20 23:30:48 +02:00
leonarski_fandClaude Opus 5.5	6557ba87f4	rugnux: several stills inputs in one run; inputs are files, never a directory ... Every input is a file named on the command line, read in the order given (a shell glob gives them sorted by name); a directory is refused with the glob form to use instead. - Several Jungfraujoch/NXmx masters of stills are read as one run (JFJochMultiReader): their images one after another, the pixel masks combined, the per-image efficiency placed at each input's images. They must share the detector, wavelength (0.1%), distance (0.1 mm) and beam centre (1 px), since stills geometry refinement refines one geometry for all frames; rotation sweeps are refused. The log gives the indexing rate of each input beside the run's (ProcessResult::image_indexed). - SACLA HDF5 / CXI (JFJochCrystFELReader::ReadFiles, ReadCXI, Open) and XTC (JFJochXTCReader::ReadFiles) take the list of files instead of scanning a directory; CanReadCXI and xtc::CanRead accept only a file; ListFiles and xtc::CollectStreams are gone. - Several inputs must be one format, --mode mx, and stored files (no broker, no --simulate-live); one frame of a CBF/marCCD/SMV sweep still names the whole sweep. Validated: two Jungfraujoch lysozyme stills runs together (49,920 frames, known cell) 57.1% indexed (59.3% / 54.9% per run), 1.22 A, CC1/2 0.986, R_split 8.8%, REFMAC R/R-free 0.253/0.269 against 0.257/0.273 for the earlier hand-built combined master. Rotation: 5reo and lyso_x10sa_90deg_1 unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-20 23:30:17 +02:00
leonarski_fandClaude Opus 5.5	59332d4694	tools/battery: the still battery - serial (XFEL stills) datasets against their depositions ... still_battery.py runs the public serial datasets of still_manifest.json with their deposition's cell and space group, refines the deposited model against each merge with dimple and the deposition's own free set, and reports per set: facility, beamline, detector, PDB, images, indexing rate, R_split, our R-free beside the deposition's published R-free, the wall time of the rugnux command, ms per image, the input's I/O floor (bytes over 5 GB/s) and how much of the input was already in the page cache. It shares battery.py's site config (a "still" section), verbs (run, compare, report, list, keep) and run-directory layout; run directories carry "still-" in their label. The datasets (CXIDB 243, 237, 236, 86, 90, 222, 242, 83, 152, 103, 213; dark and ground-state entries only) are listed with their verified DataCite DOIs in docs/EXTERNAL_TEST_DATA_STILLS.md, linked from EXTERNAL_TEST_DATA.md; CXIDB is credited in ACKNOWLEDGEMENT.md. The SwissFEL sets also run as int16 bitshuffle/LZ4 copies, made with still/float32_to_int16.py. The stray scripts in the repository root move under tools/battery: rugnux_stills_ab.py and its dataset template to still/, rugnux_anomalous.py beside battery.py; docs/TESTS.md follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-20 23:04:43 +02:00
leonarski_fandClaude Opus 5.5	d49210354b	CHANGELOG: 1.0.0-rc.178 ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-20 23:00:28 +02:00
leonarski_fandClaude Opus 5.5	02dbdb0b9d	v1.0.0-rc.178: version bump ... VERSION and the strings update_version.sh rewrites (API spec, python and TypeScript clients, C++ model, Redoc page, docs/conf.py, FPGA HDL and PCIe driver). The regenerated clients and Redoc page also pick up rc178's API changes made since the last regeneration (rugnux live settings and model without a bearer token, the live status fields). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-20 23:00:10 +02:00
leonarski_fandClaude Opus 5.5	6a1d859d49	Stills merge: R_split and a frame-split CC1/2, per shell and overall ... Serial crystallography quotes R_split and a CC1/2 whose half-sets keep each crystal whole (CrystFEL's partialator), and rugnux's stills report had neither, so its numbers could not be set beside CrystFEL's. MergeOnTheFly now also sums every frame's observations into the frame's own half (the existing per-image hash preference), and MergeStats reports from those two means R_split = (1/sqrt 2) sum|I1-I2| / (1/2 sum(I1+I2)) and their CC1/2. They are printed as a "Serial statistics" table under the shell table (also in the --report-resolution table) and as R_SPLIT / CC_HALF_SPLIT and their REFRES_ keys. The balanced per-reflection split behind CC_HALF is unchanged, and so is everything decided on it (resolution cut, space-group search, pass gates). Rotation output is unchanged: RotationScaleMerge does not fill the new fields and the table is not printed. On a 4000-frame jet subset over 30-1.5 A: CC1/2 0.830, R_split 31.8%, against 0.834 / 31.3% computed by hand from two separate even/odd runs and 0.842 / 30.5% from CrystFEL on the same frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 15:40:30 +02:00
leonarski_fandClaude Opus 5.5	9e09cd4b3d	broker: initialize drops the dataset's bearer tokens and everything they protected ... Initialize is the fresh start between users. It already cleared the images, plots, scan result and live rugnux status; it now also drops the tokens, resets the dataset settings (name, file prefix, unit cell, ...) to their defaults and forgets the last collection followed through /live/events, so after it nothing of the previous dataset is readable, with or without its token - the broker is as open and as empty as a freshly started one. The auth test checks that after /initialize the data-collection statistics and the /statistics aggregate are open and no longer carry the previous dataset's name. SECURITY.md also lists /rugnux/status among the protected endpoints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 12:16:32 +02:00
leonarski_fandClaude Opus 5.5	147f750c90	broker: rugnux live settings and model need no bearer token ... GET/PUT /config/rugnux and PUT/DELETE /config/rugnux/model prepare the next collection and expose nothing of the current one, so they are open like the rest of the configuration. Protecting them meant the next user needed the previous dataset's token to set up their own run. /rugnux/status, which reports the current collection, stays protected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 07:43:47 +02:00
leonarski_fandClaude Opus 5.5	1949155c12	Rotation: a blank sweep end is trimmed before scaling, and a delta-CC1/2 stretch takes the sliver it leaves at an end (RESULT-CHANGING) ... Shutters that open late or close early, and beams that go off, leave frames at the ends of many sweeps with no exposure at all. Until now those frames went into the scaling like any other: on a CCD their noise, divided by an incident flux of ~0.005, made the partials loop walk their scales toward zero until the 50x collapse guard dropped them - or, if the loop stopped first, merged one as noise, which delta-CC1/2 then removed with its whole 10 deg batch and reported as SWEEP_GAPS (3p85, frames 338-359). On photon-counting detectors the blank frames fitted G = 0, escaped the guard and were reported as merged although nothing of them was (cytc_x10sa, myob_x10sa, thau_micromax_pink, lysoI_micromax_pink, 3p85). TrimBlankSweepEnds (RotationScaleMerge.h) walks in from each end and clears the reflections of every frame that - recorded no diffraction: as many of its reflections stand more than 3 sigma below zero as above it (a sign count, so no sigma calibration is needed and a slight background bias cannot hide the strong reflections), and that excess is 3 sigma clear of the run's typical frame - so a crystal too weak for one frame to show it is never trimmed; and - had no beam: its mean background is nearer to none than to the run's typical frame. That is what keeps a crystal leaving the beam, which also records nothing but still scatters, in the sweep ledger as a finding (insu_I_x06da_ref, 9zm0, 9fhc, 5reo are untouched). No stretch length: one blank frame is as blank as twenty. Partly exposed first/last frames recorded diffraction and stay (7tcd, 7q6j, 9z72, lyso_micromax_pink). It runs on the integration outcome before the overload count, so the merge, post-refinement and unmerged output all see the trimmed sweep, and in --mode scale on the stored reflections. The report gives FRAMES_BLANK_START/END and a prose line - information, no warning; the frames count as rejected. The sweep ledger looks for degraded stretches, and reads its running medians, only between the first and last frame with data. Delta-CC1/2: a convicted stretch that stops short of a sweep end by less than one rocking event takes the rest, where those frames read as the stretch does (5reo: 331-396 rejected, 397-399 were left in as a three-frame weak_diffraction range with its own warning). Battery --only, 31 sets, tol5 f217f180a -> this commit: no space group, d_min or verdict changes; merged statistics identical within 0.001 except 5reo (R_meas 0.236 -> 0.231, ISa 23.0 -> 23.3, R-free 0.2013 -> 0.2004, warnings 3 -> 2). Trimmed: 3p85 0/19 (+3 empty frames; collapse-guard drops 13 -> 0), 6w4h 0/5 (guard 5 -> 0), 7rji 0/4 (delta-CC1/2 no longer removes 3406-3412, two of which diffracted: 7 -> 0), cytc_x10sa 0/2, thau_x10sa_* 0/1-2, myob_x10sa 0/1, lyso_x10sa_strong 0/1, 8a1a 0/1, lyso_micromax_mono 1/0, lysoI_micromax_pink 2/0, thau_micromax_pink 1/0. Controls (11if, 5f6m, 6moj, 8u0i, 9fhc, 9z44, 9z72, 9zm0, 6iu6, insu_I_x06da_ref, myob_x06da_split, lyso_x06da_half_image, thau_bl1a_4p6keV) trim nothing and are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 02:18:19 +02:00
leonarski_fandClaude Opus 5.5	4620130f71	RotationScaleMerge: scaling loops run to an rms step of 1e-5, accelerated from 1e-2 (RESULT-CHANGING) ... The per-frame scaling tolerance goes from 1e-3 to 1e-5, and the Anderson onset (also the round the fulls loop keeps its cross-validated lambda) becomes an absolute 1e-2 instead of 10x the tolerance. Evaluated on 35 merged sets (ci tier + scaling-audit sets), on top of the loop fixes: - plain 1e-3: 8913 iterations, 38 s of loop time on GPU, 7 loops at the 100-round cap; - plain 1e-5: 35212 iterations, 120 s, 17 loops at the cap; - Anderson 1e-3 (previous default): 3715 iterations, 16 s, merged intensities as far from the converged ones as plain 1e-3 (median 0.13%, max 4.3%); - Anderson 1e-5 (this commit): 6507 iterations, 27 s, longest loop 89 rounds, merged intensities within 0.2% of plain 1e-5 and identical SG, d_min, CC1/2, ISa, R_meas, R_free. No space group or verdict changes in any mode. The converged merge is not measurably better against the references, but it no longer depends on where a loop happened to stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-20 02:18:19 +02:00
leonarski_fandClaude Opus 5.5	0894230671	Anderson: accelerate a scaling loop only near its fixed point, and never again once its step grows (RESULT-CHANGING) ... The Anderson commit before this one changed three rc176 battery results, which is why it was reverted there. One of them is the acceleration's own doing: on a split myoglobin sweep (P2_1, ISa 2.5) the partials loops never settle - one walks at a step of ~2e-2 for as long as it runs - and the fulls loops walk from their first round (their cross-validated lambda moves every round, so the map itself is not fixed). Restarting the history whenever the step grew only rebuilt a one- or two-round history and extrapolated along the walk: the stalled loops ended somewhere else, the pre-pass fulls smoothing window went from ~7 frames to ~1, and CC1/2 fell 0.701 -> 0.595. Two rules now confine the acceleration to the regime it is for - a map contracting to its fixed point: - a loop is accelerated only once its plain step is below ten times the tolerance (1e-2); - the first round whose weighted residual grew turns the acceleration off for the rest of the loop (AndersonMixing::Stopped), instead of restarting it. A loop that walks therefore stays the plain loop it was in rc176. The growth rule alone gave CC1/2 0.625; with the onset, 0.701, as rc176. Measured against rc176 bc191bb80 (GPU, workstation): - myob_x06da_split: P2_1, CC1/2 0.701, d 1.81 A - as rc176; 481 -> 257 scaling iterations. - 7qij 434 -> 126 iterations, 9v16 273 -> 101, same space group and statistics. - ci tier (29 sets, no null model): 28 unchanged beyond noise; sum of times 580 -> 585 s (9gdj +9 s, below). The two other changes are NOT Anderson's and stay: - insu_I_x06da_weak I23 -> I222: the all-observations search refuses I23 on a merge chi2 ratio 2.08 against a 1.85 bound; plain scaling converged to 1e-5 gives 2.07-2.09 and refuses it too. Only the plain loop stopped at 1e-3 passed - the bound was set on unconverged scaling. - 9gdj +9 s, same answer: the early beam-centre comparison's "loses beyond noise" margin (-0.002 against 2 sigma 0.0016) moves by ~0.07 between scaling paths, so the converged loop no longer takes the shortcut and both centres are judged on all frames; the file's centre is kept either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-19 18:49:24 +02:00
leonarski_fandClaude Opus 5.5	138abafa14	RotationScaleMerge: Anderson-accelerate the per-frame scaling loops (RESULT-CHANGING) ... The partials and fulls scaling loops are alternating least squares that converge linearly: over the rc176 battery's 2527 loops a round shrinks the step by a median 0.86 (partials) and 0.93 (fulls), up to 0.98, so the 1e-3 step tolerance stopped them 0.6% / 1.4% (median) and up to 17% (fulls p90) short of their fixed point, and 313 of them were stopped as "stalled" when they were only slow - a stalled fulls loop also skipped ShrinkToRestrained. Each round is now accelerated in log G by type-II Anderson mixing (Walker & Ni 2011), depth 5, each frame weighted by its observation count as in the step test; frames below the credible floor take the plain round's scale. The history is dropped when the weighted residual grows or the set of fitted frames changes (new AndersonMixing class, unit-tested on a synthetic slow ALS loop). The mixed scales are gauge-pinned again before the rescale, and the stop test is unchanged: it reads the round's residual F(x) - x, so "stalled" now means a loop that does not settle under acceleration either. Measured (CPU, -N 8): 4nwv 311 -> 82 iterations over 9 loops, 9v16 76 iterations over 11 loops, the same counts as the env-gated prototype on tail-solver; no history restart fired on either. Depth 3/5/8 give 92/91/90 iterations on 4nwv. A restart on a change of the fulls loop's cross-validated lambda was tried and gave nothing (83 vs 82). Same space group and merging statistics on 4nwv and 9v16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-19 18:49:24 +02:00
leonarski_f	7ed32b899f	v1.0.0-rc.177 (#87) ... * jfjoch_broker: Faster live Rugnux analysis, with the GPU indexers prepared at the start of a collection and kept between collections; Rugnux no longer writes `.ccp4` maps (the map coefficients are in `<prefix>_maps.mtz`). Reviewed-on: #87 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 16:07:31 +02:00
leonarski_f	8261b7354e	v1.0.0-rc.176 (#86) ... * jfjoch_broker: Optional live Rugnux analysis of DECTRIS rotation collections inside the broker, with merged data, maps and the placed model written beside the master file, the rotation lattice in the scan result, and a "Rugnux live" tab and dashboard row in the web frontend. * Rugnux: Faster processing, with molecular replacement and model validation spread over several GPUs and run on the Apple GPU on macOS. * Rugnux: The polarization stated in miniCBF and d*TREK headers is used, twinned crystals' space groups are decided more reliably, and a run that finds no lattice still reports every image. Reviewed-on: #86 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 07:03:28 +02:00
leonarski_fandFilip Leonarski	19b135186d	v1.0.0-rc.175 (#85) ... * Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`). * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS). * Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead. * Rugnux: On macOS, Bragg integration runs on the Apple GPU. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. * Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`. * Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. * Rugnux: The beam-stop shadow is found correctly on tilted detectors. * Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement. * jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps. * jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images. * Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. --------- Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home> Reviewed-on: #85 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-16 15:34:43 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_f	61c603b274	v1.0.0-rc.163 (#73) ... * Packaging: each package installs its license notices under a directory of its own - `share/doc/jfjoch_broker`, `share/doc/jfjoch_writer`, `share/doc/jfjoch_viewer`, `share/doc/jfjoch_driver_dkms` - instead of all of them into the shared `share/doc/jfjoch`, so `jfjoch` and `jfjoch-writer` can be upgraded one at a time instead of only together. Reviewed-on: #73 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 13:38:29 +02:00