10 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	8a123e794d	Stills: the run's rocking curve goes to _process.h5 and --mode scale merges with it; MeasureRunWidth takes the window at each reflection's resolution ... --mode scale re-measured the rocking width on the stored reflections, so on any file whose frames were predicted through the run's curve it merged with a different curve than the mx run did. The curve the post-refinement used (formula, table, radial scale) is now carried in the EndMessage (StillsRunWidth) and written to the master: /entry/MX/stillsRockingWidthFormula [gamma0 1/A, gamma_e rad, bandwidth sigma], /entry/MX/stillsRadialScale, /entry/MX/stillsRockingWidthTableDstar and /entry/MX/stillsRockingWidthTable (1/A). The reader returns it, and --mode scale passes it to the post-refinement. A file without it is re-measured as before. MeasureRunWidth grouped each frame's reflections by the frame's window, taken as the largest excitation error it predicted. With a window that follows sigma(d*) that is the high-resolution window, applied to the frame's low-resolution reflections too. Given the curve the frames were predicted through, the window is now that curve at the reflection's own resolution; without one, the frame's largest excitation error as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 992343d0206489991880bb927efc79131134a1d9)	2026-10-26 14:09:32 +01:00
leonarski_f	4dbcb35708	Stills partiality: the rocking width measured in shells of equal width in d*, used as a table ... MeasureRunWidth measured the width in eight shells of equal count in d*^2 and fitted the three-term formula gamma0^2 + (gamma_e d*)^2 + (bw |q_z|)^2 to them. On runs predicting ~1000 reflections a frame the equal-count shells all sat at 3.8-1.8 A, so the low-resolution width was the formula's extrapolation, and the fit's form flipped with the number of frames (to a pure bandwidth at 50k frames). The same window-group moment is now also measured in eight shells of equal width in d*, made non-decreasing (pool adjacent violators, weighted by each shell's summed signal) and kept in RunWidth as a table that the partiality interpolates linearly in d*. The formula is still fitted on the equal-count shells and still sets the prediction window. With fewer than three measurable shells the formula is used as before. Still battery, 12 non-float sets, together with the sigma-following prediction window (the previous commit; run 20261026-1021_1b28d6_still-part-comb against rc180 20261026-0857_3882b0_still-part-base1, 0-cycle REFMAC R-free vs deposited, both --no-refine): 6 better, 3 unchanged, 3 worse (by 0.002), sum of changes -0.027 (the window alone: -0.007); largest gains pal90 -0.014, alvra103_int16 -0.011, lcls222 -0.005; largest losses euxfel152 +0.005, alvra213_int16 +0.005 (+0.010 with the window alone), euxfel83 +0.003. Win/draw/loss against the depositions unchanged at 9/2/1. Rotation: p.hkl byte-identical on 5reo, lyso_x06da_ref and thau_x10sa_16keV. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit de779bb10b1813eeae625e3994b1ca73d6079d7a)	2026-10-26 10:50:52 +01:00
leonarski_f	0829b9a72d	Stills prediction: every frame predicts through two rocking widths of the run, measured first and centred on its radial scale ... Each still predicted through |dist_ewald| <= 2x its own profile radius (the RMS excitation error of its indexed spots): one width at every resolution, differing eightfold between frames. Against the run's measured rocking width that was 5-8 sigma at low resolution on one SACLA run (59 % of the 40-8 A predictions empty) and below one sigma at high resolution on a crowded LCLS run (half of the curve never integrated). Now the stills image pass first integrates a sample (a tenth of the run, at least 2000 frames) the old way, scales it against its own merge and measures the run's rocking curve on it - centre (MeasureRadialScale) and width (MeasureRunWidth), as the merge measures them - then predicts every frame through 2 sigma(d) of that curve, centred on the reciprocal lattice scaled by the radial scale (BraggPredictionSettings::window_*, CPU and CUDA). The post-refinement uses the same curve instead of measuring it again: re-measured through windows that follow it, the constant term falls to ~0 and the result swung by up to +-0.015 per run. Still battery, 12 non-float sets, against the st-partial baseline (0-cycle REFMAC R-free vs deposited): 9 better, 3 worse, sum of changes -0.013; the off-centre run -0.012, worst +0.008 on one JUNGFRAU run (its sample measures a different width form than the full run). Integrated reflections 0.8-1.7x, battery wall +13 % on a loaded box. Rotation untouched (window fields zero unless set); p.hkl byte-identical on three rotation sets. Tests: BraggPrediction_WindowFollowsResolution, BraggPrediction_CPU_GPU_consistency_window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 1b28d671d780bc88525ceac5120017963d937bbc)	2026-10-26 10:50:52 +01:00
leonarski_f	2dbb9d5951	Stills partiality: merge only partials of p >= 0.5, adopt the refined model on every crystal, centre the rocking curve on the run's radial scale ... Three changes to the stills post-refinement and merge, decided on 0-cycle REFMAC R-free against the deposited models (diff = ours - deposited data, noise ~0.002): - Stills merge only observations whose model partiality is at least 0.5 (RugnuxDefaultScalingSettings; rotation keeps 0.02, and --min-partiality still overrides). Diagnosis on per-observation dumps: the empirical rocking curve has a floor of 5-15 % of its peak out on the tail (mis-oriented crystals), and dividing such observations by a small model p makes them biased, not noisy. Variance terms in (1-p) or (1-p)/p recovered a third of what the cut does; a narrower (correct-looking angular) width or a Lorentzian form was worse. On one set in scale mode the optimum is flat over 0.3-0.6. - The CC adoption gate in RefineOne is gone: it left 25-33 % of crystals at p = 1 next to the refined ones, and collapsed to 3 % on a run whose curve is off-centre. - The run's radial scale of the reciprocal lattice (a wavelength or cell a little off, invisible to spot positions since they fix only lambda d*) is measured from where the curve is centred against d*^2 (MeasureRadialScale, iterated three times) and applied to q in the width measurement and the post-refinement. One run had its curve centred at +2-3e-3 d*^2 (scale -1.6e-3 measured); the others measure below 7e-4. --mode scale on stills now starts where the full pipeline starts: p = 1, the excitation error recomputed from the stored lattice (the window was read as zero before, so no run width was ever measured), and collapsed scales rejected after the refinement. Still battery, the 12 non-float sets, change of diff vs the previous build: -0.0035 mean, 9 better, three within +-0.0006 except one +0.0053 (an edge win becoming a draw); the off-centre run -0.005 here and -0.017 in an earlier run of the same algorithm (that set varies by ~0.01 between runs). Rotation p.hkl byte-identical on three sets. Test StillsPartialityRefine_MeasureRadialScale added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0d85e5b975ba863828bd17ebc3b5308e9aa294ce)	2026-10-25 07:20:21 +01:00
leonarski_f	789e064e6a	Stills scaling: a model error in the per-crystal scale fit, and the rocking width measured per window group ... Two changes to the stills scale model; rotation is untouched (p.hkl md5 identical on two rotation sets). The per-crystal scale fits (ScaleOnTheFly and the partiality post-refinement's G and tilt) measured residuals in counting sigmas. The strong reflections of a still scatter about the model by tens of percent (robust sd of ln(I/<I>) 0.45-0.8 by shell on one run), so with a Cauchy loss at 3 counting sigma every strong reflection was an outlier and the scale was set by the weak ones: re-fitting each frame's scale against the final merge moved it by a factor 0.45-1.5 (5-95%). The fit's variance is now the expected one at the model, background + counting + (0.3 * model)^2. The coefficient is not delicate: 0.1-1.0 fit alike in an offline replay; what matters is that the term is there. The run's rocking width came from the intensity-weighted moment of the excitation error over all frames pooled. Frames predict through windows of their own that differ by an order of magnitude, and the widest windows dominated both moments: on a run whose windows are mostly narrower than the curve at high resolution the measurement read the window at every resolution, 2.8e-4 1/A with no angular term, several times too wide at low resolution (low-partiality observations there merged at 0.1-0.3 of their reflection's mean). The moment is now taken per group of frames of similar window (4 per shell), each against its own window, and the shell takes the median of the groups whose curve is narrower than their window. On that run the width becomes 6.7e-5 1/A + 0.018 deg angular; where the pooled measurement was already right (a SASE-bandwidth run), it stays where it was. 0-cycle REFMAC R-free against the deposited models, 9 still battery sets (6 used while developing, 3 held out), change from rc179: -0.0100, -0.0095, -0.0057, -0.0055, -0.0051, -0.0045, -0.0030, -0.0024, +0.0009 (held out: -0.0055, -0.0051, +0.0009). The width change alone accounts for most of the largest gain; the scale-fit variance for 0.000-0.003 per set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ccc0e01f56827bbe0648ed7c5b8049912d80650f)	2026-10-24 16:57:12 +02:00
leonarski_f	b00c668e3a	v1.0.0-rc.179 (#90) ... * Rugnux: Stills processing needs only the unit cell and space group beyond what the file records - detector distance, beam centre, tilt, panel placement, saturation level, gain and polarisation plane are found from the images - and is up to twice as fast. * Rugnux: Stills can be indexed without a GPU, with a CPU implementation of the fast-feedback indexer (macOS, computers without a GPU). * Rugnux: Rotation and stills data are integrated in the detector's own units with one gain model, improving data from CCD detectors. * Web frontend: The Rugnux live page shows the placed model with its electron density and anomalous difference map in a Mol* viewer. Reviewed-on: #90 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-24 14:52:29 +02:00
leonarski_fandFilip Leonarski	19b135186d	v1.0.0-rc.175 (#85) ... * Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`). * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS). * Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead. * Rugnux: On macOS, Bragg integration runs on the Apple GPU. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. * Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`. * Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. * Rugnux: The beam-stop shadow is found correctly on tilted detectors. * Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement. * jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps. * jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images. * Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. --------- Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home> Reviewed-on: #85 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-16 15:34:43 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00