v1.0.0-rc.183 #94

Merged
leonarski_f merged 47 commits from rc183 into main 2026-11-06 03:34:08 +01:00
47 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 5.5 61744c1133 receiver: live stills report last image -> all written, and Done once written
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 29m59s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 32m32s
Build Packages / Unit tests (push) Successful in 1h44m30s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 30m40s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 32m3s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m50s
Build Packages / Generate python client (push) Successful in 32s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 27m28s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 18m2s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 29m52s
Build Packages / Validation Battery (push) Successful in 44m58s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 30m8s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 34m21s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 26m29s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 30m0s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m28s
Build Packages / Build documentation (push) Successful in 46s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 10m37s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m33s
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 25m58s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m5s
The serial (FPGA + GPU) live path now times the latency from the last image
to the end message out and the writer finished, as the rotation receiver
does: StillsGPUAnalysis::Written, called by FinalizeMeasurement after the
writer is finalised, sets last_image_to_written_s and only then the run's
outcome. Until then the status shows a "Writing" step, so the run no longer
reads Done (and its elapsed time no longer stops) before the end message and
the writer are through. The timeline's latency bracket and the headline
latency now show for serial collections.

The END message is serialised into a scratch buffer first, as on rotation:
merged data that would not fit the END buffer are dropped with a warning
instead of failing the collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 20:52:44 +01:00
leonarski_fandClaude Opus 5.5 b33291ae58 docs: shorter rc183 CHANGELOG
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 20:42:43 +01:00
leonarski_fandClaude Opus 5.5 7248831fce docs: CHANGELOG for the XFEL IDs in _process.h5
Build Packages / build:jfjoch:rocky8:nocuda (push) Canceled after 1m55s
Build Packages / build:jfjoch:rocky9:cuda (push) Canceled after 0s
Build Packages / build:portable:linux-x86_64:nocuda (push) Canceled after 2m1s
Build Packages / Validation Battery (push) Canceled after 0s
Build Packages / build:jfjoch:rocky8:cuda (push) Canceled after 0s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Canceled after 0s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Canceled after 1m16s
Build Packages / build:portable:windows-x86_64:nocuda (push) Canceled after 1m26s
Build Packages / build:portable:macos-arm64:nocuda (push) Canceled after 1m17s
Build Packages / build:portable:windows-x86_64:cuda (push) Canceled after 1m26s
Build Packages / Create release (push) Successful in 8s
Build Packages / Unit tests (push) Canceled after 2m17s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Canceled after 0s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Canceled after 0s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Canceled after 2m15s
Build Packages / Build documentation (push) Successful in 47s
Build Packages / build:portable:linux-x86_64:cuda (push) Canceled after 1m59s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Canceled after 1m57s
Build Packages / build:jfjoch:rocky9:nocuda (push) Canceled after 1m53s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Canceled after 0s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Canceled after 0s
Build Packages / Generate python client (push) Canceled after 0s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 20:38:04 +01:00
leonarski_fandClaude Opus 5.5 ab649aad87 rugnux: _process.h5 keeps each image's XFEL pulse ID and event code
The per-image XFEL identifiers Jungfraujoch records for a pulsed source
(/entry/xfel/pulseID, uint64, and /entry/xfel/eventCode, uint32) were not
carried through reprocessing: the reader did not read them and rugnux built
its per-image messages without them, so a _process.h5 lost the pulse ID of
every image. Merging a pump-probe run by pulse ID needs them next to the
integrated reflections.

- The HDF5 reader reads /entry/xfel/pulseID and eventCode from the master
  (VDS, integrated, _process.h5) or, for a legacy master, from its data
  files, and passes them on per image. Several inputs read as one keep them
  where every input has them.
- rugnux copies them into the messages of the image pass that writes the
  _process.h5, so /entry/xfel/{pulseID,eventCode}[i] there belongs to image
  i of the file - the index of /entry/MX/...[i], /entry/reflections/image_<i>
  and the source image /entry/detector/number[i] - including under -s/-e and
  a stride.
- A pulsed-source start message now asks for the /entry/xfel datasets in the
  VDS master: the flag was never set, so a broker master held them only in
  its data files.

Test: Rugnux_ProcessFileKeepsXfelIds (a pulsed VDS input, images 1, 4, 7 of
ten processed; IDs checked in order in the _process.h5 and on read-back).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 20:31:32 +01:00
leonarski_fandClaude Opus 5.5 4b47037b89 tests: spot-tile tests register the HDF5 filters; stills fixture sized past the pre-pass
Build Packages / Create release (push) Successful in 9s
Build Packages / build:portable:linux-x86_64:cuda (push) Canceled after 5m18s
Build Packages / build:portable:linux-x86_64:nocuda (push) Canceled after 5m20s
Build Packages / Build documentation (push) Successful in 46s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Canceled after 0s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Canceled after 0s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Canceled after 4m36s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Canceled after 5m34s
Build Packages / build:portable:macos-arm64:nocuda (push) Canceled after 4m37s
Build Packages / build:portable:windows-x86_64:cuda (push) Canceled after 4m59s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Canceled after 5m16s
Build Packages / build:jfjoch:rocky8:nocuda (push) Canceled after 5m14s
Build Packages / Generate python client (push) Canceled after 0s
Build Packages / build:jfjoch:rocky8:cuda (push) Canceled after 0s
Build Packages / build:jfjoch:rocky9:cuda (push) Canceled after 0s
Build Packages / build:portable:windows-x86_64:nocuda (push) Canceled after 4m58s
Build Packages / Unit tests (push) Canceled after 5m36s
Build Packages / Validation Battery (push) Canceled after 0s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Canceled after 0s
Build Packages / build:jfjoch:rocky9:nocuda (push) Canceled after 5m12s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Canceled after 0s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Canceled after 0s
SpotTiles_MatchWholeFrame(_GPU) read the bitshuffle-compressed test file without registering the
HDF5 filters, so they passed only when an earlier test had registered them or a system plugin was
installed; CI's random test order ran one first and failed. ReadFrames() now registers them.

The StillsGPUAnalysis fixture declared 200 images while its frames are numbered from
PREPASS_IMAGES on (500 since the pre-pass was enlarged), so the merge's per-image arrays were
indexed past their end and the failing-frame test passed for the wrong reason. The fixture now
declares PREPASS_IMAGES + 200 images.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 19:37:54 +01:00
leonarski_fandClaude Opus 5.5 61a917b52b docs: CHANGELOG for the rc183 map coefficients and constrained cell
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 30m13s
Build Packages / Generate python client (push) Successful in 26s
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m11s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 28m12s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 33m14s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 27m0s
Build Packages / Unit tests (push) Failing after 1h40m5s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 30m40s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m37s
Build Packages / Build documentation (push) Successful in 45s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 26m0s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 31m13s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 32m26s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 10m49s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m38s
Build Packages / Validation Battery (push) Successful in 45m20s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m28s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 26m28s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 32m19s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 18m0s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 29m47s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 16:50:45 +01:00
leonarski_fandClaude Opus 5.5 62e77cdc48 rugnux: the written cell obeys the written space group
A group adopted after integration sits on a cell refined in a lower Bravais
class (most often the triclinic primitive cell of a centred lattice). The
existing projection onto the group's metric fired only above a metric
violation of 1e-3, so a free fit's usual drift - a C 1 2/c 1 cell with
alpha 90.03 and gamma 89.97, a tetragonal or cubic cell with a != b - went
unchanged into the report, the mmCIF, the SHELX .ins, the MTZ and the
_process.h5, while the e.s.d.s beside it treated those angles as fixed.
In the latest full battery 20 of 258 rows reported such a cell, in every
crystal system and every input format.

The cell (and the rotation lattice recorded beside it) is now projected
onto the metric of the written group once, after the group and its setting
are final and before anything is written from it, wherever the violation
lies between float noise (1e-5) and MAX_METRIC_VIOLATION. That is an
info line; the re-refinement and the warning above 1e-3 are unchanged.
--mode scale does the same on the stored cell under the file's or -S's
group. The integrated reflections are not touched.

Measured on the 20 affected open-arm rows plus the 3 private and 1
multisweep rows with the defect (--no-refine, against an rc183 base on
the same sets): every cell now satisfies its group (violation <= 1e-16,
from up to 1.0e-3); nothing else changed beyond noise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 16:50:38 +01:00
leonarski_fandClaude Opus 5.5 08594fe6c9 rugnux: fill 2mFo-DFc at the unmeasured reflections with D Fc, keep an unfilled copy
FWT/PHWT in <prefix>_maps.mtz and <prefix>_refined_maps.mtz now carries, beside
the observed reflections, every reflection of the data's resolution range that the
data do not have, at D Fc - its expected value given the model - with D that of
the observed reflection nearest in resolution. This is the convention of
phenix.refine (fill_missing_f_obs) and BUSTER: a missing reflection left at zero
biases the map towards it, which at atomic resolution shows as broader, lower
atoms where completeness is short. The filled rows have FP, FOM, FREE and the
difference coefficients missing. FWT_no_fill/PHWT_no_fill is the same map on the
observed reflections only (phenix's 2FOFCWT_no_fill), and DELFWT/PHDELWT is never
filled. For the refined model, F_model at the filled reflections is computed on
the CPU with the bulk solvent and scale of the last fit.

The map-coefficient tests check the filled FWT and the missing FWT_no_fill/DELFWT
at unmeasured reflections. The R-factors, the blobs and the maps the refinement reads are unchanged. On an
86%-complete 0.99 A set the median FWHM of well-ordered carbons drops from 1.061
to 1.034 A and their peak rises by 4%; on a 99%-complete set nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 16:48:36 +01:00
leonarski_fandClaude Opus 5.5 ab6b6740f8 rugnux: map coefficients with the overall anisotropy of the scale divided out
The 2mFo-DFc and mFo-DFc coefficients written to <prefix>_maps.mtz and
<prefix>_refined_maps.mtz were on the observed scale, which keeps the anisotropic
part of the overall scale (k_total). On anisotropic data every atom of the map was
stretched along the weak direction. Each coefficient is now divided by
exp(-h^T B*_aniso h / 4), B*_aniso the overall anisotropic B of the fit less its
isotropic part, as phenix.refine does with its maps. FP and FC stay on the
observed scale, and the maps the refinement and the validation read themselves
(blob finder, solvent audit, mean density at the atoms) are unchanged, so the
R-factors, the blobs and every REFINE_ key are the same.

On a 0.97 A set with dB ~10 A^2 the median half-maximum axis ratio of well-ordered
carbons rises from 0.67 to 0.75; on isotropic sets the maps hardly change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 16:30:24 +01:00
leonarski_fandClaude Opus 5.5 0ee71f1971 docs: bring the stills geometry and live merge pages in step with rc183
- Stills geometry refinement is on by default for all stills and, without a
  reference cell, anchors on the cell the sampled frames index to; the
  distance is then a hypothesis too (CPU_DATA_ANALYSIS_IMAGE 1.4, overview,
  the -C row of the model/reference table).
- The live serial merge is made in the consensus cell of the frames indexed
  by the first merge, not the dataset settings' cell (JFJOCH_BROKER).
- The CCD inactive-area mask, and the beam-stop finder reading those pixels
  as data, described in CPU_DATA_ANALYSIS_IMAGE 1.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:59:49 +01:00
leonarski_fandClaude Opus 5.5 e87a17b075 docs: CHANGELOG for 1.0.0-rc.183
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:59:49 +01:00
leonarski_fandClaude Opus 5.5 c4df332618 Solvent build: write each new water beside the model, not a symmetry mate
A difference peak is taken once per symmetry-equivalent set (the map's
asymmetric unit) and was to be moved to its copy nearest the model's own
atoms (NearestCopy) before a water is built there. That step took the
nearest atom's NeighborSearch mark, whose position is already the
symmetry image, and applied the image's operator to it a second time:
outside the identity image the water was written beside a symmetry mate
or a unit cell away (by a lattice translation for a two-fold, beside
another mate for a three-, four- or six-fold).

The copy is now taken against the atom as the model has it: the
symmetry operator and lattice translation that put the peak nearest to
that atom (UnitCell::find_nearest_image / find_nearest_pbc_position). The
water's environment, and so the refinement, is the same; only the
coordinates written move. New test: a peak beside a four-fold mate in
P4 is written 2.8 A from the model's water (the old step wrote it a
lattice vector away).

5reo (open arm, C2), --refine-solvent, share of the added waters whose
nearest non-water, non-hydrogen model atom over all copies is in the
written copy: from the model without waters 67/155 (43%) before,
154/155 after; from the deposited model 27/43 before, 42/43 after
(the one left a near tie moved by refinement). R-work/R-free
0.2046/0.2335 -> 0.2046/0.2334 and 0.1979/0.2172 unchanged; 156 and 43
waters added either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:55:15 +01:00
leonarski_fandClaude Opus 5.5 caf2461bfa Model validation: no model or maps files for a model placed nowhere
A --model the rigid body rejected (MODEL_FIT= REJECTED) and molecular
replacement did not place (MODEL_PLACEMENT= NOT_PLACED) still had its
rejected rigid-body placement written as <prefix>_model.cif/.pdb with
its <prefix>_maps.mtz, by design ("a rejected model gets its maps too").
Nothing is refined for such a model, so a run given the wrong protein
ended with a model and maps beside the reflections and no refinement,
which reads as a placement that only failed to refine.

Such a run now writes no model and no maps (nor the _mr maps); its
R-factors stay in section 5 as the negative result, and the report says
that the model is placed nowhere and nothing is written for it. A model
that is placed, by the rigid body or by molecular replacement, is written
as before, refined or not.

Molecular replacement's acceptance is unchanged. Lysozyme (193L) given as
--model to 14 non-lysozyme open-arm sets and 7 in-house sets (thaumatin,
insulin, cytochrome C, myoglobin), and as --mr to the open-arm ones, was
placed by MR nowhere; it was written as _model/_maps on the two sets
whose cell let the rigid body be tried (9b22, an insulin set), which
this change removes. The closest call is tNCS: on five tNCS sets a
second copy put beside a wrong first one at the tNCS vector reaches a
translation-function Z of 21-55 (the candidate's TF_Z is the highest of
its copies'), so such a candidate is PLACED by the TF-Z gate; the null
then rejects it (7n0i: NULL_SIGMA +0.7 against 4.51) where the tNCS
half-vector test has not already left it UNDECIDED. Requiring the first
copy to stand out instead would lose true tNCS placements whose first
copy reads no better than at random (9rci by its own chain: LLG Z -1.0
for the first copy, TF Z 10.5 for the second, R-free 0.28).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:55:15 +01:00
leonarski_fandClaude Opus 5.5 94b205c375 Refinement report: the strongest blobs carry the anomalous map's height
The five strongest unexplained blobs listed in the report gave the
blob's reading, peak, volume, position and nearest atom, but not the
anomalous difference map at its peak, although every blob had it
(DensityBlob::anomalous_sigma, in _blobs.json) - the one number that
tells an anomalously scattering ion from a water or a light ion.

Where the refinement made the anomalous map (the merge kept the Bijvoet
pairs), each listed blob now ends with ", anomalous <x> sigma".

9t6s (open arm) with its Cd2+ ions deleted from the model:
  ION  peak 41.8 sigma, 23.5 A^3, at 0.44 3.99 -13.37, 2.7 A from SG CYS D5, anomalous 8.8 sigma
  ...
  MODEL_ERROR peak 5.1 sigma, 6.3 A^3, ..., 1.6 A from O HOH D341, anomalous 0.1 sigma

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:55:15 +01:00
leonarski_fandClaude Opus 5.5 0147d51605 Solvent build: waters it adds go into a chain of their own
A water the build added went into the chain of the model atom nearest
to it, numbered on after that chain's last residue: the refined model
had new HOH residues in the protein chains (HOH A405, ...), and where the
model brought no waters of its own they took the placeholders' subchain
(label_asym_id "~xw") in the mmCIF.

They now go into one water chain, as PDB and mmCIF entries keep waters:
chain W (or, where the model's chain W holds more than waters, the first
of X, Y, Z, V, U, ... that holds only waters), HOH residues numbered from
1 on (after that chain's last where the model has one). The model's own
waters stay where they were. In the view the output is written from, the
placeholders put in take no subchain but that of the waters already in
the chain, and a chain's waters follow its other residues (a site
relabelled from an added water), so that setup_entities gives the chain
one water subchain and entity of its own (label_asym_id Wxw, entity
water). ApplySolventEdits makes the chain where the model has none.

5reo (open arm), --refine-solvent: from the model without waters, 156
waters added - before in chain A (HOH A405...; mmCIF label_asym_id ~xw),
now in chain W (HOH W1..; PDB "HOH W   1", mmCIF auth_asym_id W,
label_asym_id Wxw, entity water; a water relabelled Cl in its own
subchain Wx1), R-work/R-free 0.2046/0.2335 unchanged; from the deposited
model, its waters stay in A and the 43 added are W1-W43.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:55:15 +01:00
leonarski_fandClaude Opus 5.5 cfc54709e5 Refinement report: REFINE_SCORE_* describe the refined model
REFINE_SCORE_R_WORK, REFINE_SCORE_R_FREE and every other REFINE_SCORE_
term were taken from RefinementResult::score_start, the score of the
model as given, and the geometry terms from geometry_start. After a
refinement the report therefore carried the starting R-work under a name
that reads as the result, beside a different, lower REFINE_R_WORK and
the CLI's "refined: R-work ..." line.

The refinement now scores the model it ends with when it makes its maps
(RefinementResult::score), and the REFINE_SCORE_ terms are that score,
with the final geometry where the restrained stage ran:
REFINE_SCORE_R_WORK/R_FREE equal REFINE_R_WORK/R_FREE and the CLI's
numbers. Where the model is only scored (--refine-score-only) it is the
model as given, so refine_parity.py reads the same values as before.
The starting values stay under their own names (REFINE_R_WORK_START,
REFINE_R_FREE_START, REFINE_RMSD_*_START). The neighbouring keys were
checked: the bulk solvent, the geometry and the R-factors other than
these were already the final model's.

9t6s (open arm): REFINE_SCORE_R_WORK/R_FREE 0.2790/0.2904 (= START)
before, 0.2798/0.2994 (= REFINE_R_WORK/R_FREE) after. 1exr (refine arm,
--mode refine): 0.1253/0.1320 before, 0.1271/0.1377 after; with
--refine-score-only every REFINE_SCORE_ line is identical before and
after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:55:15 +01:00
leonarski_fandClaude Opus 5.5 5c7cd2510f Refinement: write the anomalous difference map's coefficients to the refined maps
The refinement made the anomalous difference map on the refined model's
phases (for the blob finder and the solvent audit) but did not write it:
<prefix>_refined_maps.mtz carried FP..FREE only, while the validation's
<prefix>_maps.mtz (and _mr_maps.mtz) carried ANOM/PHANOM too.

Where the merge kept the Bijvoet pairs, the refined maps now carry ANOM
and PHANOM as well, the same coefficients the refinement reads (F(+) -
F(-) on the model phase turned back by 90 degrees), missing (NaN) on
reflections without a pair - the layout of the validation's file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:55:15 +01:00
leonarski_fandClaude Opus 5.5 b7d3ddb2b6 Model validation: report anomalous peaks no modelled atom explains
The anomalous difference map was read only at the model's own atom
centres (ANOMALOUS_SITE_01..10), so a strong anomalous scatterer that is
not in the model - an unmodelled metal ion, or one modelled as a water -
never appeared in the report, however high its peak.

The map's own peaks are now searched as well (gemmi's flood fill, each
symmetry copy once): every peak of 5 sigma or more with no model atom
from phosphorus up within 1.5 A is reported as
ANOMALOUS_UNEXPLAINED_PEAKS / ANOMALOUS_UNEXPLAINED_PEAK_nn, with its
height, position (the copy nearest the model) and the model's nearest
atom, and logged. It runs wherever the map is read (model validation and
molecular replacement's reading of a placement).

On 9t6s (open arm) with its four Cd2+ ions deleted from the deposited
model the four ions come out at 8.7-10.9 sigma within 0.4 A of the
deposited sites, 2.5 A from their Cys SG ligands; the ANOMALOUS_SITE list
reads noise (3.6 sigma at most). With the full model, and on 5reo (no
anomalous scatterer) and 6ukf (iodinated, fully modelled), none is
reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:55:15 +01:00
leonarski_fandClaude Opus 5.5 3908956540 rugnux: SHELX .ins that SHELXL accepts - short REM lines, ZERR always, TEMP
SHELXL refused the .ins rugnux writes beside the .hkl: its REM line was
over 80 characters ("** INPUT INSTRUCTION REM Rugnux 1.0.0-rc.... IS LONGER
THAN 80 CHARACTERS **"), and wherever the run had no cell e.s.d.s the ZERR
line was left out, which SHELXL stops on ("** ZERR MISSING **"). SHELXT read
both versions, so the problem showed only once a SHELXT solution, or the
user's own model, went on to SHELXL.

- The remarks are split into lines within 80 characters, TITL is cut to 80.
- ZERR is always written, after CELL; where the e.s.d.s were not measured
  (stills, --mode scale) they are 0 and a REM says so.
- The joint merge of several sweeps measured no e.s.d.s (its fulls carry no
  detector positions), so its .ins had no ZERR on every multi-sweep run; it
  now takes the reference sweep's, which also reach its mmCIF and report.
- TEMP (Celsius) is written from the sample temperature the data file
  records, the same value as the mmCIF's _diffrn.ambient_temp; SHELXT carries
  it into its .res and SHELXL into its CIF (_diffrn_ambient_temperature
  100(2)). Without one a REM says SHELXL will assume 20 C.
- The NXmx reader takes /entry/sample/temperature in its declared unit: K
  (or none) as is, Celsius converted, an unknown unit warned about and not
  read. It was read as K whatever the file said.

The .hkl was already right (3I4,2F8.2, |I| and sigma scaled to at most
9999, 0 0 0 terminator, unmerged at the measured index on rotation data so
Friedel mates stay apart) and is unchanged. Checked with SHELXT (solutions
matching the COD structures atom for atom), SHELXL from the SHELXT solution
and from the COD model, SHELXC/SHELXD on two anomalous protein sets, and
SHELXL reading a 1.2 million-record protein .hkl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 15:29:00 +01:00
leonarski_fandClaude Opus 5.5 cce40c66dc Live stills: refine the geometry on the frames as they are analysed, holding no images
The pre-pass refines the centre and distance on the frames the image buffer
holds. At a serial hit rate that is a few dozen indexed frames (20 of 363 on
a 129.6k-frame JUNGFRAU 8M serial collection), with a buffer of 4 GB none (45
frames held), and on a fixed-target chip whose first wells are empty none at
all: the collection was then analysed at the stated distance, 2.6% short on
that collection, and merged to 1.93-1.96 A.

Where a cell is given and the pre-pass fitted the geometry on fewer than 200
frames, StillsRunningGeometry refines it again from the spots and lattices of
the frames as they are analysed: once 200 strong frames (20 indexed spots or
more) are in, or 5000 indexed (empty frames do not count), the strongest are
bundled and the geometry fitted once, as rugnux fits its sample, and the
analysis switches to it (the analysis threads rebuild their analysis at the
new geometry, a gate keeping new frames back until the switch has the
geometry exclusive). A second round does the same on frames analysed at the
first one's geometry. Two fits, a fixed cost. The frames analysed before the
last switch are left out of the merge: their reflections were predicted at an
earlier geometry and the images are not kept to integrate them again.
DropIntegrationOutcome takes their cells out of the consensus too.

The pre-pass reports how many frames its fit rested on
(ProcessResult::stills_geometry_frames, recorded by RefineStillsGeometry;
nothing it computes changes). jfjoch_stills_gpu_replay holds frames in half the
buffer, as the receiver does, and gives the first images as empty frames with
-e.

On that collection, with the right cell (rugnux: 117.94 mm, 1.30 A, CC1/2
0.9946, R_split 0.062):
- 4 GB buffer: 115.00 -> 117.78 -> 117.96 mm; 1.32 A, CC1/2 0.9962, R_split
  0.070, 99.7%, merge 3.1 s after the last frame
- 32 GB buffer: pre-pass 117.49 on 20 frames, then 117.93 -> 117.98 mm; 1.32
  A, 0.9951, 0.070, 99.6%, 3.6 s (pre-pass alone: 1.35 A, 0.074)
- first 30% of the frames empty: 115.00 -> 118.62 -> 118.06 mm; 1.35 A,
  0.9913, 0.093, 99.7%, 2.2 s (before: 1.96 A; rugnux on the same 70% of the
  frames: 1.33 A, 0.9935, 0.079)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 14:58:16 +01:00
leonarski_fandClaude Opus 5.5 27e0103bcb Live stills: the pre-pass on 500 frames, enough for the distance to be refined
On a serial collection indexed at 18%, the 100 frames of the pre-pass indexed 7
at a stated distance 2.6% short, too few for the geometry refinement to run, so
the whole collection was analysed at the stated distance: 7.6% indexed, merged
to 1.93 A, R_split 0.162. On 500 frames 34 indexed, the distance was refined to
117.51 mm (rugnux on the whole run: 117.94 mm) and the collection merged to
1.35 A, R_split 0.074 (rugnux: 1.30 A, 0.062; at rugnux's centre and distance
the live path merges to 1.32 A, 0.068). 1000 frames gave the same (118.33 mm,
1.35 A, 0.071).

The frames wait in the image buffer, in half its slots; a buffer that holds
fewer starts the pre-pass on as many as it holds, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 14:58:16 +01:00
leonarski_fandClaude Opus 5.5 c723a7b85f Live stills: merge in the run's consensus cell, pre-pass geometry for an untilted detector
Two faults of the live serial path, found replaying a 129.6k-frame JUNGFRAU 8M
serial collection through StillsGPUAnalysis against `rugnux` on the same
frames.

The live merge counted completeness in the collection's cell, where rugnux
merges in the consensus of the indexed frames (the collection's cell only picks
the frames merged). Given a cell 2.4% off, the live merge reported completeness
104.6%. It now takes the consensus cell at its first merge. The consensus
counted only frames whose retained outcome was still in IndexAndRefine, so once
the merge thread moved the outcomes out the END message lost its unit cell;
TakeIntegrationOutcome marks a frame taken and its cell still counts.

The pre-pass refined a detector tilt beside the centre and distance and handed
the centre and distance of that tilted geometry to an analysis made with the
detector square to the beam. On 54 strong frames it fitted 0.18 deg (rugnux, on
the whole run: 0.03 deg) and a distance of 120.05 mm against 118.33 mm untilted,
and the run merged to 1.75 A instead of 1.35 A. The pre-pass now refines for the
detector the receiver analyses with (detector_tilt_given).

jfjoch_stills_gpu_replay prints the consensus cell and the merging statistics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 14:58:16 +01:00
leonarski_fandClaude Opus 5.5 17674b8280 Broker: live serial collections merged, with provisional merges at 1/2 and 3/4
StillsGPUAnalysis keeps each analysed frame's reflections (IndexAndRefine
retains them; a merge thread moves each one out as soon as it is analysed,
TakeIntegrationOutcome, so they are held once) and gives them to a
StillsLiveMerge: at 1/2 and 3/4 of the images a provisional merge on 8
threads, reported as a note of the live status (crystals, resolution, CC1/2,
R_split, completeness, <I/sigma>), the crystals between prepared against it
as they come, and after the last frame the merge of the whole collection on
every thread, before Finish returns. The merged MTZ goes into the end message
as for rotation; the status gets a "Merge" step and the headline resolution.

jfjoch_stills_gpu_replay takes -S (the space group merged in) and -M (no
merge) and prints the merge. Replaying a 129.6k-frame JUNGFRAU 8M serial set
on one GPU: 627 Hz with the merge against 616 Hz without, and the merge done
3.1 s after the last frame was analysed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 14:58:16 +01:00
leonarski_fandClaude Opus 5.5 77958e5944 Stills merge in memory, with checkpoints that leave a bounded merge after the last frame
The stills merge of `rugnux --mode scale` (cell form, relabel, ice rings, the
reference-free ambiguity, StillsScaleMerge) moves out of rugnux_cli.cpp into
MergeStills (rugnux/StillsLiveMerge.h), which takes the reflections in memory
and does no file I/O. --mode scale calls it and merges exactly as before
(every report key unchanged on a 129.6k-frame serial set).

StillsLiveMerge makes the same merge for frames that arrive one by one, at a
fixed cost after the last of them. Its first checkpoint is MergeStills on the
frames so far, which leaves them scaled, post-refined and in a consistent
indexing. Every crystal after it is prepared on its own against that
checkpoint's merge: relabelled, placed in its indexing
(ReindexOnReference, the cells' anchoring kept), scaled, and post-refined with
the checkpoint's rocking width and crystal prior
(StillsPartialityRefine::RefineAgainst, one fit per crystal). A later
checkpoint renews the reference from all the crystals so far. After the last
frame the run's indexing is taken as a whole as the cold merge takes it
(RelabelRunIndexing, the global relabel of ResolveReindexReferenceFree factored
out, unchanged), and one merge is made: the error model, started from the last
checkpoint's and fitted in one sampled and one full round
(MergeOnTheFly::ErrorModelStart), the merge, the resolution cut and the
statistics. Nothing iterates until it converges: the checkpoints are a fixed
list and every fit has a fixed number of rounds.

--mode scale --live-checkpoints f1[,f2] replays a stored collection in image
order through these checkpoints and logs the time and CPU time of each part.
On a 129.6k-frame JUNGFRAU 8M serial set (19.3k crystals, 45M observations,
32 threads, quiet machine): the cold merge takes 20.4 s (320 CPU s); with a
checkpoint at 1/2, 3/4 or both, the merge after the last frame takes
3.5-5.2 s (31-44 CPU s), with CC1/2, R_split, <I/sigma>, d_min and
completeness within 0.001, 0.002, 1.5%, 0 and 0 of the cold merge, the merged
intensities correlating with the cold ones at 0.9995 (0.983 in the outermost
shell).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 14:58:16 +01:00
leonarski_fandClaude Opus 5.5 465a8da51a broker: API GET answers carry Cache-Control: no-store
The broker sent /statistics and the other API answers with no caching headers, so Firefox kept them
in its HTTP cache. A page restored from Firefox's session read the cached /statistics before asking
the broker, and the web frontend's spot-finding panel adopted settings the broker had since replaced
- after a broker restart, the ones that browser last saw rather than the configuration file's.
API GET answers are now marked no-store; the static frontend under /frontend stays cacheable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 14:17:13 +01:00
leonarski_fandClaude Opus 5.5 32ea76356a frontend: spot-finding sliders opt out of Firefox form-state restore
The broker keeps the ice-ring width as float32, so /statistics reports 0.02 as 0.0199999...; the
browser snaps the slider's range input to its 0.001 step ("0.02"), which differs from the value React
set, so Firefox's session store saves the field as user-edited. On session restore (after a crash,
or a restart with "open previous windows and tabs") Firefox writes "0.02" back and fires an input
event; MUI treats it as a committed change and the panel PUTs its whole settings object - by then the
stale /statistics copy from the HTTP cache. A broker restarted from its file went back, with no user
input, to whatever settings that browser last saw: 5 of 22 restore cycles in a graphical Firefox 128.

The panel's slider (one shared component) carries autocomplete="off" on its input, which Firefox's
session store does not save. In this panel a restored change becomes an unsaved edit, sent only with
live sync engaged, but it should not appear at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 14:17:13 +01:00
leonarski_fandClaude Opus 5.5 f7e6ead405 frontend: the /statistics poll gives up after 5 s
/statistics was polled with no timeout, and TanStack Query attaches each interval refetch to the
request still pending, so a single request the broker does not answer stopped all polling: the page
kept showing stale values, or none, after the broker recovered. In Firefox such a request was still
pending after more than 50 minutes. The query is now aborted after 5 s, so polling resumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 14:17:13 +01:00
leonarski_fandClaude Opus 5.5 caf8187dd1 broker: accept adaptive_threshold on FPGA spot-finding detectors and ignore it
PUT /config/spot_finding no longer answers 400 when adaptive_threshold is
set on a JUNGFRAU or EIGER broker. Those workflows find spots on the FPGA at
its own fixed threshold, so the setting is stored and simply not used there;
the DECTRIS/SIMPLON workflow keeps using it. This matches what the config file
already did and lets one set of spot-finding settings be shared across
detectors. The API description, its generated C++ model, frontend client,
Python client docs and Redoc page, and the image-analysis doc now say the
setting is ignored rather than refused. A state-machine test checks the
setting is accepted on a JUNGFRAU configuration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 13:48:34 +01:00
leonarski_fandClaude Opus 5.5 48dcf8218a Report: a stretch removed whole is named as the frames to process on their own
The SWEEP_GAPS warning of a rejected range that the merge removed whole and
took out of the data (RotationScaleMerge::TakenOut) now says what the frames
may be - another crystal, radiation-damaged or dead data - and gives them as
the -s / -e range that processes them on their own: file images, the end
exclusive, through the run's own -s and stride (ProcessResult::first_image,
image_stride). Stretches removed one after the other can overlap; the range
is all of them the warning's range touches.

Processed alone, such stretches range from dead frames to valid data of
another crystal; the user is the one who can tell which is wanted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 13:31:40 +01:00
leonarski_fandClaude Opus 5.5 d04f91fd74 delta-CC1/2: a flagged stretch both CC1/2 call harmful is removed below the least-harm floor
A stretch the per-image channels flag is removed whole when keeping it lowers the plain CC1/2 by at
least DELTA_CC_MIN_HARM. The floor is there for a stretch the merge still learns from - a weak tail on
a crystal merging at 0.999 - but it also kept stretches that carry nothing: frames with the sample out
of the beam, at a tenth of the run's scale, whose removal raises the plain CC1/2 by 0.001-0.002 at
6-20 standard errors and whose noise dominates the reported R_meas. Such a stretch is now removed when
the information-weighted CC1/2 says, as significantly, that keeping it is harmful too, and its plain
cost is beyond the null of the unflagged batches: then the merged intensities do not want it either.

The decision no longer depends on whether the batch loop happened to slide its edges over the same
frames, which flipped between versions on a strong data set with two out-of-beam arcs.

The plain and weighted measurements of every flagged stretch are logged at debug level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 10:39:20 +01:00
leonarski_fandClaude Opus 5.5 f0832ee39e Rotation merge: a stretch taken out of the data drops the memoised partial scaling loops
The CPU partial scaling loop is memoised by grouping and settings, and a memo holds the corr of every
partial by index. Taking a stretch out re-ingests the sweep without it, so a memo made before then
indexes a different set of partials: the rescale took the stale corr, its per-frame scales collapsed
(hundreds of frames dropped at scales thousands of times below the run's) and ISa fell to ~1. Only the
CPU path memoises, so --no-gpu runs (and the CPU-only builds) were hit wherever a removed stretch was
taken out and the partials were scaled; the GPU path refits and was unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 10:39:20 +01:00
leonarski_fandClaude Opus 5.5 c5fc3fa1d6 Beam stop: looked for with the CCD's inactive pixels read as data
The pixels outside a CCD's active area, constant over the gain frames, are masked as dead before the run
starts. A CCD reads the deep shadow of a beam stop at one value too, so that mask took the shadow's
core, and the shadow finder - which reads only unmasked pixels and grows the shadow from where it is
deep - then found a fraction of it or nothing. Of the 48 open-arm CCD rotation sets, about half lost
most of their shadow: 9.35% -> 1.11% of the detector on PDB 3MEB, 10.79% -> 2.60% on 3R6O, 1.54% -> none
on 9UPT; the CCD stills sets lost theirs too. What went unmasked was the penumbra about the core, which is
not constant and was integrated as if exposed, and the shadow bit the integration's test of clipped
reflections reads. On 7BGU the background centre the finder draws its rings about moved 56 px.

The beam-stop step - the shadow finder and that centre - now reads the inactive pixels as the input has
them (ProcessConfig::inactive_pixels, passed by the CLI and the viewer where they mask them); everything
else keeps them masked as dead. On 11 CCD rotation sets the shadow is again exactly what it was before
the inactive mask, the mask's gain on 7BGU stays (d_min 2.42 A, R-free 0.190), 9UPT R-free 0.225 -> 0.219,
2WNN ISa 14.5 -> 15.4, 3MEB d_min 1.65 -> 1.56 A; the CCD stills sets get their shadow back (R-free within
0.001). Photon-counting sets are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 09:17:23 +01:00
leonarski_fandClaude Opus 5.5 d3d426aa8a Stills given a cell: the population kept is told by the cell's proportions, not its volume
A sample holding two crystal forms indexes both against the given cell, and their refined cells fall
into two volume populations. The one kept - by the geometry refinement's bundle and again by the merge
- was the population whose mean volume lay nearest the given cell's. But the volume a still refines to
scales with the detector distance it was refined at, and that distance is what the geometry refinement
is there to find: it starts from the stills distance search, which on a two-form sample settles between
the two forms' best distances.

On a two-form CCD stills set (Rayonix, a P21 form and a second form 9% smaller in volume) the search
started the refinement 2% long. There the given form's frames came out 5% larger than the given cell
and the other form's 4% smaller, the other form was kept, the bundle fitted the given cell to it and
moved the distance a further 2% the wrong way (205.6 mm, against the 197.6-198.5 mm both forms' runs
otherwise agree on), and at that geometry the merge kept the other form again: R-free of the deposited
model 0.285 -> 0.432, d_min 2.03 -> 2.28 A, a twinning warning. Which way the start fell depended on
little: the CCD inactive-area mask moved it by 0.5% and flipped the choice.

The populations are still found on the volume histogram; among them, the one kept is now the one whose
mean cell is nearest the given cell in its proportions - the logarithms of its edges less their mean,
and the unsigned cosines of its angles, each frame put on the given cell's axes (AlignedCell) - with its
size counted at a third, so that two forms of one shape are still told apart. The proportions do not
move with the distance. On the set above both forms differ most in the monoclinic angle (91.2 vs 93.5
deg), and the population distance separates them 7-20x at every stage and start geometry seen.

On that set, with the inactive-area mask: the given form kept at both stages, distance 197.6 mm, R-free
0.432 -> 0.286 (0.285 before the mask), d_min 2.28 -> 2.08 A; the same frames given the other form's
cell are unchanged (R-free 0.237). A run with one population is unchanged, and every other set of the
still battery has one. GivenCellPopulation takes the frames' cells and the given cell instead of
relative volumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-05 09:17:23 +01:00
leonarski_fandClaude Opus 5.5 52c2c15b85 frontend: spot-finding panel edits locally, sends on Upload or with Live sync
The spot-finding panel was the only settings panel that sent every control
change straight to the broker, so a stray click, a tap on a touch screen or an
arrow key on a focused slider reconfigured spot finding on the running
acquisition. It now works like the other panels:

- Edits are kept on top of the broker's copy and sent, merged onto it, only
  when Upload is pressed. Revert drops them. The title marker and a status line
  say whether the panel shows the broker's settings, unsaved edits, or live sync.
- Live sync sends each finished change at once, as before. It is off after
  every page load (and every remount), and can be engaged only from a clean
  panel, so it never sends edits made before it.
- An edit counts as done when the broker reports the same value (compared with
  a float32 tolerance); a rejected one stays and keeps the panel marked unsaved.
- No controls until the broker's copy has arrived (unchanged).
- Switches get labels (FormControlLabel), sliders an aria-label; the nine
  slider blocks share one small component.

SettingsPanel gains an optional `actions` slot beside its Upload button.

Checked with jfjoch_broker (HLS-simulated) behind a logging proxy and headless
Chromium: across 20 scenarios - including a knob click, a touch tap, an arrow
key on a focused slider, a double-click on the view entry and a click storm
while the panel mounts, and failing/slow/hanging /statistics - the only PUTs
were the one Upload press and the two changes made with live sync engaged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 9f0e8458c9 frontend: spot-finding panel never PUTs its built-in defaults
The panel started from a hard-coded settings object (adaptive threshold off,
high-resolution limit 2.5 A, count threshold 8, SNR 3, ...) and showed it until
the /statistics poll delivered the broker's copy. Every control PUTs the whole
panel object, so any touch of a control while that default was on screen -
even a click on a slider knob that changes nothing - replaced the broker's
spot-finding configuration with the frontend default. The window opens on page
load while /statistics is slow, and for as long as /statistics fails or hangs
(retry is off, so one failed poll is enough): the dashboard view switched away
and back during a failed poll remounts the panel on the default.

The panel now has no default: it shows a waiting note and no controls until the
broker's copy is in hand (and while the poll is failing), and each control sends
only its own field on top of the panel's copy of the broker settings.

Reproduced with jfjoch_broker (HLS-simulated) behind a logging proxy and headless
Chromium: before, 6 of 12 scenarios (delayed, failing, hanging or stripped
/statistics, remount during an error) PUT the default object; after, none do,
and PUTs from an adopted panel carry the broker's values with one field changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 690fdb0600 Stills de novo: refine the detector distance on the cell the frames index to
The stills geometry refinement (beam, distance, cell bundle) and the distance search ran only with a
reference cell (-C / -z). Without one the file's distance was kept for the whole run. On two serial
datasets whose header states the distance 2.6% short, every frame then refined its cell at the wrong
distance, the cells scattered with orientation and the run split into two spurious crystal forms:
CC1/2 0.13-0.14, 72-77% complete. With -C the same frames refine the distance to within 0.1 mm of
the value both runs agree on and merge cleanly.

De novo, RefineStillsGeometry now takes the consensus cell of the frames it samples (keeping only
the frames on it) as the bundle's anchor, --refine-geometry defaults on for all stills, and the
provisional cell is cleared before the main pass, which is still indexed de novo.

10000-frame subsets, de novo:
  set with a 2.6% distance error (C2): 115 -> 118.02 mm, 1 form, CC1/2 0.13 -> 0.92, 77% -> 99% complete
  set with a 2.6% distance error (R3, merges in P1 without -S): 115 -> 118.2 mm, 2 forms -> 1,
    CC1/2 0.14 -> 0.44
  standard lysozyme set with a correct header: 115.18 mm, CC1/2 0.990 (0.991 with -C/-S)
The stills battery always passes -C/-S, so it does not exercise this path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 e40eb08a3b Stills with a cell but no space group: keep the cell's angles and setting
With -C and no -S, two things broke a cell whose angles are not all 90 deg:

- DiffractionExperiment::GetCrystalSystem() returned Monoclinic when no
  space group was set, and the stills geometry bundle adjustment (and the
  tilt trial) parametrise the cell by that system, so a hexagonal cell was
  refined with alpha = gamma = 90 deg. The refined cell then fed FFBIDX for
  the whole run: on a trigonal crystal 5.9% of the frames indexed instead of
  16.4%. Without a space group the system is now Triclinic.

- IndexAndRefine replaced the lattice FFBIDX returned on the given cell's
  axes with LatticeSearch's conventional setting, chosen frame by frame, so
  frames of one crystal form landed in different settings (89% of an
  R-centred hexagonal run reported as a second crystal form). The FFBIDX
  lattice is now kept, refined with every angle free.

Runs with -S take the other branch and are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 0dd0b89b48 Rotation merge: a stretch removed whole is taken out of the data and the sweep scaled again without it
The delta-CC1/2 stretch rule decides last, on the corrected fulls: by the time
it removes a flagged stretch - a dead stretch at a tenth of the crystal's
scale, a second crystal or a slipped orientation taking over - every per-frame
scale, correction surface and the error model had been fitted with that
stretch in, and only the error model was refitted after it. So the first merge
that may be written (in the experiment's own group, not a search or pre-pass
merge) and removes a stretch whole now clears that stretch's reflections from
the outcomes and makes itself again from the start without them; every later
merge, and an engine made again after a change of basis, keeps them out
(RotationScaleMerge::TakenOut). The run's own batches are judged again on what
is left. The ledger keeps the stretch at the per-image scale, CC and cost it
was removed on; the quality guard's CC1/2 before corrections stays that of the
merge that still held it. A copy of the engine (a merge made ahead on another
card) takes nothing out; where such a merge removed a stretch the engine has
not taken out, it is made again on the engine.

The stretch-whole rule fires on 10 of 118 sets of the battery subset (78
open/in-house, 25 private, 15 multisweep); the other 108 are identical to the
rule alone. Against the rule alone, on the 10 (--no-refine): R_meas lower on 7
(an open set 0.425 -> 0.280, ISa 17.2 -> 19.9, REFMAC R-free 0.244 -> 0.243; a
private monoclinic set CC1/2 0.868 -> 0.922; the in-house half-image lysozyme
0.620 -> 0.559), about equal on 3. Costs: a powder-ring myoglobin sweep is
cut at 1.90 instead of 1.49 A (R_meas 1.48 -> 0.85), one private triclinic set
gives up another 5 points of completeness, and the ISa of two private sets
with very high ISa falls (60.8 -> 23.3, 45.3 -> 30.9) at equal R_meas. One
extra scaling per triggered run; wall time unchanged within noise.

Processing the sweep again as segments (branch stretch-v2-seg) recovers more
on two sets whose problem is integration-side, at 2-4x the wall time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 756bf454b9 delta-CC1/2: judge a flagged sweep stretch whole, on the plain CC1/2
A rotation sweep can hold a long stretch that no longer carries the main
lattice's signal - the crystal dying part-way through a room-temperature
sweep, a second crystal or a slipped orientation taking over the second half,
the crystal out of the beam. The batch loop could not remove it: its candidates
stop at the 25% cap, its null is the run's own batches (meaningless when half
of them are the defect), and it measures the information-weighted CC1/2, which
by construction cannot see the noise such a stretch puts into the reported one.

After the batch loop, each stretch the per-image channels flagged (the ledger
ranges, neighbours closer than one rocking event joined) is measured whole on
the plain sigma-tau CC1/2 (every reflection at weight 1) and removed when the
cost is resolved beyond 3 standard errors, at least 0.01, and below the robust
floor (median - 3 MAD) of the batches neither channel flags; with fewer than
three such batches nothing is removed. Worst first, the rest re-measured.

(Cherry-picked from stretch-reject 2d56ddb39 without its per-frame scale
choice change, which is the previous commit.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 ce2d2280fd Per-frame scale choice: an error model counts as resolved when the bootstrap bounds ISa
The choice between the partials-scaled and fulls-alone per-frame scale read an
error model as resolved only when the bin fit's b was two standard errors clear
of zero. The bin fit reads a b too small for its 16 bins to measure exactly as
it reads one that failed, so a merge that got better - a sweep whose dead second
half had been removed, its ISa bounded on both sides by the bootstrap over the
reflections - was taken for a failed scale, and the fulls-alone arm, with nothing
resolved either, won by default. Resolved is now either test saying so.

On the rc181 battery logs this flips no arm decision; it only skips running the
fulls arm where partials were chosen anyway. One private set moves to the
fulls-alone arm, now resolved by the bootstrap (ISa up, merging statistics equal).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 3beb19f09f Mask: the inactive-area test is gated on the frames' zeros, not on the file format
The constant-pixel test needs frames that carry read noise. It was switched off for the XFEL
formats by name (and not for XTC, which is one too). It is now switched off by the data: where
the pixels that do vary read zero on 1% or more of their readings - counted photons with no peaks
to show it, or a file cut at a threshold - a pixel at zero on every frame may simply have had
nothing to record, and nothing is listed. A CCD's live pixels sit on its offset: on every
rotation CCD set of the battery they read zero on under 0.001% of their readings, on the CCD
stills sets under 0.3%; frames of a JUNGFRAU converted to whole photons on about 8%.

Stills: the Rayonix CXI sets now get the mask as well (frame-edge zeros, about 0.15-0.19% of
the detector): R_split 0.0908 -> 0.0874 and 0.0802 -> 0.0792, R-free +0.0004 and -0.0015. The
MPCCD sets show no constant pixels beyond the file's own mask; the photon-converted JUNGFRAU and
AGIPD sets are unchanged. Rotation results are identical to the previous commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 8241869f09 Mask: pixels outside a CCD's active area, found as constant over the gain frames
An input in detector units whose background shows no photon peaks (a CCD, image plate
or other integrating detector read through an amplifier) has read noise on every live
pixel, so no live pixel reads the same value on all 16 frames of the gain measurement.
A pixel that does is outside the active area - the corners of a round CCD, a border or
seam the detector's corrections leave at zero - and is now masked as dead (NXmx bit 1)
before the run starts. The frames are already read for the gain, so this costs one
comparison per pixel and frame.

The existing dead-pixel test compares a pixel with its own resolution ring and cannot
see an area that covers whole rings: on a round CCD every pixel beyond the circle's
radius is zero, so its ring reads zero too. Integrated there, reflections straddling
the edge read as strong, and the high-resolution corners set a bogus resolution limit.
On PDB 7BGU (round marCCD, 22% of the frame outside the circle): d_min 1.78 -> 2.42 A
(deposited 2.43), completeness 55 -> 98%, ISa 16.9 -> 23.1.

Not applied to photon-resolving inputs (their pixels may read zero photons on every
frame) or to the XFEL formats (a file cut at a threshold reads zero on live pixels).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 f31ac225e9 Integration: the profile-vs-box guard bounds the profile's excess by the tail the disk can miss
rc182 kept every profile intensity above a box sum of more than 20 sigmas. That removed the
fallback on strong spots whose profile reads a few per cent above the box (the r1 disk misses the
tail), but it also kept profiles that read 1.5-30x the box sum. Those are not tails: the fit grid
reaches out to r2 and picks up signal the disk does not hold - a second lattice, split spots, an
unpredicted neighbour, a dead detector edge. Per-reflection dumps showed them on the sets that
lost (on 7bgu 99% of the strong reflections at the outer detector edge, with a negative background;
on split and multi-crystal sets 10-25% of the strong reflections, at every resolution), while on the
sets that gained the kept reflections read what the strong population reads in their shell
(typically 1.03-1.20). Against the symmetry mates in 9fhc's unmerged output the rc182 guard added
~14000 observations more than 100% off.

The guard now falls back above the box sum only when the profile exceeds it by more than the
10-sigma margin plus 25% of the box sum (PROFILE_SUMMATION_MAX_TAIL). Below the box sum it is
unchanged. Weak reflections see a margin of 10 sigma plus at most 5 sigma, so the runaway guard
stands.

CPU-only, against the rc182 guard: 9fhc goes back to d_min 1.90 / ISa 11.1 (rc182: 1.95 / 13.5,
before rc182: 1.90 / 10.8); 7kcn R_meas 6.39% and 6v2r ISa 36.6 keep rc182's gain in full. GPU
equals CPU on 7kcn and 6v2r.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 21:53:52 +01:00
leonarski_fandClaude Opus 5.5 5bc5fdf6cf Gain: the frames of the gain measurement are read and measured side by side
RugnuxMeasureGain read and measured its 16 frames one after another, 1.25-1.5 s
of step 1 on every CCD/SMV input. Each frame now runs on its own thread with its
own levels and histogram, put together in frame order: the same integer sums and
the same list of levels, so the same gain. Step 1 on marCCD/SMV sweeps 1.3 -> 0.5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 17:35:21 +01:00
leonarski_fandClaude Opus 5.5 c76304d899 Integration: the shadow-clip test reads only each row's span of shadow pixels
DropShadowClipped scanned the whole signal disk of every predicted reflection
in the mask, ~10% of all cycles of a rotation run. The analysis now takes, once
per object, the first and last beam-stop pixel of each mask row (ShadowRows),
and the test reads only that part of the disk's rows; no shadow pixel lies
outside it, so the same reflections are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 17:35:21 +01:00
leonarski_fandClaude Opus 5.5 79a80f086c Twin-composite hypothesis: count the spots per class of hkl mod n, not per pair of forms
Whether a reflection lies on an index-n sub-lattice depends only on its indices
mod n, so TwinCompositeAt histograms the validation spots over the n^3 classes
once and counts each congruent pair of forms over the classes on the first form's
sub-lattice. The counts are the same integers as before, so the decision and the
log are unchanged; the index-2..10 loop drops from ~0.45 s to ~10 ms a cell, and
it runs for the standing cell and every multiple in every first pass
(on a median open set ~3 s of the first passes, single-threaded).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 17:35:21 +01:00
leonarski_fandClaude Opus 5.5 6f8c77a6ef 1.0.0-rc.183: VERSION and the version strings update_version.sh writes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu
2026-11-04 17:35:21 +01:00