7 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	4af23e9b27	model: build the null only where the model claims something ... The null costs about 12 s - five replicates, each a full fit and a rigid-body placement - and it gates exactly two decisions: the enantiomorph label and the merohedral indexing. A model that claims neither, already in the group the data were merged in on a crystal with no indexing ambiguity, has nothing for the null to arbitrate, and paying for it there is 12 s spent gating a decision nobody is making. That is the isomorphous fragment-screening run - hundreds of crystals of one form against one apo model, a map wanted seconds after the last image - and it is the case that has to be fast. So the null runs only when a decision is pending: an enantiomorph candidate exists, or the indexing probe returned a winner that is not the identity. The probe is a handful of scaling fits and had to run first anyway, so the question is answered before the expensive part starts. Measured on a rotation dataset, --mode scale --model, interleaved on a loaded machine: the isomorphous case is 3.6-4.7 s before this whole line of work and 3.8-4.7 s with the gate, against 16.5 s while the null ran unconditionally. The case that still pays is unchanged at 17 s, and the three wrong-model cases give bit-identical verdicts and bit-identical files to before. The rigid-body refinement of the replicates stays: 3 of 5 commit a placement, so without it the null would score a weaker procedure than the one it judges. MODEL_FIT gains a third value, NOT_TESTED, alongside ACCEPTED and REJECTED - the question was never put, which is not the data answering it badly - and the MODEL_FIT_NULL_* and MODEL_FIT_SIGMA keys are absent with it rather than reporting a null that was never built. REPORT_VERSION stays 6: MODEL_FIT is itself new in this release, so its vocabulary has never shipped and a consumer cannot be relying on the two-value form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:06:29 +02:00
leonarski_f	c999883eb1	model: --model is a hypothesis, and it decides nothing until it fits ... A model supplied with --model rewrote the space group of every reflection written out on the strength of its file having parsed. Measured on a rotation dataset merged in P4(1)2(1)2: an unrelated protein and the correct model rigidly rotated 90 degrees each produced a .mtz, .cif and _unmerged.mtz byte for byte identical to what the crystal's own model produced - relabelled P4(3)2(1)2 - at R-free 0.601 and 0.674 against the correct model's 0.591, with no warning. The adoption was pure space-group-number arithmetic and ran before the model had been fitted at all. A model changes exactly two things on the rotation path, and both rewrite the data: the enantiomorph label and the merohedral indexing. Both now wait for the fit. Everything else a model produces - R-factors, maps, the rigid-body placement - is a statement about the MODEL, cannot corrupt a reflection, and is computed and reported either way. The gate is not a threshold on R, because no threshold works: the classical acentric random value is 0.586 at unit scale but 0.550 at the R-minimising scale, observed nulls land at 0.599-0.615, and the value moves with the model's atom count and B-factors as much as with the data. Instead the same model is re-oriented at random about its own centroid five times and run through the identical path - same scaling, same rigid-body placement, same R - and the real fit is asked how far above that distribution it sits. R-work carries the decision: nothing is refined against the working set here, and it has 12615 reflections to R-free's 709. Measured on the case above: the crystal's own model +15.0 sigma, the unrelated protein +1.8, the 90-degree rotation -1.0, and the two rejected runs now write files byte-identical to a run with no model. The nulls are rigid-body refined like the real fit, or the comparison would be between a placed model and unplaced nulls. That is what the null costs: about 12 s for the five replicates, on a --mode scale run that merges in 2 s. The merohedral margin gets the same treatment - a random placement also picks a winner, and measured, by a comparable lead - and the candidate operators are now enumerated from the DATA's space group. Taking them from the model's enumerated zero operators wherever the two groups differ, which is exactly the case the probe exists for: a model in P4(3)2(1)2 against data merged in P4(3) probed nothing at all, and now probes the twin law. The anomalous difference map is the only measurement here sensitive to the hand - inverting the model through the origin moves R-work by less than 1e-4, since |F(h)| of the inverted structure is |F(-h)| - so where it says the hands disagree it vetoes the adoption outright, fit or no fit. Report: MODEL_FIT, MODEL_FIT_SIGMA and the null beside it, MODEL_DECISIONS_TAKEN, the indexing margin against its null, and MODEL_VALIDATION= PERFORMED as the counterpart of the failure line. SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL becomes ASSUMED_FROM_MODEL and is written only where the model was accepted: nothing here measured the hand, the model asserted it. That is a reason code changing name and meaning, so REPORT_VERSION is 6. Stills are untouched: the per-image indexing hand a model can set at integration time is not reachable on the rotation path and is not gated here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 13:34:52 +02:00
leonarski_fandClaude Opus 5	200a2d53b7	report: name the groups the data could not separate, as keys ... SPACE_GROUP_NAME is a scalar and reads like a determination, and the docs nominate a grep of it as the interface. Where several groups predict the same absences it is one of them, picked by convention: the report said so in prose and nothing machine-readable carried it, so a script recorded a coin flip as an answer. Refining against a deposited model in the wrong enantiomorph gives R = 0.549. Four keys, none of which change which group is adopted: the alternatives the data cannot separate; whether the enantiomorph was determined, given, or is undetermined - decidable from the group number alone, so it is answered even where the search did not run; and the higher point group whose promotion was refused, with its reason, which until now existed only as prose. REPORT_VERSION is 5. No existing key changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	26fc4b02b3	symmetry: carry the space group as the group, not as its number ... The adopted space group travelled the pipeline as a bare int and was rebuilt downstream with find_spacegroup_by_number, which returns the reference setting. So every setting a number cannot name was destroyed one line after it was determined: P 1 1 2 came back as P 1 2 1, I 1 1 2 as C 1 2 1, R 3:R as R 3:H. DatasetSettings now holds the gemmi::SpaceGroup itself, DiffractionExperiment exposes it as GetGemmiSpaceGroup() / GetSpaceGroupOrP1(), and everything that used to take an int - HKLKeyGenerator (its int constructor is gone, so the compiler finds the callers), the merge, the R-free flags, French-Wilson, the reindexing ambiguity, the completeness enumeration, the MTZ and mmCIF exports, the model validation - takes the group. -S keeps the setting the symbol names rather than reducing it to a number. The end message carries both spellings and a reader prefers the name, since only the name keeps the setting while the number is what a reader written before the name understands. It carries them over CBOR too: the determined group was never serialised at all, so a group rugnux chose reached the master file only when the same process wrote it, and an online writer fell back to whatever the user had supplied at the start. Both keys are optional additions, so an older reader skips them and a newer one reads an older sender. On disk the master's /entry/sample/space_group carries the extended Hermann-Mauguin name and is what the reader takes the group from, so a setting survives a _process.h5 and the --mode scale that re-reads it; the number stays beside it and is the fallback for files written before. Every one of the 230 reference settings the old writer could produce reads back as itself, so older files are unaffected. Stage A and Stage B of the search still enumerate reference settings only, so this determines no group differently today - it is what the enumeration needs before it can be widened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:16:43 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00