5 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	94ed864dbe	model validation: the model is put into the data's description of the lattice before it is placed ... A deposited model arrives in the cell its depositor chose and rugnux indexes in the cell its own reduction chose. The two are frequently different descriptions of the SAME lattice - I-centred where the other is C-centred, unique axis c where the other took b, a cyclic permutation of an orthorhombic cell - and the space-group NUMBER is identical in every one of those, so the existing enantiomorph check, which compares numbers, cannot see any of them. Re-fractionalizing straight across such a pair scrambles the model, and the rigid body cannot undo it: six parameters about a centroid are not a change of basis. The run then reported a placement R-free near 0.6, CC(model,data) around 0.3 and MODEL_FIT= NOT_TESTED, with no warning, for data that were perfectly good. Before re-fractionalizing, every integer change of basis with entries in {-1,0,1} and determinant +1 that carries the model cell onto the data cell is enumerated, quotiented by the model's own point-group rotations, and scored on an 8-3.5 A shell after an overall scale and an isotropic B. The lowest wins. The identity is always among the candidates, so a probe that finds nothing better than leaving the model alone leaves it alone; and on the ordinary isomorphous run the only operators that map the cell are the identity and its own symmetry equivalents, so they collapse to one candidate and the scoring is never reached. A change of basis is a matrix AND an origin shift, and the shift is not optional. An odd permutation of a screw-axis group lands on the same group on a moved origin, whose operator list GEMMI cannot name because it compares those lists exactly - swapping b and c in P 21 21 21 needs (1/4, 1/4, 1/4) first. Where the bare matrix names nothing, the shift that makes it name something is searched for on the twelfths every crystallographic origin shift lies on. The coarse shell is what makes trying every candidate affordable: a frame that is wrong is wrong at low resolution. The lowest resolution is excluded with it, because there an unmodelled bulk solvent would dominate - equally for every candidate, so it would only add noise. Measured cost 0.04-0.52 s, and only on runs whose cells actually differ. A halved or a doubled axis is deliberately outside the search space: those are indexing errors, not a choice of description, and a probe that silently repaired one would hide a real defect. So is the case where the lattice is right but the symmetry was undercalled to P1, which would need the model expanded to P1 and a centred-to-primitive basis change. Measured over the deposited-data battery: eight runs whose model had never been placed now place it, R-free falling by 0.14 to 0.36 and CC(model,data) rising from 0.14-0.47 to 0.51-0.96; density at atom centres 0.35-2.1 sigma to 1.5-4.8, map mean FOM 0.13-0.28 to 0.36-0.93. Five controls chosen to exercise the probe moved by no more than run-to-run noise and took no operator, one of them after correctly rejecting a real alternative frame at coarse R 0.56 against 0.19 for the identity. The reflections are not touched: they keep the indexing the reduction determined, and it is the model that is expressed in it. What this fixes is that the written model and the written reflections now agree - Fcalc from the placed model against the merged amplitudes, both as written, gives R 0.18-0.31 where it gave 0.54-0.65. The CRYST1 header was already correct before, since WriteModel forces the data's cell and group, so a downstream program would have loaded the pair without complaint and refined against scrambled atoms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XbMaLuuwX4D58QCxZ686Ku	2026-09-08 11:06:07 +02:00
leonarski_fandClaude Opus 5	e5c96cd41b	model validation: the placement target carries a bulk solvent that means something ... The rigid-body target refits the scale at every evaluation, deliberately - held fixed, the target would measure the scale as much as the placement, and the body would translate to repair a scale error instead of moving to where the density is. That refit was gemmi's unbounded fit, the one already replaced for the reported scale, and here it was worse: measured over a corpus of deposited models, 40% of the evaluations that decide where the model goes came out with a b_sol outside 10-80 A^2, on 57 of 89 datasets, ranging from -8072 to +1721. A negative b_sol is a solvent term that GROWS with resolution. One crystal ran its entire committed placement between 290 and 430 A^2, and that placement went into the reported maps. The bulk solvent is fitted once per zone instead, inside the same physical box the reported fit searches, and then held while the overall scale and the anisotropic B keep following the body. That split is the point: k_sol and b_sol describe the crystal's disordered solvent, not the fit of one placement, and measured across a whole zone they drift by a single grid step. Fitting them at every evaluation costs three times the wall clock, makes the scaler 82% of the run, and moves discontinuously under a forward difference - which is poison for a numerical Jacobian. This costs 4%, leaves no evaluation outside the box, and lands the body within 0.37 degrees of the expensive version, against 4.33 degrees for what it replaces. Placements change on fourteen crystals in eighty-nine. R-free is a wash on the mean; the step buys more total R-free from fewer commits, and the gain sits where the runaway actually bit. Two things found while auditing the file and left as they were, because they are right: the refinement sees working reflections only, end to end, and the gauge that removes the origin-free directions of a polar group carries more than half the whole refined translation in ninety-four of a hundred and thirty-nine polar zones. It had no test. It has one now. An observation with no calculated amplitude gets a zero residual, which drops it from the target rather than scoring it as a perfect fit, and is now counted and reported - a large count says the model's reflection conditions do not match the data's, which is a statement about the model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 15:27:30 +02:00
leonarski_fandClaude Opus 5	004fa5a781	model validation: the model says, shell by shell, where the data still carry signal ... A placed and scaled model gives an expected intensity for every reflection, and comparing that with the merged intensity is a measurement nothing else in the report can make: CC1/2 and R_meas compare the data with themselves, and can only say whether a shell is reproducible, not whether it is right. CC(model, data) is reported per shell, on the merge's own shells so the rows line up, with the reflection count and a significance beside each. Read it in one direction only. A correlation significantly above zero proves signal is present, because a model cannot invent agreement with data it never saw. A correlation near zero proves nothing, since the model may be the thing at fault - one crystal in this corpus reads negative across its outer shells against a deposited model of a different crystal form. So this can support keeping a shell and must never be used to discard one, and nothing in the pipeline acts on it. Intensities rather than amplitudes, matching CC1/2 and CCref beside it. The raw merged intensity rather than the French-Wilson amplitude: that amplitude is a posterior mean under a Wilson prior which pulls weak reflections towards their shell mean, and the shells this number is read in are exactly the weak ones, so it would correlate the prior with itself. The free flag is ignored on purpose. Nothing has been refined against these reflections, so the correlation is unbiased over all of a shell rather than over the test set alone - which is where the outer-shell reflection counts are, and with them the difference between a statistic that can resolve something and one that cannot. REPORT_VERSION is 7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 13:12:32 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00