267 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	e421053b30	docs: the viewer's calibration fix is user-visible, so the release note says it ... It rides the existing viewer entry rather than taking a bullet of its own - the release note is deliberately about a dozen lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 17:44:46 +02:00
leonarski_fandClaude Opus 5	b8fa8e67d5	docs: the pages catch up with the last day on rc166 ... An audit of docs/ against the code at HEAD, concentrating on what landed after the previous documentation audit: the balanced half-set split and CCanom, the declared-range completeness denominator, the one rule for a quantity nobody measured, and the PONI the calibration now refuses to write. Five statements the code had made false: - The rotation merge's CCref column shows a dash, not "nan". - Two pages still promised unweighted 2Fo-Fc / Fo-Fc maps; they have been sigma_A-weighted since the rigid-body work, and one of the two sat six lines from a paragraph that said so. - Inserting the CCanom prose into the SigAno paragraph left the sentence about the PDBx items and the SigAno column with CCanom as its subject, so it read as a claim about a quantity that is in neither. CCanom is also rotation-only and is not in the mmCIF, which nothing said. - Two cross-references did not resolve - a heading that was renumbered, and a slug spelled without the hyphen MyST puts in "TCP/IP". Added where the behaviour is new and a user meets it: the report's contract for a quantity a run did not measure - no key, and a dash in the table, which is not the same claim as a measured zero; the half-set rule behind CC1/2, which is why a CUDA and a non-CUDA build now agree on it and on CCanom; and the second reason a calibration writes no .poni, a detector whose stored image is mirrored or quarter-turned, which the PONI format cannot state. The rc.166 changelog stays a release note. One entry is widened from the shell table to the rule it is a case of, and two are added for output that was wrong rather than merely undocumented: FITTED_RESOLUTION was the P1 cross-check's, and the unmerged MTZ carried the reference setting where the merged file carried the adopted one. Sphinx builds clean with -W on the pinned docs/requirements.txt, and every intra-doc anchor resolves against the generated HTML. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 17:44:20 +02:00
leonarski_fandClaude Opus 5	7bb74a9dc8	review fixes: two silent wrong outputs, and one rule for a quantity nobody measured ... From four whole-branch code reviews of rc166. Reviewing the net diff rather than the commits found what per-commit review structurally cannot: a later commit leaving an earlier one's claim standing, and two cases of a later commit quietly undoing an earlier one. THE FITTED RESOLUTION WAS THE P1 CROSS-CHECK'S. The block that merges in P1 to write <prefix>_P1.mtz saves and restores the error model around itself, because that merge is not the run's answer. A later commit taught the same function to report the CC1/2 resolution fit and did not extend the list, so every de-novo rotation run in a non-P1 group has been quoting FITTED_RESOLUTION - the number the report itself calls the one to quote - from a merge with n_ops times the unique reflections at a fraction of the multiplicity. AND IT ROUND-TRIPPED THE SPACE GROUP THROUGH ITS NUMBER. A number names only the reference setting, which stopped being enough when the search learned to adopt P 1 1 2(1) or I 1 2 1. Everything written after that block - the unmerged MTZ included - therefore carried the reference setting while the merged file carried the adopted one: two files describing one dataset in two different settings. It now carries the group. A PONI CANNOT STATE A MIRRORED OR QUARTER-TURNED DETECTOR, and the fits became orientation-aware on this branch while the writer did not. It wrote five numbers that silently described a different geometry from the one measured; it now refuses, and says the JSON beside it has the full one. A REFUSED FIT'S ERROR BARS COULD BE HANDED BACK AS AN ACCEPTED GEOMETRY'S. RingOptimizer::Run writes its uncertainty only where the solve is usable, and CalibrateFromSpots runs it twice - tilt free, then tilt pinned. A failed second fit kept the first's sigmas, valid flag and all. It is cleared on the way in. ONE RULE FOR NOT MEASURED. The report had four conventions for it and printed the same missing quantity two ways on adjacent lines: SIGANO as the literal "nan" and CC_ANOM by absence, for a Friedel-merged run that split no Bijvoet pair - which is the default. A quantity a run did not measure now writes no key, and the shell table's dash follows the same rule rather than a 0.0% that reads as a measured total failure. ANISOTROPY_D_MIN_BEST also stops printing nan when only its first principal direction is unmeasured. Four claims that a later commit made false are corrected where they stand: the merge header promising an order-independence the balancing rule gave up, the reference page arguing against CCanom 28 minutes before it shipped, the screw threshold whose "three dead reflections clear it" the evidence floor caps at 19.2 nats, and a shell comment calling equal width in 1/d^2 equal volume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 17:27:56 +02:00
leonarski_fandClaude Opus 5	8661193712	merge: balanced half-sets by rank, and CCanom read off them ... Two statistics and the machinery they share. THE SPLIT. CC1/2 correlates two half-set means, so a reflection whose observations all land in one half has no second mean and contributes nothing. The half was a hash of the image index alone, which does that to 2^(1-n) of the reflections at multiplicity n: half of the doubly measured ones, a quarter of the triples. Measured on in-house rotation data that is 0% of a 26-fold redundant sweep but 6-30% of a 3-to-5-fold one, and 35-60% of its outer shell - the shell the automatic resolution cutoff is read from. An observation's half is now the PARITY OF ITS RANK among its reflection's observations, ordered by a key built from the raw Miller index and the peak frame. That is exactly balanced - floor(n/2) against ceil(n/2), the split cctbx's compute_cc_one_half uses and phenix.merging_statistics through it - and, because a rank is a property of the set rather than of the order it is walked in, it is the same on every path. A sequential "put it wherever the counts are more even" rule is balanced too but not that: the device holds the fulls in emit order and the host in frame order, and on three test crystals that alone moved CC_HALF between a CUDA and a JFJOCH_USE_CUDA=OFF build by up to 0.0077, with 4 to 12 shell rows differing. Both now agree bit for bit, on the whole scaling-and-merging section, while the radiation-damage B in the same report still differs between the builds - so the agreement is a property of the split, not of the two pipelines being identical. It is also stable across repeated runs and across -N 1, 4 and 12. Ranking is quadratic in a reflection's multiplicity - tens - so it is one multiplicity-weighted sweep of the observations, assigned once per pass and read by both the host loop and MergeAccumKernel. The kernel no longer decides anything, which is why the two cannot drift apart. CCANOM. SigAno was the only anomalous quality statistic reported, and it is a ratio against the error model: an optimistic sigma raises it, and it cannot separate real anomalous signal from an underestimated sigma. CCanom carries no sigma. For each acentric reflection the anomalous difference is formed twice, once per half, and the two are correlated over every pair where both hands split into two non-empty halves - per shell, and overall as one correlation rather than a mean of shells. The halves for it are balanced within each MATE, not within the reflection: a mate left entirely in one half loses the whole pair, and balancing per pair instead measured 0.223 where per mate gives 0.260. It hangs on the anomalous accumulation that already existed for the I(+)/I(-) export, which runs whether or not the merge is Friedel-averaged - so CCanom is reported on a default run too, and -A only changes the counting basis. The whole-mate sums SigAno and the reflection files read are the two halves added back together, so neither moves. Against external programs on the same observations: AIMLESS 0.264 and cctbx 0.266 where this gives 0.260, per shell within about a point. XDS's CORRECT.LP has a column named `Anomal Corr` which is NOT this quantity - two to three times larger in the low shells, with a total below every one of its own shells - so the report says so where it describes the key. On data with no anomalous signal and around two observations per mate the statistic is unstable and goes strongly negative; phenix reads -0.83/-0.45/-0.42 where this reads +0.12/-0.33/-0.41 on the same file, so that is the statistic, not this implementation, and it is reported as measured. REPORTING. A shell prints `-` where it could not measure a quantity instead of `nan` - which every run printed for CCref, and a multiplicity-1 shell printed for most of the row - and CC_ANOM is omitted from the report rather than written as a placeholder when no pair could be split in both hands. A dash and an absent key are the same statement; a measured value always prints, including a negative one. Reported CC1/2 moves, so baselines keyed on CC_HALF need regenerating, and the automatic resolution cutoff reads the same statistic and can move a shell edge with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 17:07:32 +02:00
leonarski_fandClaude Opus 5	0c3e1462ef	merge: the completeness denominator is the declared range, not the surviving one ... CalcPossibleReflections was handed d_min/d_max derived from the reflections that came out of the merge, so a loss at either extreme took the numerator and the denominator with it. At the high end that is right: d_min is the finest d reached anywhere and the denominator is the full sphere down to it, so anisotropic loss shows. At the low end it was a tautology - d_max was the coarsest reflection that happened to survive, so anything the beam-stop shadow mask (on by default), a detector mask or the low-resolution limit itself removed left the denominator along with the data and could not be reported as missing. Both statistics paths now bin, and count, between the DECLARED low-resolution limit and the finest d reached: MergeOnTheFly::MergeStats (stills) and RotationScaleMerge::MergeAndStats (rotation). The grid and the denominator keep sharing their bounds, so no possible reflection falls outside a shell. An undeclared low limit is the whole sphere - 1/d^2 down to 0 - spelled as an infinite d_max, which ResolutionShells already handles and which gemmi's for_all_reflections special-cases; the change therefore reads correctly whether or not the 50 A default stays. The innermost shell keeps a finite d_max label, falling back to the coarsest reflection measured when the bound is infinite. This makes the shell boundaries the ones the integration document already claims: XDS lays its nine 1/d^2 bins between INCLUDE_RESOLUTION_RANGE's two values, not between the extremes of the surviving data, and counts POSSIBLE against the declared low limit - which is why its innermost shell reports the beam stop's loss. Verified against a CORRECT.LP: all nine boundaries reproduce to the printed precision from the declared 50 A, and not from the coarsest observed reflection. Measured on stored merges of seven rotation datasets, small-molecule and protein, re-scaled with --mode scale: the overall denominator moves by 0 to 2 reflections out of 70,000-100,000, because on every one of them the coarsest reflection the declared limit allows was itself measured - the corpus has no dataset whose stop eats a whole low-resolution class. What does move is the shell grid: the innermost boundary shifts by 0.1-0.4% in d (e.g. 7.21 -> 7.22 A), which changes the innermost shell's counts by up to a few per cent and its R_meas by around 0.1 percentage points. Stored battery baselines for rmeas_lo must therefore be regenerated, not compared across this commit. Two decisions read merged completeness (the two-pass wrong-cell guard, which only fires above 100.5% and only under -S); a larger denominator can only lower the figure, so the guard can fire less often, never more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 16:39:40 +02:00
leonarski_fandClaude Opus 5	a0db11193a	docs: the low-resolution limit guards a depressed background, not the beam stop ... The comment, the usage text and the advanced docs all said reflections coarser than 50 A are "behind or beside the beam stop", which invites the conclusion that beam-stop masking makes the limit redundant. Measured, it does not. On raw images, the azimuthal background in the pixels the shadow mask leaves OPEN is 30-44% of the field value out to ~34 px and recovers to 95% of it only at d ~ 52 A: the stop suppresses air scatter well beyond the shadow it casts. A background ring inside that zone over-estimates the background, so the intensity comes out negative - 74% of the reflections coarser than 50 A on that geometry, against 2% in the shell just inside it, and they sit a median 15 px clear of the mask (reflections touching the mask are only 33% negative). The mask covers 47-79% of the area coarser than 50 A across three datasets, so the part that matters is exactly the part it leaves open. The zone is set by the stop's angular size, so it tracks wavelength rather than detector distance, which is what makes a fixed d limit reasonable. Freeing the limit on the most affected large-cell dataset in the corpus adds 1329 observations (+0.11%), moves I/sigma 7.01 -> 7.00 and completeness not at all, and stretches the declared low-resolution range from 46 A to 102 A on the strength of mostly-negative data. The advanced docs additionally told large-cell users to change the setting, implying recoverable data; corrected. "The value XDS configurations use" is also sharpened: every dataset here whose XDS.INP leaves INCLUDE_RESOLUTION_RANGE unset reports 50.000 in CORRECT.LP, so 50 A is XDS's own program default rather than a habit of our inputs. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 16:26:20 +02:00
leonarski_fandClaude Opus 5	b971f99b53	docs: audit of the rugnux and analysis pages against the code ... The one outright falsehood: the R-free convention paragraph in RUGNUX_INTEGRATION.md still described the pre-flip FreeR_flag numbering (0 = work, phenix/CNS) and told REFMAC5 users to pass FREE 1 - on a current file that keyword designates the 95% working set as free and REFMAC stops. The file has carried 0 = free (CCP4) since the flip; the paragraph now says so, the keyword is gone from the worked script, and a note keeps the old error message findable for files written before it. A contradiction within RUGNUX_ADVANCED.md: the --mode scale section said the mode never reindexes the reflections it writes, while the indexing- ambiguity section (correctly) said --mode scale --model does resolve a rotation ambiguity. The code adopts the model's frame before writing, so the former now agrees with the latter, and says what genuinely cannot be repaired: a stills _process.h5 integrated without a reference. Half-updated model-validation prose: the pages that predate the hypothesis gate still described --model deciding the enantiomorph and the indexing unconditionally. Every such statement (quick start, tutorial, the ambiguity table and bullets, the validation section) now carries the gate: the model decides nothing unless it beats the null of its own random placements, and the indexing choice must also beat the null's margin. The map names now say sigma_A-weighted 2mFo-DFc/mFo-DFc. Missing files: the tutorial's output-file list did not mention the --model outputs at all; it now lists the maps, the map-coefficient MTZ, the anomalous map and the placed model in both formats, and points out that <prefix>.cif is reflections while <prefix>_model.cif is coordinates. _model.pdb is added beside _model.cif everywhere the placed model is described, with the PanDDA/dimple reason it exists. Undocumented indexing behaviour: the axis-harmonic spot-count arbiter and the default-on short-axis second hypothesis existed only as changelog lines; CPU_DATA_ANALYSIS_INDEXING section 6 now describes both, and the --fft-min-unit-cell texts no longer claim a crystal below the 10 A floor cannot be indexed at all. Small corrections in RUGNUX_REPORT.md: sweep-quality warnings live in section 11, not 9; JFJOCH_DATASET_SETTINGS carries the poni_rot*_rad angles too whenever any is non-zero. The tutorial's post-refine sentence now states the real commit bounds (distance under 1%, beam within 15 px of the header or the run's own measured centre, whichever is nearer) instead of "restrained toward the header", which f98442077 made stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 15:48:14 +02:00
leonarski_fandClaude Opus 5	79a0b1358f	docs: keep the viewer's merge-plot axis fix in the changelog ... Collapsing the section dropped it as minor. It is small but it is the kind of thing a user notices and then wonders about - a CC1/2 curve drawn over the full height with tick labels covering only the top tenth of it - so someone who saw that should be able to find out it is fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:38:37 +02:00
leonarski_fandClaude Opus 5	ea07d7c9dd	docs: the rc.166 changelog collapsed into a release note ... The section had grown one line per commit, 38 entries, a development log. It is now twelve statements a user of rugnux or the broker can act on, each folding the lines that shared a user-visible consequence: - the model work is two entries (the hypothesis gate; the outputs, now including <prefix>_model.pdb) plus its report keys. - reading foreign data - miniCBF, Eiger 1.x, third-party NXmx, rugnux and viewer alike - is one entry. - lattice, indexing-on-the-true-cell, point group, setting, absences and the -S refusal are one entry; beam centre, tilt and the 2theta arm are one; the unmerged-MTZ lines are one; the reflection-file conventions (FreeR direction, HKL_base, _refln.status) are one. - the six report lines and both REPORT_VERSION bumps are one entry that says what the report carries and states the version once. - the calibration mode's .json, its refusal to return a non-measurement (non-zero exit) and --no-refine-tilt are one entry. - the recorded-metadata lines (direct_beam, incident_beam_size, peakCountUnfiltered, smargon.chi_deg) are one; the two documentation lines are one. Kept standalone, because holders of existing files need it: the snake-grid mirroring fix. Kept explicit inside their entries: the FreeR_flag direction and the DETERMINED_FROM_MODEL -> ASSUMED_FROM_MODEL rename. Dropped entirely: the goniometer-axis direction-vs-length line (internal consistency of the first pass) and the viewer merge-plot label fix; the null-replicate cost work, tests, regenerated python-client docs and review-fix internals never earned lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 15:38:27 +02:00
leonarski_fandClaude Opus 5	ab4a89e0a9	review fixes: the calibration accept test, a NaN warning, and two grid defects ... From the code review of the writer, API and geometry work. The powder calibration's re-bin accept test was one-directional. The best-of fold twelve lines above it is symmetric on purpose - a fit that is not a measurement never beats one that is, whatever its residual - but the final test applied the residual clauses regardless, so a CONVERGED re-fit could lose to a non-converged first pass on rms alone. That is exactly the motivating case: a first pass that declined the tilt and pinned the header's. It now matches the fold. The anisotropy warning seeded max_element and min_element on an array that can hold NaN, and every comparison against NaN is false, so both returned the same entry and the sentence named one direction twice with nan for both limits. The non-finite entries are skipped, and where fewer than two directions have a limit the warning states the anisotropy without naming one. The grid-scan plot added a SIGNED half-step to a grid laid out in |step|, so with a negative step every cell centre landed one whole step outside the extent - the same family as the snake-parity fix, in the same configurations. And one guaranteed no-op: /entry/MX/imageScaleFactor was written twice. Changelog: the short-axis pass had no entry at all, and the mmCIF one carried its rationale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:20:49 +02:00
leonarski_fandClaude Opus 5	b2c0a5ea48	model validation: review fixes - the run's output, the origin gauge, and two claims ... From two independent code reviews of the model-validation work. Nothing here changes a verdict: the acceptance set still reads +17.69 / +1.70 / -0.37 sigma and the rejected runs still write files byte-identical to a run with no model. THE RUN'S OUTPUT. Model validation runs BEFORE the reflection files are written, and two paths through it could throw: the null's replicates (rotated models fed to a scaling path that fails outright on data it cannot pair up - the adversarial input for it), and the mmCIF coordinate writer. Either would have taken the .mtz, .cif, .hkl and _unmerged.mtz with it, after the merge had already been paid for. A null that cannot be built is a question that could not be put, which is the NOT_TESTED state this design already has; a coordinate file that cannot be written is a lost convenience. Both now degrade instead of aborting. THE ORIGIN GAUGE. Translating the whole cell content along a free-origin direction - all three in P1, the unique axis in a polar group - leaves every |F| exactly unchanged. The code said the LM damping and the R-free gate made that harmless between them. Neither does: the gate is a function of |F| and is blind to exactly this, and the gauge column of the Jacobian is not zero but noise divided by the difference step. It is now projected out after every zone, against the group's own common fixed subspace. P2_1 alone is a large share of deposited structures, and the reported shift was partly fiction in every one of them. TWO CLAIMS THAT WERE FALSE. The report told the user R-work carries the decision "because nothing was refined against it", six lines from where six placement parameters are refined against it; the real argument is that the null is placed the same way, so the optimism is common-mode and cancels. And the constant's own comment quoted a +4 vs +1 sigma gap where the measurement is +17.7 vs +1.7. Also: the map file's phase columns are back to [0, 360), the convention they carried before sigma_A weighting; with no data space group the map file follows the reflections into P1 rather than taking the model's group; the sigma is floored against a near-zero null spread rather than only an exactly-zero one; the model is restored whether or not the solver reported a usable answer; and the zone list is the ladder walked rather than the ladder planned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:18:55 +02:00
leonarski_fandClaude Opus 5	63a2f6744a	docs: say what the MTZ base-dataset fix actually changed, and what it did not ... The page said the fix makes mtzinfo and mtzdmp report the collection wavelength, which implies mtzdmp was wrong before. It was not. Verified across 226 files from two battery arms, one built before the fix and one after: every pre-fix merged MTZ reports 1.54187 under mtzinfo and the true wavelength under mtzdmp, truncate, ctruncate, gemmi, iotbx and phenix.xtriage. No program was found whose output or behaviour differs between the two files - cad silently repairs the layout on the way through. So the fix buys a conformant file and a correct mtzinfo line, not a rescued phasing run, and the f'/f'' consequence is the risk it removes rather than a measured effect. Also records that <prefix>_unmerged.mtz still reads 1.54187 under mtzinfo and is not a regression: its columns sit on HKL_base deliberately, which is what POINTLESS expects, and the per-batch wavelength AIMLESS and POINTLESS actually read is correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:49:10 +02:00
leonarski_fandClaude Opus 5	28a58e94b9	model validation: write the model as it was placed ... `--model` re-fractionalizes the model into the data cell and then places it as one rigid body, but the placed coordinates never reached disk. On a lysozyme sweep against a non-isomorphous deposited model the move is 3.058 deg and 1.035 A, so a user overlaying their input model on rugnux's maps was out by exactly that, and no file on disk corresponded to the maps at all. `<prefix>_model.cif` is that file: the input's chains, residues, ligands, waters, B-factors, occupancies and anisotropic Us, at the coordinates the maps were computed from. `<prefix>.cif` is already the merged reflections, hence the suffix. The cell and space group come from the same two values WriteReflections is given - the unit cell and DiffractionExperiment::GetSpaceGroupOrP1() after AdoptModelFrame has settled the enantiomorph - so the coordinate file and the .mtz beside it always agree. Taking them from the input model would not: with data merged in P4(1)2(1)2 and a P4(3)2(1)2 model, the written reflections take the model's group, which is neither the data's original label nor, when the model is rejected, the model's own. Written whenever the maps are, not only where the rigid-body step was committed. The model is re-fractionalized and may be relabelled whatever the placement decided, so an unmoved model is still not the input file; and a model the null rejected is scored, placed and mapped like any other - the negative result, and the case where the density is most worth looking at. Nothing in the tree could write coordinates: gemmi_gph declared to_mmcif.hpp but src/to_mmcif.cpp had been trimmed from the vendored subset. Both it and to_pdb.cpp (to_mmcif.cpp calls its use_hetatm) are vendored from the same gemmi 0.7.5 the rest of gemmi_gph comes from, unmodified, and every header they include was already there. Same package, same MPL-2.0, same LICENSE.txt already collected into licenses/gemmi.txt and already listed against `gemmi_gph/` in THIRD_PARTY_NOTICES.md, so no new row and no new licence text. Verified end to end: read back with gemmi the file differs from the re-fractionalized input by exactly the reported 3.058 deg / 1.035 A with 0.0000 A rms about that rigid move, and REFMAC5 at zero cycles against rugnux's own .mtz starts at R-free 0.3667 where rugnux reports 0.3826 - against 0.5832 for the unplaced input model, where rugnux reports 0.5911 before the placement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:47:38 +02:00
leonarski_fandClaude Opus 5	8a0bbae3c4	model: nine null replicates, now that they cost the slowest and not the sum ... The verdict is (real - mean)/sd of the null sample, so its reliability is set by how well the SPREAD is pinned, not the mean. The relative error on an sd from n draws is 1/sqrt(2(n-1)), and a sample that happens to come out narrow is exactly what turns a model that does not fit into one that appears to. Measured on the case that sits nearest the gate - an unrelated protein, 1.7-1.8 sigma against a threshold of 3 - the chance of it reading over the gate on a different seed is 31% at n=3, 17% at n=5 and 6% at n=9. Nine is affordable only because the replicates now run concurrently: the null costs the slowest of them rather than their sum, and the slowest of nine is barely above the slowest of five. Measured end to end on the accepted case, 5.98 s against 6.46 s for five - inside the run-to-run scatter, so the four extra replicates are free. Verdicts unchanged, on more evidence: correct model ACCEPTED at +17.69 sigma (+14.96 at n=5), an unrelated protein REJECTED at +1.70, the same model rotated 90 degrees REJECTED at -0.37. Both rejected runs still write .mtz, .cif, .hkl and _unmerged.mtz byte-identical to a run with no model at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:47:10 +02:00
leonarski_fandClaude Opus 5	53a8c428c3	model: run the null's replicates at once ... The replicates are independent by construction - the same model under a different random rotation, scored the same way, nothing flowing between them - so each takes a copy of the model and of its structure factors and they run concurrently on the run's own -N budget. The real model is not moved by any of them, which also drops the save/restore/recompute the serial loop needed to put it back. Measured on an idle machine, decision-pending --mode scale --model: the null 10.0 s -> 2.5 s (median of 5), the whole run 13.9 s -> 6.5 s. The speedup saturates at 4.0x, not 5x, because the replicates cost 1.58-2.31 s each (how many placement evaluations an orientation needs) and the null now costs the slowest one; sum/max on the serial run is 4.13x, so 97% of what is there to get. More threads than replicates buy nothing. The rotations are drawn up front, in order, from the same fixed seed, so replicate i gets the same orientation whatever order the threads run in. A verdict that depended on the interleaving would not be a measurement. Verified: every reflection file, map, map-MTZ and report is byte-identical to the serial build's, on the accepted, both rejected and both no-decision cases, and identical again with -N 1. Thread safety was read out of the vendored gemmi source, not assumed. Each replicate owns its DensityCalculator, SolventMasker, Scaling and Structure; their statics are the const IT92 tables (constant-initialised, never written on these paths), pocketfft is built with POCKETFFT_CACHE_SIZE 0 and POCKETFFT_NO_MULTITHREADING so it holds no plan store, Scaling's Levenberg-Marquardt is stack-local per call, and nothing on these paths writes into the Structure it was handed. Ceres defaults num_threads to 1, so the rigid-body solve adds no threads of its own. Peak RSS is unchanged - the merge, not the null, is the high-water mark. Five concurrent replicates add 83 MB over the serial null on a 78x78x37 A cell at 1.56 A, ~17 MB each, which is the two grids at d_min/(2*rate). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:41:28 +02:00
leonarski_fandClaude Opus 5	0eb9fb8a8b	docs: the phenix label line, and why the MTZ carries no DANO ... Two measurements, neither of which changes what we write. phenix's "Multiple equally suitable arrays" is a tie between the two INTENSITY arrays, IMEAN and I(+)/I(-); iotbx scores F and F(+)/F(-) below them, so writing amplitudes as well is not what causes it. ctruncate's own output ties in the same place, so this is what phenix does with a CCP4 merged file rather than something rugnux does to a user. Column order, dataset and project names, and dropping the amplitudes all leave the tie exactly where it was; only removing one of the two intensity arrays clears it, and removing the Bijvoet columns would take the SHELX route with it. So the file stays as it is and the page now carries the label line per program, both formats, including the quoting the anomalous one needs. A program that asks iotbx for anomalous data by preference - hyss, find_peaks_holes, molprobity, the autosol import - needs nothing at all, which is now said as well. The page also said the MTZ form of that message names no choices. It names both, exactly as the mmCIF form does; corrected. DANO/SIGDANO stay out. Against a ctruncate file, DANO is F(+)-F(-) and SIGDANO is the quadrature sum of the two sigmas, bit-identical on every reflection, so the pair is a restatement of columns we already write - and the quadrature sum is the convention whether or not the mates share a scale model. Every consumer in the documented routes takes the Bijvoet columns directly, CCP4's own bp3 and afro ask for them in preference to F/DANO, and adding the pair costs 12.5% of the merged file while changing nothing phenix or Phaser sees. fft's anomalous Fourier is the one caller with no other spelling; the ctruncate command that makes it is now on the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:12:45 +02:00
leonarski_fandClaude Opus 5	4af23e9b27	model: build the null only where the model claims something ... The null costs about 12 s - five replicates, each a full fit and a rigid-body placement - and it gates exactly two decisions: the enantiomorph label and the merohedral indexing. A model that claims neither, already in the group the data were merged in on a crystal with no indexing ambiguity, has nothing for the null to arbitrate, and paying for it there is 12 s spent gating a decision nobody is making. That is the isomorphous fragment-screening run - hundreds of crystals of one form against one apo model, a map wanted seconds after the last image - and it is the case that has to be fast. So the null runs only when a decision is pending: an enantiomorph candidate exists, or the indexing probe returned a winner that is not the identity. The probe is a handful of scaling fits and had to run first anyway, so the question is answered before the expensive part starts. Measured on a rotation dataset, --mode scale --model, interleaved on a loaded machine: the isomorphous case is 3.6-4.7 s before this whole line of work and 3.8-4.7 s with the gate, against 16.5 s while the null ran unconditionally. The case that still pays is unchanged at 17 s, and the three wrong-model cases give bit-identical verdicts and bit-identical files to before. The rigid-body refinement of the replicates stays: 3 of 5 commit a placement, so without it the null would score a weaker procedure than the one it judges. MODEL_FIT gains a third value, NOT_TESTED, alongside ACCEPTED and REJECTED - the question was never put, which is not the data answering it badly - and the MODEL_FIT_NULL_* and MODEL_FIT_SIGMA keys are absent with it rather than reporting a null that was never built. REPORT_VERSION stays 6: MODEL_FIT is itself new in this release, so its vocabulary has never shipped and a consumer cannot be relying on the two-value form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:06:29 +02:00
leonarski_f	c999883eb1	model: --model is a hypothesis, and it decides nothing until it fits ... A model supplied with --model rewrote the space group of every reflection written out on the strength of its file having parsed. Measured on a rotation dataset merged in P4(1)2(1)2: an unrelated protein and the correct model rigidly rotated 90 degrees each produced a .mtz, .cif and _unmerged.mtz byte for byte identical to what the crystal's own model produced - relabelled P4(3)2(1)2 - at R-free 0.601 and 0.674 against the correct model's 0.591, with no warning. The adoption was pure space-group-number arithmetic and ran before the model had been fitted at all. A model changes exactly two things on the rotation path, and both rewrite the data: the enantiomorph label and the merohedral indexing. Both now wait for the fit. Everything else a model produces - R-factors, maps, the rigid-body placement - is a statement about the MODEL, cannot corrupt a reflection, and is computed and reported either way. The gate is not a threshold on R, because no threshold works: the classical acentric random value is 0.586 at unit scale but 0.550 at the R-minimising scale, observed nulls land at 0.599-0.615, and the value moves with the model's atom count and B-factors as much as with the data. Instead the same model is re-oriented at random about its own centroid five times and run through the identical path - same scaling, same rigid-body placement, same R - and the real fit is asked how far above that distribution it sits. R-work carries the decision: nothing is refined against the working set here, and it has 12615 reflections to R-free's 709. Measured on the case above: the crystal's own model +15.0 sigma, the unrelated protein +1.8, the 90-degree rotation -1.0, and the two rejected runs now write files byte-identical to a run with no model. The nulls are rigid-body refined like the real fit, or the comparison would be between a placed model and unplaced nulls. That is what the null costs: about 12 s for the five replicates, on a --mode scale run that merges in 2 s. The merohedral margin gets the same treatment - a random placement also picks a winner, and measured, by a comparable lead - and the candidate operators are now enumerated from the DATA's space group. Taking them from the model's enumerated zero operators wherever the two groups differ, which is exactly the case the probe exists for: a model in P4(3)2(1)2 against data merged in P4(3) probed nothing at all, and now probes the twin law. The anomalous difference map is the only measurement here sensitive to the hand - inverting the model through the origin moves R-work by less than 1e-4, since |F(h)| of the inverted structure is |F(-h)| - so where it says the hands disagree it vetoes the adoption outright, fit or no fit. Report: MODEL_FIT, MODEL_FIT_SIGMA and the null beside it, MODEL_DECISIONS_TAKEN, the indexing margin against its null, and MODEL_VALIDATION= PERFORMED as the counterpart of the failure line. SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL becomes ASSUMED_FROM_MODEL and is written only where the model was accepted: nothing here measured the hand, the model asserted it. That is a reason code changing name and meaning, so REPORT_VERSION is 6. Stills are untouched: the per-image indexing hand a model can set at integration time is not reachable on the rotation path and is not gated here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 13:34:52 +02:00
leonarski_fandClaude Opus 5	d5cf3a6e58	docs: worked Phaser and SHELX routes, and the MTZ base dataset they need ... Both programs were run on merged rugnux output and the commands are the ones that worked, error messages included. Phaser takes myrun.mtz with no LABIN, picking IMEAN/SIGIMEAN and the cell, group and resolution out of the file; MODE MR_AUTO already searches both hands of an enantiomorphic pair and returned the hand opposite the one in the header on a tetragonal test. SGALTERNATIVE SELECT ALL covers the screw variants the report lists as indistinguishable, MODE CCA prints the list without searching, and a wrong point group has to be re-merged instead. mmCIF is refused by 2.8.3 in both the CCP4 and the phenix build; gemmi cif2mtz gives the same solution. SHELXC reads only myrun.hkl - it refuses an MTZ and exits 0 while doing it - and needs CELL and SPAG repeated, since HKLF 4 carries no metadata. A default rotation merge already writes the Bijvoet split, so no flag is needed and --no-export-unmerged hides nothing. On a 5 keV cubic sweep SHELXD separated a space-group pair the merged intensities could not (CFOM 69.4 against 52.0) and SHELXE solved it, 42.9 % against 15.3 % autotrace CC between the two hands; a tetragonal sweep at lower completeness and multiplicity did not separate at all, and that is recorded too. Verifying it exposed one defect. The merged MTZ was written with its data dataset at id 0, the id MTZ reserves for HKL_base, so mtzlib dropped the wavelength and mtzinfo reported the 1.54187 A default - the wrong edge for anything taking f'/f'' from the file. Writing HKL_base first, as the unmerged writer already does, puts the real wavelength back; Phaser's solution is unchanged either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 13:07:03 +02:00
leonarski_fandClaude Opus 5	6b738713e8	model validation: place the model, then weight the maps by sigma_A ... --model re-fractionalized the model into the data cell and then left it there. On a non-isomorphous pair that is a placement error, not a cell error: the box is squeezed, the body inside it is not moved. Six parameters now recover it - an angle-axis rotation about the model's centroid and a translation, refined over a 6 / 4.5 / 3.5 A ladder, the scale (k_overall, anisotropic B, k_sol, b_sol) re-fitted at every evaluation so the target measures the placement and not the scale. The refinement sees only the working reflections and the step is committed only if R-free, on the free set it never saw, drops; otherwise the model goes back where it was read. Measured on merged lysozyme data against a deposited lysozyme model whose cell differs by 3.4% in c: R-work 0.559 -> 0.400, R-free 0.591 -> 0.383. Over the same 3.5 A range the external arbiter (REFMAC rigid body through dimple) works in, 0.524 -> 0.330 against REFMAC's 0.522 -> 0.355, and the recovered movement agrees with REFMAC's to 0.25 deg and 0.03 A (3.05 deg / 1.04 A vs 2.76 / 0.98). 2.4 s of added wall clock, 234 structure-factor evaluations. The map coefficients become 2mFo-DFc and mFo-DFc. sigma_A is estimated by maximum likelihood per resolution shell on the free reflections only, with the number of shells taken from the size of the free set so no shell is thin; centric and acentric reflections carry their own likelihoods, and a centric reflection's bias-free coefficient is mFo. Cross-checked against CCP4 SIGMAA on the same reflections: mean FOM 0.404 against its 0.396, with the same per-shell structure. The figure of merit is written to _maps.mtz so the weighting can be undone. The per-shell scaling refusal in fit_model stands - Fobs is never rescaled and the R-factors are untouched - but m and D are per-dataset, so maps from one campaign are no longer scaled identically. That is argued at the code and in docs/CPU_DATA_ANALYSIS_DECISIONS.md 14.4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 13:06:05 +02:00
leonarski_fandClaude Opus 5	5cc2f811b3	indexing: an axis harmonic is a small multiple, and the Wilson B is XDS's ... Two unrelated notes, one on each side of what the merge reports. The axis-harmonic precondition accepted any near-integer volume ratio. Its window is absolute in a unit integer spacing, so a ratio between two UNRELATED lattices passes it about a third of the time whatever the multiple is, and at 124x the pair is not a cell and its harmonic in any sense. That branch was harmless while the rule was "always take the smaller cell"; deciding the pair on the evidence makes it reachable, so it is now bounded at 4. Measured over the corpus: of 44 firings, all 36 at n <= 4 read 1.1 to 51.3 points BELOW the chance occupancy (n-1)/n, so an index-n sub-lattice really exists in each of them; all 8 above it - two datasets, both decided by 8 to 34 sigma, so not a margin problem - read within 7.8 points OF chance, so none does. The largest n at which a real sub-lattice was ever seen is 3, and no dataset moves either way. Separately, WILSON_B and _reflns.B_iso_Wilson_estimate are fitted to log<I> directly, which is what XDS's Wilson line does and is not what TRUNCATE, ctruncate or phenix.xtriage do - they divide out Sigma = sum f^2(s) first. Leaving Sigma in the slope inflates B by 2-8 A^2 on our own merges, and against those two programs on the same files this number runs 10-36 A^2 high, in the same direction every time. The docs said "the analogue of XDS's Wilson-line B", which is accurate but easy to read past; they now say the two conventions are not comparable and which one this is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 13:03:25 +02:00
leonarski_fandClaude Opus 5	476a849c0e	indexing: decide an axis harmonic on which cell explains more of the spots ... When two first-pass schemes return cells whose primitive volumes differ by a small integer, the validation-FRAME count cannot tell them apart: a spurious axis multiple indexes every frame its true sub-cell does, so both reach 60/60 and the count saturates. The rule that then decided the pair was unconditionally against the larger cell, so on a crystal with a real pseudo-translation the true cell could not win in any scheme order. Ask the same question at the granularity where it does not saturate: how many of the validation frames' SPOTS does each cell account for? That comparison leans towards the smaller cell by construction, and needs no threshold to do so. Acceptance is a fractional-Miller test, so multiplying an axis by n multiplies that axis's residual by n: the larger cell places every shared reflection n times less accurately than the sub-cell does, and loses outright the spots that sit in the tolerance margin. The only thing that can pay for that loss is the class of reflections the larger cell ADDS - empty for a spurious multiple, the superstructure's satellite rows for a real one - so the larger cell wins the count only when the extra periodicity is really there. The count is taken over the validation frames' whole spot lists, which reach far deeper into each frame's intensity distribution than the first pass's own accumulation cap, and a superstructure layer is faintest exactly where that cap cuts. Measured over the five crystals of this corpus where the two schemes return an integer-related pair, the larger cell accounts for 1.37x and 1.98x the spots on the two whose true axis was being halved, and 0.30x, 0.36x and 0.71x on the three where the doubling is spurious. All five come out right: the two keep the true cell and its deposited space group, the three reproduce the answer the old rule gave, to the digit. (One of the three has a bistable first pass - four builds give three answers, one of them without this change at all - so it is not evidence either way; the other two are reproducible.) What the added class holds is computed and reported next to the decision, because it is the physics the count is a consequence of. It is deliberately NOT thresholded, and that is the part of this that took the measuring. Refuted along the way: - An occupancy floor, which is how this was first written. Over the five crystals the arbiter is asked about, the emptiest index-n class reads 48.2, 41.6, 37.3, 25.4 and 7.1 %. The two the larger cell should win are the 41.6 and the 37.3, so the three it should lose bracket them on both sides, and a real superstructure elsewhere on the corpus reads 3.4 %, below all five. Recomputing the same statistic on the merged intensities over a sweep of I/sigma cuts leaves the ordering unchanged, so this is a continuum and not two populations: no floor separates them, and no amount of extra data would. The bimodality a floor needs was an artefact of a calibration set that contained no failure. - Requiring the two cells to stand in a genuine sub/super-lattice relation, the change of basis being integral. Measured, all five pairs are index-n relations to within 0.016 of an integer - the volume ratio is not the weak link. - Deciding it on the merge, by integrating and merging both cells. The worst failure does announce itself there (CC1/2 0.9994 -> 0.9566, ISa 18.6 -> 1.4), but it costs a second full integrate-and-merge, and the successes lose 13-30 % of their ISa where another failure loses 27 %, so the metric does not separate them either. - Requiring the sub-lattice class to be the STRONGER of the two, which is the right mechanism but the wrong observable at this point in the run: the sign it turns on lives in integrated intensities, and the spot finder reports no spot at all where a class is absent, so at first pass the same ratio reads 0.90 against 0.39 and 0.27. The ordering survives, the sign does not. The occupancy is maximally wrong on the worst failure because that cell is not a superstructure at all. A beam-centre error along the spindle translates the derotated cloud rigidly, and a lattice shifted by half a spacing is indexable only on a doubled axis - the shift needed scales as 1/(2L), so a long axis is the easy one to half-offset. In that doubled setting the even class is empty and the zero layer reads a negative mean intensity, which no crystal can do, while the odd class carries everything. Such a cell fits no index-n sublattice at all, so its added class reads the chance value (n-1)/n, the largest the occupancy can take, and the occupancy test reports the artefact as more real than any genuine superstructure. The spot count sees it for what it is, at 0.30x. So the beam-centre warning below is now suppressed only when the larger cell WINS. Declining it is a fall back to the default, and that warning - which names the beam centre and offers --estimate-beam-center - is then the most useful thing the run can say; on the half-offset mode it is the correct diagnosis. The same question asked unconditionally of the committed cell, the halved-axis probe, is not included. Measured over the committed cells of 29 crystals, 19 of 189 axes read above the 2 % floor it would have used and the largest read 30 %, and those largest readings are on crystals whose committed cell is already wrong - where doubling an axis is the wrong response. It rescues one crystal whose superstructure layer reads 3.4 %. That is not worth the rest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 12:46:57 +02:00
leonarski_f	54f2de31bb	grid scan: say how the stationary spindle angle is stated, and pin it ... A grid scan is a set of stills at a stationary spindle, and the angle it stood at is what relates one grid to another taken elsewhere on the circle. Users were finding an all-zero omega in the file and concluding the angle could not be recorded at all. It already can, and has since the goniometer and the grid scan stopped being alternatives: send the axis with step 0 and its start angle, and that angle is written per image into the NXmx sample chain, read back by reader/, and taken by dials.import as a set of stills. Measured on a generated 12-image grid: the placeholder file carries omega = 0 x 12, the same file with the axis sent at step 0 carries omega = 90 x 12, and dials.import reports "still: 1, sweep: 0" for both. Nothing in the code needed changing, so nothing was; what was missing was that nobody could tell, and that no test held the behaviour down. So: the API and the HDF5 documentation now say it in as many words, and three tests pin the three legs the value crosses - the OpenAPI request (which used to drop the grid scan whenever an axis was present, unpinned until now), the CBOR start message, and the file round trip. Also corrects a claim two comments and the HDF5 page were making. NXmx can express "no rotation" perfectly well - a sample may depend_on "." - so the placeholder is not there for the standard's sake. It is there because dxtbx cannot read a sample chain of translations alone: strip the rotation axis from a grid scan master and dials.import dies in get_dxtbx_goniometer with a matmul dimension mismatch. Recorded so nobody removes the placeholder on the strength of the standard. One thing the change does not fix, because it cannot: a stationary angle is invisible to DIALS when a grid scan is present. dxtbx picks the first varying axis as the scan axis, which is a grid translation, so the oscillation reads (0, 0); and with exactly one rotation axis in the chain it builds a single-axis goniometer whose fixed rotation is the identity, never consulting the angle. The same angle IS visible when it is the only candidate (oscillation reads (90, 0)) or when a Smargon head puts a second rotation axis in the chain (the setting rotation then carries it). The value is in the file and correct either way.	2026-09-02 11:04:29 +02:00
leonarski_fandClaude Opus 5	92a615085d	grid scan: the snake reverses on the acquisition row, not the display row ... GetElementPosFast_step took the snake parity from GetElementPosSlow_step, which is the DISPLAY row: it is the acquisition row r = image / n_fast, flipped to (n_slow-1) - r when the slow step is negative. So for a negative slow step the parity it hands back is parity(n_slow-1) XOR parity(r), and with an even n_slow that is inverted on every row - the whole raster comes out mirrored along the fast axis. An odd n_slow leaves it correct, so the same scan collected with 20 or 25 images disagreed about where image 0 sat: n_fast=5, fast +1.5 um, slow -2.5 um, snake on gives images 0..4 at fast index 4,3,2,1,0 with 4 rows and 0,1,2,3,4 with 5 rows. A positive slow step was correct at both counts, and so was every non-snake configuration. Snake means the stage reverses direction on alternate rows in acquisition order, so the parity has to come from the acquisition row. Taking it from image_number / n_fast directly makes the fast index independent of the slow axis and of the row count, and drops the call into the display-row function that caused the coupling. vertical_scan only relabels which axis is fast, so it was wrong in exactly the same way and is fixed by the same line. Affected files: written by an affected build, with snake on, a negative grid slow step (step_y for a horizontal scan, step_x for a vertical one), and an even number of rows. Their /entry/sample/transformations/grid_scan_x or _y is mirrored along the fast axis, as was the grid map in the frontend and the viewer - both mirrored together, which is why neither showed it. Tests: the interaction of snake with the step signs was never asserted, only each in isolation, so add a table over snake x {+,- fast step} x {+,- slow step} x {even, odd row count} x {horizontal, vertical} asserting positions, plus a case running one affected configuration through GetXContainer_m / GetYContainer_m and Rearrange. Every pre-existing assertion is unchanged and still passes; only the four negative-slow, even-row cells of the product move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:57:15 +02:00
leonarski_fandClaude Opus 5	3be79fb1b6	report: name the direction the anisotropy warning is about ... The warning said the diffraction limit "runs from 3.34 to 2.45 A depending on direction" and stopped there, so a reader was told the crystal is anisotropic and given no direction to act on. It fires on 19 of 113 datasets in the battery. The eigenvectors are already measured and already written to the merged mmCIF as `_reflns.pdbx_aniso_B_tensor_eigenvector_N_ortho`; they had simply never reached the human report. The tensor is fitted on s = frac.mat * (h,k,l), so in that Cartesian frame a*, b*, c* are the rows of frac.mat and naming the axis is one dot product per eigenvector. The label is exact in every Laue class that has a free tensor direction except triclinic, and the cosine is printed so a loose fit shows as one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:53:31 +02:00
leonarski_fandClaude Opus 5	7b1caa6ce5	mmcif: the free set is _refln.status = f, not a column of our own ... The merged mmCIF carried the R-free flag in `_refln.status_free` as 1/0 and wrote `o` into `_refln.status` on every row. `_refln.status_free` is not in the PDBx/mmCIF dictionary - it is absent from mmcif_pdbx v4.0, v5.0, v5.288 and v5.362, from CCP4's and phenix's shipped copies, and its wwPDB item page is a 404 - and no deposited structure-factor file uses it. `_refln.status` is the item that carries the free set, with `f` for a test reflection and `o` for a working one; on a deposition that also carries `_refln.pdbx_r_free_flag` the two agree exactly. Measured on a real merged file this run wrote: CCP4 cif2mtz refuses the file outright - "Unexpected context type for category REFLN" from its dictionary-validating parser, exit 1, a 12-byte truncated MTZ. Dropping the non-dictionary column is what fixes it: the same file without it converts. gemmi converts, but its cif2mtz spec knows only `status` and `pdbx_r_free_flag`, so FreeR_flag comes out 1 everywhere and the free set is silently lost - R-free would then be computed on the working set. phenix worked, but only by a filename heuristic matching the words "status" and "free". Writing `f` while keeping the extra column is worse than either, because phenix then finds two candidate free-set arrays and refuses the file, so the column goes in the same change. After it, all three read the same 5% test set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:53:31 +02:00
leonarski_fandClaude Opus 5	90d0d3c3f9	docs: the python client reference, for the two schemas that gained fields ... docs/python_client/docs is generated from the API spec and copied in by update_version.sh, so it goes stale between releases. The calibration convergence gate and the beam size both added properties without it; this is the same generator run those commits should have carried. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 10:35:12 +02:00
leonarski_fandClaude Opus 5	aa6073d41f	api: chi is a reported angle, not a travel limit ... The Smargon at the SLS 2.0 MX beamlines reports its own axis positions with readout noise, so a chi parked at zero comes back as about -1e-7 degrees. The schema bounded chi_deg to [0, 90], so an ordinary "chi is at zero" setup was refused - and the same holds at the other end of the arc, where a chi parked at 90 reads just above it. Both end-stops are exactly where a static positioner is left, so widening the range would only move the problem. The bound bought nothing. Chi never enters any computation: OpenAPIConvert puts it in SmargonPosition, DiffractionExperiment::BuildTransformationChain hands it to a rotation transformation verbatim, HDF5NXmx writes it and HDF5MetadataSource reads it back. No downstream reads its sign, and a rotation is defined for any angle. phi, the sibling angle in the same object with the same semantics, has never been bounded. What was left was a restatement of a hardware travel limit that the goniometer enforces itself, and its only observable effect was to refuse a value the instrument genuinely reported. It was also not enforced where a server-side check would matter: the cpp-pistache-server generator does not recurse into a nested object model, so Dataset_settings::validate never calls the Smargon model's. The rejection was raised by the generated python and TypeScript clients, which do check. Regenerated the C++ server model, the TypeScript client and redoc-static.html; python-client is gitignored and comes from gen_python_client.sh. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:33:30 +02:00
leonarski_fandClaude Opus 5	8a04773d1d	api: record the beam size at the sample, and write it where NXmx puts it ... dataset_settings gains beam_size_x_um and beam_size_y_um, the horizontal and vertical size of the X-ray beam where it meets the sample. They follow the same route total_flux takes - OpenAPI, DatasetSettings, the CBOR start message, the HDF5 master, and back out of a stored file - and nothing consumes them; this is metadata a beamline can state and a downstream program can read. NXmx puts this in the application definition rather than the base class: not NXbeam's extent (rank 2, nP x 2, per scan point, always FWHM of a rectangular aperture) but NXmx's own incident_beam_size, a recommended rank-1 two-element array in the order x, y. Both are live and neither is deprecated, so the choice matters; the MX definition wins in an MX file. Written as one array with a units attribute of "m", like every other length in the master, so the settings hold micrometres and FillMessage converts once. The unit table of ReadLength_m becomes LengthUnitFactor so the array read can share it: a master written elsewhere may state this in millimetres. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:59:34 +02:00
leonarski_fandClaude Opus 5	7890149ad6	writer: peakCountUnfiltered in the master, like its four siblings ... /entry/MX/peakCountUnfiltered was written only by the data-file plugin. The four other per-image spot counts are written to both the data files and the master, so a reader holding just the master got every count except the unfiltered one. It was not only a missing dataset. HDF5MetadataSource already reads /entry/MX/peakCountUnfiltered from the master and falls back to /entry/MX/nPeaks when it is absent - and nPeaks is the number of spots *stored* for an image, i.e. after the spot budget truncates. On a VDS master the fallback therefore substituted the post-filter count for the unfiltered one silently, and the two differ precisely on the images that hit the budget. EndMessage::spot_count is the right member: it and the data-file plugin's spot_count_total both come from DataMessage::spot_count (the end-message copy via ScanResultElem::spot_count), and CountSpots() sets that from spots.size() before FilterSpotsByCount() applies the budget - "spots found before filtering", as docs/HDF5.md already described it. The CBOR end block already carried spot_count on both encode and decode, so no message or protocol change was needed. SaveVectorIfMissing keeps the NXmxIntegrated case correct, where the data-file plugin has already written the dataset into the same file. Verified by writing files in all three formats and dumping them: the new dataset appears in the legacy, VDS and integrated masters and matches its sibling peakCountLowRes in value, datatype and read-back precedence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:37:53 +02:00
leonarski_fandClaude Opus 5	ef5da29319	calibration: a fit that hands back the file's own tilt is not a calibration ... A powder calibration is run because the file's geometry is in doubt, so a fit that quietly returns part of that file has answered nothing - and it is indistinguishable from one that worked, down to the residual and the sigmas arranged around it. On one of four LaB6 exposures of one detector the tilt came out at 2.93x its own sigma, a hundredth under the significance gate, so it was declined and pinned - at the master's hardcoded rot1 -0.08, rot2 -0.22 deg. That is eight times the tilt just refused, on no evidence, and worth 10 px of PONI at 190 mm. rugnux printed it to four decimal places, wrote the .poni, and exited 0. Judge the result on provenance instead of on any residual: a geometry is a measurement only if every parameter in it came from this data. Two ways out of the fits do not qualify - a covariance that never conditioned, so the fit cannot say what it determined, and a declined tilt pinned at a non-zero value from the file. A declined tilt over a file stating no tilt still qualifies, because reporting no tilt is then exactly what was measured; so does --no-refine-tilt, because a hold that was asked for is a stated choice and not a silent substitution. No single number separates the four. rms is 2.465 px against 1.44-1.64; the significance of all four lies between 2.93 and 4.47, so the gate is nearly a coin flip at these distances and moving it would only recalibrate on one population; and the failed fit has the TIGHTEST parameter sigmas of the set, because pinning the tilt removes the tilt/centre correlation that inflates a good fit's. The spot cross-check reads 13.5 px against 0.98-2.66, but 10.4 px of that is the pinned tilt moving the PONI - the same defect one step downstream, not independent evidence. On a failure rugnux says so, writes no .poni - a PONI file states where the detector is and has no field in which to say it does not know - writes the JSON with converged false and the reason beside it, and exits non-zero. The re-binning pass now prefers a converged refit over a non-converged one whatever its residual, so a tilt an earlier pass measured is not what a later one gets pinned at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 09:37:46 +02:00
leonarski_fandClaude Opus 5	4bb3d44983	viewer: label the merge plot over the range it is drawn on ... The merge-statistics window asked for an absolute y-axis - CC1/2 and CCref on 0..100, everything else from 0 - by setting the range on the chart's value axis after JFJochSimpleChartView::UpdateData had already built the chart. The visible tick labels are a separate QCategoryAxis whose entries UpdateData had generated from the range of the data, and those entries were not rebuilt. So the plot was drawn over 0..100 while the labels down its left edge covered only 91.5..100 and crowded into the top tenth; the numbers matching the drawn range appeared only on the right-hand grid axis, which the same call had made visible. CC1/2 showed it worst because its range is the narrowest, but every metric was affected. Give UpdateData the range instead, so the ticks and both axes come from one number. The side-panel azimuthal-integration chart passes no range and renders pixel-identically. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:23:02 +02:00
leonarski_fandClaude Opus 5	34e1ec8fb9	docs: keep the analysis landing page and drop a trailing transition ... The four-way split leaves CPU_DATA_ANALYSIS.md as the landing page the toctree points at, and one part ended on a horizontal rule, which docutils refuses at the end of a document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:20:27 +02:00
leonarski_fandClaude Opus 5	889c9b6cbb	docs: one changelog line for the rugnux documentation work ... Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	67bf380401	docs: credit DENZO/SCALEPACK and MOSFLM for the 2D-then-merge architecture ... Both programs were acknowledged for specifics (profile-fit variances, FFT autoindexing, post-refinement practice) but not for the paradigm rugnux's rotation pipeline is built on: integrate each image in 2D, then combine the partials into fulls, as against XDS's 3D profiles. One paragraph names it; the citations were already on the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	ee4c23b9a2	docs: the phenix label override, for the mmCIF as well as the MTZ ... Both merged formats hit the same refusal - the file carries the mean and the Bijvoet pairs - but the label vocabularies differ per format and the MTZ incantation fails on the mmCIF with a fresh error. Give both measured commands, and say it is one behaviour in two formats, not a difference between our files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	64182bb295	docs: an overview page - what a rugnux run does, in order ... The page that did not exist: one paragraph per stage from opening the file to the written reflections, each linking into the data-analysis reference part that carries the depth, with the stills differences at the end. First entry after the landing page, and the landing page says to read it first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	20f869c0b8	docs: split the data-analysis reference into four parts along the pipeline ... CPU_DATA_ANALYSIS.md becomes a short landing page (scope, part map, references) over four parts in pipeline order - images to spots (0-3), indexing and geometry (4-7), integration/scaling/merging (8-12), space group and validation (13-14). Pure moves: the section numbering is continuous and unchanged, since the rest of the documentation and the source cite sections by number. Inbound topical links now land on the right part; the build has zero warnings and the rendered-HTML anchor check finds no dead link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	376aa0a2e0	docs: split RUGNUX.md into one page per job, and put rugnux first ... The 1274-line page becomes a landing page (quick start, the page map, where it fits) plus seven pages a reader can answer one question from: installing, what rugnux reads, running it, integration with other programs, the results report, advanced usage, and powder calibration. Content is moved, not rewritten - only the connective sentences at each page top are new. Every internal anchor is remapped to its new page and every inbound link (DEPLOYMENT, TOOLS, HDF5, CPU_DATA_ANALYSIS) updated; the built site has zero Sphinx warnings and an anchor check over the rendered HTML finds no dead link. index.rst leads with the rugnux group, then acquisition, FPGA, reference and project. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	784cf87cc8	docs: put the generated python-client reference behind one landing page ... The OpenAPI Python client owned the sidebar: DefaultApi's 128 method anchors plus the 64 hidden-glob model pages were 195 of its 256 entries, because sphinx_material's globaltoc includes hidden toctrees by default. A new PYTHON_CLIENT.md landing page carries the links and a hidden glob toctree, and globaltoc_includehidden is off, so every generated page is still built and reachable (verified: 64 model pages + DefaultApi render, zero warnings) while the sidebar drops to 61 entries. docs/review/ joins exclude_patterns so a local, gitignored review report can never again be rendered into the published site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	6081b6bc43	docs: credit the L test, FFT indexing, TORO, Niggli, peakfinder8 and SparseCCL ... Six methods the pages name or describe carried no citation: Padilla & Yeates (the L test), Steller, Bolotovsky & Rossmann (the projection/FFT autoindexing MOSFLM implements), TORO (what ffbidx implements), Krivy & Gruber and the ITA lattice-character table (the reduction and Bravais assignment), Cheetah's peakfinder8 (the per-ring background statistics of the adaptive finder) and Hennequin et al.'s SparseCCL (already credited to traccc, now also to its authors). Each gets its ACKNOWLEDGEMENT.md paragraph, a References entry in CPU_DATA_ANALYSIS.md, and a one-line credit at the algorithm. The Sheriff & Hendrickson / Popov & Bourenkov entry is re-scoped so each claim sits on the paper that supports it - P&B 2003 is titled, and credited for the sigma-aware anisotropy estimation its statistic modelling contains, not for the tensor and its constraints. All DOIs verified against the publishers; the SparseCCL DOI resolves to IEEE document 9049184 (IEEE blocks content scraping, so verified by the resolved document id plus two independent sources). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	86f8edfabc	docs: the FFT axis ceiling, one-sweep inputs, exit status and _anom.ccp4 ... The longest FFT axis (500 A, no flag; -C moves it) was implied twice and never stated, and its failure mode is a plausible sub-cell rather than a refusal. One input is one sweep - said affirmatively where inputs are described instead of in an aside about pointless. A default 50 A low-resolution cut discards real reflections on a very large cell; the option row says so. Exit status is documented for scripts (0 = completed, non-zero = stopped), and the --model output list gains _anom.ccp4, which was written but undocumented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	87b1402c50	docs: hand the output onward - downstream commands and the three blocking facts ... The page named phenix, REFMAC, POINTLESS, AIMLESS, careless and SHELXC a dozen times without one command line, and left unstated the three facts those programs stop on: the free-flag convention (0=work 1=free; REFMAC needs FREE 1), the phenix label choice the double intensity array forces, and the unmerged file's header symmetry and sort order (determined group, sorted H K L M/ISYM BATCH - WriteReflections.cpp sorts it). A new 'Taking the data onward' section carries the worked lines, the careless column renames (BGVAR is a variance), and the Phaser SGALTERNATIVE keywords for the enantiomorph the report leaves open. The POINTLESS series trap now names ALLOW OUTOFSEQUENCEFILES instead of telling users to touch their data, the report-grep block warns that SPACE_GROUP_NAME carries one member of an enantiomorphic pair by convention, and the P1 cross-check's free set is declared to be its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	b1ae1ece67	docs: a default rotation run already writes real anomalous data ... The -A row and FRIEDELS_LAW=TRUE together read as 'without -A your Friedel pairs were averaged', which is false: the rotation merge always keeps the Bijvoet split and the default .mtz/.hkl carry it (WriteReflections.cpp, BuildMergedRows). Say what -A actually changes - the counting basis and the error model - and give the merged MTZ's exact column labels, which scripting against phenix needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	40cb5cd6db	docs: the quick start tells the truth about inputs, outputs and the cutoff ... A default rotation run writes seven files, not five - the two the list omitted are most of the bytes. The input is any NXmx/EIGER master or miniCBF sweep, which the page said only 350 lines later after twice implying Jungfraujoch data only. The CC1/2-0.30 resolution trim moves up to the quick start, and _image.dat's columns are finally named (ScalingResult.cpp writes a # header). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	40403d7beb	docs: the direction grid bounds the longest findable axis; say so in 5.3 ... The ranking-not-sampling observation was measured on the coplanar-shortlist rescue and is scoped to it now. The angular-resolution bound theta < d_min/(2a) means the shipped 16384-direction grid resolves axes only to roughly 120-150 A, far below the 1200 A the accepted maximum admits, and the failure mode is a plausible sub-cell, not a refusal - the reader raising fft_max_unit_cell alone deserved to know it cannot work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	32adb882eb	docs: disambiguate the reused symbols and the bandwidth definition ... sigma_bw is one physical smear written in reciprocal units in 8.2/11.1 and in pixels in 9; the two Delta-phi of the partiality formula are named; 13.5's |s| = 1/d is reconciled with the s = sin(theta)/lambda of 10.6/14.2; bandwidth is the rms spread, with the FWHM-input conversion (/2.355, BraggIntegrationEngine.cpp) stated. The ice-extinction clause now covers (104), not only (00l), and the AIMLESS <I/sigma> = 2 constant is named as AIMLESS's default rather than pointed at a criterion this project does not use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	e94548c614	docs: the twinning exemption list, and why -1, 2/m and mmm are not on it ... The stated criterion (a merohedral twin law exists) formally exempts every holohedral Laue class, but the code (TwinningAnalysis.cpp) deliberately keeps the low-symmetry ones eligible because pseudo-merohedral twinning through a special metric cannot be excluded there. Say both halves, and give the reference values of the statistics so the mmCIF numbers can be read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	9814846ebd	docs: state the q convention at every numeric q and on the q flags ... Every q in the pipeline is 2*pi/d (Definitions.h, the azint bin mapping), but the numbers in 3.3, 7.6 and 10.10 and the --azim-* flags never said so, and a reader taking q = 1/d would set --azim-q-spacing or --azim-max-q wrong by 2*pi. The 7.6 ice triplet is spelled out so 'within 0.06 of one another' reads as the adjacent-ring spacing it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	14b79b0314	docs: fix the sigma-ratio, the profile-fit term and the outlier cut as written ... The 1.109 in 9.2 is the variance ratio of the shipped 4/6/13 stencil (45/408 pixels), not the sigma ratio the sentence attached it to, and the effect is a bound attained on weak reflections, not uniform. The profile-fit background term is (sum P/v / sum P^2/v)^2 var(b), matching the code; the undefined w is gone. The 13.3 refit cut is N^2 times the model variance (RotationScaleMerge.cpp), not N*sigma^2. The sigma floor is 1 count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00