209 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	d4f280047f	grid scan: a raster reaches the spot engine, and the crystal cap says none rather than zero ... Two defects the merge created and one the API carried. Rugnux gated the per-image spot engine on AnalysisModeIsMX, so AnalysisMode::Grid fell through to the azimuthal-integration-only path: a raster ran, scored nothing, and reported no crystals. The gate now asks the stages table whether the mode does spot finding, which is the actual question - three modes need that engine for three different reasons, and a fourth would otherwise have to be remembered here too. max_crystals was a required integer defaulting to 10, with 0 meaning "all". Zero reads as "report no crystals", the opposite of what it did. It is now optional, and absent means no cap; a crystal found and then dropped is information the caller cannot get back. grow_score_threshold was missing from the schema entirely. Measured over the labelled corpus after these fixes: 34 of 34 confirmed-protein rasters yield a crystal, 0 of 8 water, 0 of 6 ice, 19 of 19 heldout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 08:22:14 +02:00
leonarski_fandClaude Opus 5	37efa573c4	generated: regenerate the API clients for the analysis mode, the grid settings and the crystal list ... Output of update_version.sh. The five lanes each reverted their generated trees before handing back, so that two independently regenerated copies never had to be merged; this is the single pass that replaces them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:59:42 +02:00
leonarski_f	cb1d88dd4d	Merge branch 'worktree-agent-a4c58d76a0be29dd6' into grid-scan-integration ... # Conflicts: # rugnux/rugnux_cli.cpp	2026-09-08 07:55:31 +02:00
leonarski_f	36a082c2f8	Merge branch 'worktree-agent-aa55c81220288558b' into grid-scan-integration ... # Conflicts: # common/ScanResultGenerator.cpp # docs/CBOR.md # docs/HDF5.md # frame_serialize/CBORStream2Deserializer.cpp # frame_serialize/CBORStream2Serializer.cpp # writer/HDF5DataFilePluginMX.cpp # writer/HDF5DataFilePluginMX.h # writer/HDF5NXmx.cpp	2026-09-08 07:54:46 +02:00
leonarski_fandClaude Opus 5	748d14d2cd	file format: the ice ring ratio is stored and sent under its own name, and the retired one is still read ... Completes the rename the C++ identifiers already carried. Producers now emit ice_ring_ratio / ice_ring_ratio_mean over CBOR and write /entry/MX/iceRingRatio and iceRingRatioMean; both readers accept the retired ice_ring_score / iceRingScore spellings as well, so nothing that exists on disk or on the wire stops being readable. The REST plot_type gains ice_ring_ratio and KEEPS ice_ring_score, both mapping to the same plot, so no client breaks. --ice-min-score is deliberately unchanged: it is a threshold on the ratio, it is the one surface a user types, and this program has 91 long options and no aliases. The fallback is not a formality. rugnux --mode scale reads the stored per-image value to reproduce the ice gate the writing run applied, and an unread dataset does not fail loudly - ice_n == 0 sends the gate to its fail-safe branch, "ice present". Demonstrated on a clean crystal whose stored file uses the old names: with the fallback the gate reads 1.06 against a 1.50 threshold and skips ice handling; with the fallback removed the same file has 272056 of 1032493 reflections (26%) excluded from the scale fit, and ISa, R_meas, I/sigma, SIGANO and both error-model terms all move. That is a silent change to merged intensities on files already written, which is why the two new CBOR test cases were each checked to FAIL when their fallback is taken out rather than merely to pass. Verified: same binary on an old-name and a new-name copy of one file gives byte-identical .hkl, .mtz and unmerged .mtz and an identical report; a file written after the rename scales to the same bytes again; the master and the data-file read paths were each exercised with both spellings; and the live API serves the same plot under either name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:42:23 +02:00
leonarski_fandClaude Opus 5	273591af11	grid scan: a crystal is grown out of its seeds, so one weak grid point no longer splits it ... A crystal in the corpus was reported as two because the grid point joining its halves scored 0.498 - two thousandths under the threshold. That is an artefact of putting a hard edge through a continuous quantity, not a gap in a crystal. The patch search is now hysteresis. A patch is labelled at a lower grow threshold (0.35) and kept only if it holds a cell above the seed threshold (0.50), which is the same thing as growing out of the seeds in one pass of the labeller rather than two. A cell between the two levels joins a crystal that already exists but can never start one, so no lowering of the grow threshold can turn a background into a crystal - and the negatives are safe by construction, since water peaks at 0.146 and ice at 0.490, both under the seed level. Both halves of the small-patch rule read the GROWN patch - the count includes the cells growth added and the peak is the patch's best cell wherever it lies - which is stated at the test, because a reader will otherwise wonder whether a grown cell can rescue a patch no seed would have admitted. It cannot: the patch is discarded before that test unless it holds a seed. The reported extents are those of the grown patch, so they reach the 0.35 contour. Over the corpus that is major_um x1.04 and n_images x1.07 at the median (x1.11 and x1.14 at the mean, +7% cells overall), and the cells it adds are spread fairly evenly over 0.35-0.50 rather than piled at the bottom - a crystal edge sampled at one grid step, not bleed into a neighbour that never diffracted. The patch mean falls with them, and can now sit below the seed threshold; the peak beside it is what the admission was decided on. RASTER_REPORT_VERSION is 3, because N_CELLS and the extents are a different measurement under the same names. The corpus table is unchanged: 17/17 protein, 0/4 water, 0/3 ice, 10/10 heldout. The split heals into one crystal, and one raster's five patches become three. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:34:36 +02:00
leonarski_fandClaude Opus 5	98132d0f83	analysis: every analysis method carries its own settings, and a raster's indexing is one of them ... AnalysisSettings had begun collecting per-method parameters - the calibrant was already in it, and the grid thresholds were about to be. That makes the structure every method reads grow whenever any one method gains a knob, and it puts a field in front of readers for whom it means nothing. So: AnalysisSettings keeps what all methods share, which for now is the mode, and each method gets a class of its own bound the same way. GridScanAnalysisSettings holds the protein-score threshold, the minimum cells per crystal, the decisive single-cell score, the maximum crystals reported and the indexing switch. CalibrationSettings holds the calibrant and the ring source. Both sit on DiffractionExperiment outside the per-run dataset member, both have an Import/Get pair, and both have their own endpoint - /config/grid_scan_analysis and /config/calibration - which is how every other settings group in this API is already reached. Grid indexing is no longer fixed in the stages table. It was turned off there on cost grounds, and that reasoning does not hold: a raster runs at up to 100 Hz, which the FFT indexer keeps up with, and a fixed-target serial experiment with a known cell wants ffbidx on every cell, where a raster that indexes is most of the measurement. So it is a setting, and DEFAULTS ON. It is additive rather than a change of answer - blobs are still found on the protein score, so indexing alters nothing about which cells are called crystals and only adds what was found in them, including the per-cell lattice count, which is the cheapest multi-lattice or cracked-crystal signal there is. That makes indexing the one stage a mode does not decide. AnalysisModeStages still carries a value for it, but only as the setting's default, and DiffractionExperiment::GetAnalysisStages - which is what every gate reads - substitutes the configured one. The table row is marked so nobody reads it as the mode's answer. The calibration knobs stay coupled to the mode but the rule now lives with them: CalibrationSettings::ApplyToAzimuthalIntegration moves azimuthal integration onto the CPU and supplies sectors where fewer than four were asked for, carrying the reason with it - the FPGA integration core holds 2048 bins in total, so 32 sectors leave 64 q bins, which cannot locate a ring. Stated there because it will otherwise read as an FPGA defect to be fixed back onto that path, and it is not one: the core is sized for a detector at full rate, and a calibration exposure is a few images at a few Hz. Both imports apply it, so the order the mode and the calibration settings are set in does not matter. CalibrationMethod moves from image_analysis/geom_refinement/PowderCalibration.h into common/CalibrationSettings.h, which that header now includes. One enum, so the setting and the code consuming it are not two vocabularies; every existing user sees it unchanged. The grid thresholds have one home and it is this class. The raster work owns AnalyzeGridScan's parameter surface and carries PROTEIN_SCORE_THRESHOLD_DEFAULT / MIN_BLOB_CELLS_DEFAULT beside that header today; the header here states the signature that replaces them, so the two do not become competing defaults. The beam size deliberately stays a separate argument to AnalyzeGridScan: it is measured, not configured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:27:41 +02:00
leonarski_fandClaude Opus 5	eba81013cd	analysis: a scanning goniometer under an MX mode is normal, and no longer warned about ... The warning claimed a sweep collected under mx_stills was being analysed as something the data did not support. It is not a problem to report: a scanning goniometer is REQUIRED for the file to carry correct metadata, mx_stills included, so its presence under an MX mode is the ordinary case and firing on it would train an operator to ignore the log. Removed rather than downgraded. Nothing is left behind saying the condition is tolerated, because there is no condition. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:26:58 +02:00
leonarski_fandClaude Opus 5	0f00b76a9a	image analysis: the ice score takes the pipeline's own band width, and the ice quantities are named for what kind of number they are ... Two things, both about telling one ice quantity from another. The ice score's spot channel had its own band half-width of 0.02 A^-1 while the spot finder marks ice rings at 0.03 (ice_ring_width_Q_recipA). The 0.02 was justified by a 5 pp specificity gain measured on the PYTHON PROTOTYPE, which used a fitted beam centre and a mask-derived coverage table; the shipped port, which takes the geometry's centre and the azimuthal profile's own live pixel count, does not reproduce it. Measured over the corpus by truth class rather than by directory label, at 0.02 vs 0.03 on the combined score: ice loops 62.13/62.19%, _icy protein 89.03/89.79%, _clean protein 16.51/16.31%, water 17.19/20.03%. The widths are indistinguishable except on water, where one of the four loops is independently known to carry a full hexagonal pattern. So the width is now a parameter and the pipeline's own value is passed in - one band width, not two. The 0.012 tolerance in the radial channel is NOT a second band width, and is renamed CENTRE_SMEAR_Q to say so: it is how far either side the channel looks for the bin a mis-set beam centre moved the ring to. The rest is naming. Three kinds of number were all called score, or built from things called count, and a reader could not tell from the name whether 1 meant "none" or "certain" - which are opposite. The convention, now stated in docs/CPU_DATA_ANALYSIS.md: *_score is bounded [0,1] and 1 is certainty, *_ratio is unbounded and 1 is nothing, *_count is a count. The C++ identifiers for the ice ring ratio follow it (ice_ring_score -> ice_ring_ratio, GetIceRingScore -> GetIceRingRatio, PlotType::IceRingScore -> IceRingRatio), and the local in the scaling gate that shadowed the new ice_score while meaning the ring ratio is renamed with them. Nothing outside the source moved: the CBOR keys ice_ring_score and ice_ring_score_mean, the datasets /entry/MX/iceRingScore and iceRingScoreMean, the ice_ring_score plot type and the --ice-min-score flag are all unchanged, and were checked to be after the rename. Renaming those changes stored files, the stream format, the REST API and a CLI flag, and is a separate decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:19:03 +02:00
leonarski_fandClaude Opus 5	cdb60c2916	grid scan: one cell is a crystal when that cell is decisive, and the list can be capped ... The minimum patch size was a flat floor, so a crystal that covers one or two grid points was thrown away however strong its diffraction. It is now a floor OR a piece of evidence: a patch is reported when it has min_blob_cells cells, or when its best cell reaches decisive_protein_score. One condition, and both halves are parameters. The bar is the patch PEAK, not the patch mean. A two-cell patch with one strong cell and one marginal one is the case this exists for, and the mean averages exactly that evidence away. Over the 67 labelled rasters the two populations do not overlap: no water raster reaches a peak of 0.15 and no ice raster reaches 0.50, while the weakest protein raster peaks at 0.67 - so 0.6, the middle of that gap, is the default. The peak is reported beside the mean, in the table and in the JSON, so an admission can be checked against the number that decided it. max_crystals caps the returned list after the sort, best first; 0, the default, is all of them, because a crystal that was found and then dropped is information the caller cannot get back. The four parameters now travel as one GridScanAnalysisParameters, which is the shape the GridScanAnalysisSettings class in common/ is to take: when it lands it replaces this struct in the signature and nothing else changes. On the corpus this reaches 17/17 protein loops (it was 16/17 - the miss was a crystal covering two grid points of a 4x4 raster) with water still 0/4 and ice still 0/3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:15:30 +02:00
leonarski_f	cd9cc4800e	Merge branch 'rc167' into worktree-agent-ab3a3e173261a95d8 ... # Conflicts: # common/JFJochMessages.h # common/ScanResultGenerator.cpp # docs/CBOR.md # docs/HDF5.md # frame_serialize/CBORStream2Deserializer.cpp # frame_serialize/CBORStream2Serializer.cpp # frontend/src/components/DataProcessingPlot.tsx # tests/CBORTest.cpp # writer/HDF5DataFilePluginMX.cpp # writer/HDF5DataFilePluginMX.h # writer/HDF5NXmx.cpp	2026-09-08 00:24:42 +02:00
leonarski_fandClaude Opus 5	550f7c7d7b	scan result: a grid scan's crystals and a calibration's geometry reach the stream, the file and the API ... A grid scan and a powder calibration each produce an answer that is not a per-image number, and neither had anywhere to go. ScanResult gains two top-level optionals, grid and calibration, in the same idiom the three rotation_* members already use: a mode's payload is present when that mode ran and absent otherwise. images stays required, so every existing /result/scan client - the python client and the beamline scripts on it - is untouched. No oneOf or discriminator. The spec contains none today, so polymorphism is unexercised across all three generators at once, and the only thing it would buy is making "exactly one payload" structurally true rather than true by convention. powder_calibration_output has been fully specified and referenced by nothing since it was written; the calibration member reuses it. The crystal list travels the same three hops a per-image quantity does. In the CBOR end message grid_crystals is an array of maps keyed by field name - the shape a spot already uses - so a reader that does not know a field skips it and the record survives a field being added; the deserializer steps over an unknown key rather than reading its payload as the next one. In the HDF5 master it is parallel 1-D datasets under /entry/MX/crystals, one per field, because crystals are numbered rather than named and a group per crystal would put the index in the path. The reader reads them back so a stored raster re-opens with what it found. Nothing caps the list at one. A raster can find several crystals and the ordering carries the answer: sorted by score, best first. angle_deg is an axis, not a direction. It lies in [0, 180) and wraps, so 179 and 0 are adjacent and two of these do not average arithmetically - a pair of nearly parallel needles averaged across the wrap comes out as a right angle. That is stated at every layer it passes through, since it is the kind of field a consumer will reach for without reading the producer. The producer side is not wired yet: nothing fills ScanResult::grid or EndMessage::grid_crystals here. The seam is receiver/JFJochReceiver.cpp, where rotation_lattice is filled in EndDataCollection and GetFinalStatistics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:17:10 +02:00
leonarski_fandClaude Opus 5	a7d3ada3ab	analysis: what runs over the images is one stated mode, shared by broker, rugnux and viewer ... Until now nothing in the tree said what analysis a run performed. The answer was composed at each site out of four independent scalars - the detector type, two separate "spot finding off" switches, an indexing flag and a rotation flag - so what was configured and what actually ran were different things, and no single place could be read to find out which. AnalysisMode {None, MXRotation, MXStills, Azint, Grid, PowderCalibration} is that statement, in common/ because all three programs configure the DiffractionExperiment that carries it. AnalysisSettings sits on the experiment beside IndexingSettings, outside the DatasetSettings member, which is the one thing a /start replaces wholesale - so the mode is persistent by construction rather than by a rule someone has to remember. The mode does not label a run, it decides it. AnalysisModeStages() is a table - modes as rows, pipeline stages as columns - and every gate reads that table instead of testing the mode: spot finding in DiffractionExperiment::IsSpotFindingEnabled, indexing (and with it prediction and integration, which never run without a lattice) in one gate inside IndexAndRefine that serves all three front ends, azimuthal integration where the CPU engine is built. Two rows carry a judgement worth reviewing: powder calibration keeps spot finding, because --calibration spots fits the pooled spots; grid does not index, because a raster is thousands of frames and the per-image scoring it ranks on deliberately avoids an indexer that fires on ice. There is deliberately no Auto value. GetIndexingAlgorithm() resolves Auto at read time, which is exactly why an indexing setting cannot be read back off the configuration; removing that kind of implicitness is the point here, so the mode getter stays a plain accessor. MXStills is the default because None would silently switch analysis off on every deployment whose configuration predates the field. Rotation MX is absent from the OpenAPI schema rather than present and refused: jfjoch_broker has no rotation analysis path, so the REST and configuration-file routes cannot express it at all. The shared enum can still carry the value from elsewhere, so CheckAnalysisSettingsOnline refuses it on both routes with a message naming rugnux. A sweep collected under an MX mode is not refused - collecting rotation data online is normal and live spot counts are useful - but it is said out loud in the log, since the mistake worth preventing is the silence about what was done to it, not the acquisition. Powder calibration forces azimuthal integration onto the CPU and supplies 32 sectors where fewer than four were asked for. The FPGA integration core holds 2048 bins in total, so 32 sectors would leave 64 q bins - far too coarse to fit a ring. Frame rate is what this costs and a calibration exposure does not need it. The two existing "no analysis" switches, per-dataset dataset_settings.spot_finding and persistent SpotFindingSettings::enable, are interfaces in too many places to remove now. They are marked deprecated in the schema and in both headers, and the mode takes precedence over them: a mode that analyses no spots wins outright, while under a mode that does find spots they remain the finer control. The precedence is written where it is enforced. rugnux's ProcessMode is gone, replaced by the shared enum; RugnuxMode stays as the CLI spelling layer and no existing spelling changes. --mode gains mx_rotation and mx_stills, which are spellings of -R and --force-still rather than new switches; plain mx still chooses between them from the goniometer. scale keeps no shared counterpart, since it runs no analysis over images at all. The mode reaches the CBOR start message and /entry/MX/analysis_mode in the HDF5 master, so a written file records which analysis produced it. It is read back as provenance only - what a stored file was produced by is not what the next run should do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:16:26 +02:00
leonarski_fandClaude Opus 5	14e2bf3c0f	Merge rc167 into the detection-score branch ... Both lanes added a per-image scalar to the same eleven files, so every conflict was two additions competing for one line. All were resolved by keeping both, with three that needed more than that: - ScanResultGenerator: rc167 changed the per-image float vectors to resize(n, NAN) so a frame that never arrived does not read back as a real 0. v_protein_score and v_ice_score are exactly that case - 0 is a real answer ("nothing detected here") - so they take the NAN default too. - HDF5MetadataSource: rc167 established that NaN in a stored per-image array means "no value" and the optional must come back absent. The two detection scores now follow it, which they did not before the merge; without the guard a missing score would come back as a NaN that a threshold would silently compare against. - CBORTest: designated initialisers must follow member declaration order, so spindle_blind_fraction precedes the two scores in the DataMessage aggregate. Verified after the merge that every CBOR key that is encoded is also decoded (198 encoded keys, one intentional exception: the "type" discriminator), that both lanes' fields survive in the writer, the reader, the plots and the API, and that a stored file still round-trips. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:13:51 +02:00
leonarski_fandClaude Opus 5	21eca107fa	rugnux: --mode raster reads a stored grid scan and reports the crystals in it ... The offline entry point for a raster. It scores every image of a stored grid scan - spot finding for the per-cell resolution, no indexing, because a raster answers where the crystal is and not what its lattice is - hands the per-image protein and ice scores to AnalyzeGridScan, and writes what came back as two files beside the usual output: <prefix>_raster_report.txt in the idiom of the results report, and <prefix>_raster.json with the same content typed, so a battery can aggregate a sweep without scraping prose. --beam-size states the beam at the sample, defaulting to the file's incident_beam_size. It matters more than it looks: the reported crystal sizes are measured and still contain the beam, and taking an anisotropic beam back out is a subtraction of two covariance matrices, so a beam given as square when it is not rotates the reported crystal axis. --raster-protein-threshold and --raster-min-cells expose the two constants AnalyzeGridScan held at file scope, so a sweep can vary them without a rebuild; they are now defaulted parameters with the old values, and every existing call is unchanged. The observer feeding the analysis reads the file's own image number (DataMessage::original_number), not the loop's ordinal, so -s/-e/--stride cannot silently shift the grid mapping. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:01:39 +02:00
leonarski_fandClaude Opus 5	62ca2b24f6	image analysis: two per-image detection scores say whether there is protein and whether there is ice ... proteinScore and iceScore are saturating scores in [0,1] that answer presence, not quality: a superb crystal and a barely-diffracting one both read near 1, and neither the spot count nor the resolution enters either of them as a term. iceRingScore already reports a magnitude - a ratio, unbounded - which is not a number that can be thresholded; these are. The protein score counts distinct d SHELLS above 5 A rather than spots, so a parasitic ring in the low-resolution band cannot accumulate evidence, and weights each spot against the frame's own median so a scattering of the weakest detections cannot fill a shell either. The ice score carries two channels and takes the stronger: a radial one over the azimuthal profile, which runs the hexagonal and the CUBIC phase as separate hypotheses and decides between them at the end (flash-cooled loops show cubic or stacking-disordered ice at least as often as hexagonal, the two share only three lines, and dropping the cubic hypothesis costs about 5 pp on iced loops), and a spot one that reads an excess on the ice radii against the same band slid to every ice-free offset, which is what catches ice arriving as discrete crystallites and leaving the radial profile flat. Both read d out of the geometry, so both move with a beam-centre error; the centre is not fitted here, and the one they were computed with is written beside them as scoreBeamCenterX/Y so a later rescoring can tell an algorithm disagreement from a geometry one. Ported from validated prototypes and checked against them frame by frame on stored data: mean absolute difference 2.7e-5 (protein), 1.3e-8 (ice radial) and 3.4e-4 (ice spots). On a 41-loop battery the protein score reaches 98.4% of confirmed-protein frames and 0.00% of water frames, and finds no cluster on any water or ice raster. Cost is 0.01 ms/frame for the protein score and 0.08-0.32 ms/frame for the ice score. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 23:59:46 +02:00
leonarski_fandClaude Opus 5	b650cda037	Revert "integration: the background is fitted over the ring that survives, not averaged over it" ... This reverts commit 6a3ae9c65. The rotation battery refuses it. The median is clean - zero movement in R_meas and CC1/2, most crystals untouched or moving in the third decimal, which is what was predicted - but the tail is not a cost, it is a failure: one crystal's CC1/2 falls from 91.3% to 6.8% with its reflection and observation counts essentially unchanged, so the intensities are being corrupted rather than the weak ones discarded. Another loses seven points of R_meas and half its ISa; a third has its high-shell CC1/2 go negative; a fourth's ISa doubles, which is an error model measuring something different rather than measuring better. The change was predicted to cost one to two points of high-shell CC1/2 on crowded patterns, and that prediction was made on six geometries. Fifty-one say otherwise. The defect it fixes is real and stays on the record: within the outer background radius of the array edge the ring is truncated inward while the disk is not, so the surviving ring sits where the radial background is higher and the reflection reads low - measured at 1.79 counts per pixel four pixels from a border, and confirmed by a masked gap producing the same bias signed by the direction of its displacement. Fitting the background over the surviving ring is the right shape of answer. Doing it per reflection, from a ring that a neighbour mask has often already truncated, is evidently not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 16:30:22 +02:00
leonarski_fandClaude Opus 5	6a3ae9c658	integration: the background is fitted over the ring that survives, not averaged over it ... The signal disk and the background ring are concentric, which is the whole reason a linear background cancels between them. Within the outer ring radius of the edge of the sensor array that concentricity is gone: the ring loses its outer part while the disk barely loses anything, so what is left of the ring sits further into the detector, where the radial background is higher, and the reflection reads low. Measured at signal-free positions four pixels from a border: the ring reads 162.35 counts per pixel against a true background over the disk of 160.57, which over a hundred disk pixels is 182 counts of deficit, against 216 to 239 observed. <I/sigma> runs -1.83, -2.34 and -1.18 at nought to three, three to six and six to nine pixels from the border, and recovers exactly at the outer ring radius. The same reflection measured at a border reads 179 counts lower than in the interior over seven thousand matched pairs. A masked module gap does the same thing but signed by the direction of the displacement, which is why nothing has caught this: at a gap the two populations cancel in the mean, while at the sensor border the truncation is always inward, so the bias is always negative. The background is now the intercept of a straight line in radial offset over whatever ring pixels survive, read at the reflection's centre. Three extra sums per ring pixel and no extra reads; the radial distance was already computed there. It is exact under any truncation and reduces to the mean when the ring is whole, so it is unconditional rather than a mode: a badly truncated ring pays in sigma, through the variance the fit honestly reports, rather than in a rejection. On the crystal where this surfaced the outermost shell's correlation with a deposited model goes from -0.234 to +0.004, and the shell above it from -0.091 to +0.179. Correcting beats discarding: dropping every observation within fifteen pixels of a border reached only -0.019 and +0.127, because the corrected observations still carry signal. Interior reflections do not move. The cost is one geometry: where a neighbour mask has already truncated the ring almost everywhere, the fit roughly doubles the variance of the background estimate while finding no gradient worth removing, and a crowded small detector loses one to two points of CC1/2 in its finest shells. Also: the MINPK denominator counted only profile mass that lands on the detector, so a reflection whose peak is off the sensor scored a perfect one and no guard could fire. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 15:46:32 +02:00
leonarski_fandClaude Opus 5	e5c96cd41b	model validation: the placement target carries a bulk solvent that means something ... The rigid-body target refits the scale at every evaluation, deliberately - held fixed, the target would measure the scale as much as the placement, and the body would translate to repair a scale error instead of moving to where the density is. That refit was gemmi's unbounded fit, the one already replaced for the reported scale, and here it was worse: measured over a corpus of deposited models, 40% of the evaluations that decide where the model goes came out with a b_sol outside 10-80 A^2, on 57 of 89 datasets, ranging from -8072 to +1721. A negative b_sol is a solvent term that GROWS with resolution. One crystal ran its entire committed placement between 290 and 430 A^2, and that placement went into the reported maps. The bulk solvent is fitted once per zone instead, inside the same physical box the reported fit searches, and then held while the overall scale and the anisotropic B keep following the body. That split is the point: k_sol and b_sol describe the crystal's disordered solvent, not the fit of one placement, and measured across a whole zone they drift by a single grid step. Fitting them at every evaluation costs three times the wall clock, makes the scaler 82% of the run, and moves discontinuously under a forward difference - which is poison for a numerical Jacobian. This costs 4%, leaves no evaluation outside the box, and lands the body within 0.37 degrees of the expensive version, against 4.33 degrees for what it replaces. Placements change on fourteen crystals in eighty-nine. R-free is a wash on the mean; the step buys more total R-free from fewer commits, and the gain sits where the runaway actually bit. Two things found while auditing the file and left as they were, because they are right: the refinement sees working reflections only, end to end, and the gauge that removes the origin-free directions of a polar group carries more than half the whole refined translation in ninety-four of a hundred and thirty-nine polar zones. It had no test. It has one now. An observation with no calculated amplitude gets a zero residual, which drops it from the target rather than scoring it as a perfect fit, and is now counted and reported - a large count says the model's reflection conditions do not match the data's, which is a statement about the model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 15:27:30 +02:00
leonarski_fandClaude Opus 5	004fa5a781	model validation: the model says, shell by shell, where the data still carry signal ... A placed and scaled model gives an expected intensity for every reflection, and comparing that with the merged intensity is a measurement nothing else in the report can make: CC1/2 and R_meas compare the data with themselves, and can only say whether a shell is reproducible, not whether it is right. CC(model, data) is reported per shell, on the merge's own shells so the rows line up, with the reflection count and a significance beside each. Read it in one direction only. A correlation significantly above zero proves signal is present, because a model cannot invent agreement with data it never saw. A correlation near zero proves nothing, since the model may be the thing at fault - one crystal in this corpus reads negative across its outer shells against a deposited model of a different crystal form. So this can support keeping a shell and must never be used to discard one, and nothing in the pipeline acts on it. Intensities rather than amplitudes, matching CC1/2 and CCref beside it. The raw merged intensity rather than the French-Wilson amplitude: that amplitude is a posterior mean under a Wilson prior which pulls weak reflections towards their shell mean, and the shells this number is read in are exactly the weak ones, so it would correlate the prior with itself. The free flag is ignored on purpose. Nothing has been refined against these reflections, so the correlation is unbiased over all of a shell rather than over the test set alone - which is where the outer-shell reflection counts are, and with them the difference between a statistic that can resolve something and one that cannot. REPORT_VERSION is 7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 13:12:32 +02:00
leonarski_fandClaude Opus 5	cb08f63a52	model validation: the bulk solvent is searched inside its physical range, not fitted without bounds ... gemmi offers two scalers and we were using the one without bounds. Its Levenberg-Marquardt path has nothing stopping the flat-solvent parameters from leaving the range the model means anything in; the alternative path that does declare bounds is behind a compile guard we have never enabled. On this corpus six datasets in fifty-one fitted a b_sol outside it, the worst at 1707 A^2. What that does is subtler than a bad scale, and worth recording because it is why nobody noticed: a b_sol that large does not corrupt the solvent term, it switches it off - 1.4% of it survives at 10 A - so the model is simply scaled without a solvent contribution and the R-factors look unremarkable. k_sol and b_sol now come from a grid search over the physical box, with the scale and the anisotropic B refitted at each candidate pair, following Afonine et al. Refitting at each point is what makes it work: clamping the parameters after an unbounded fit costs up to 0.044 in R-free, because it leaves the scale and B where the rejected fit put them. Non-physical fits go from six in fifty-one to none, and both R-work and R-free come out slightly but significantly better rather than merely no worse. Which reflections are fitted remains the caller's business - the function scales whatever it is handed - so the working-set restriction of the previous commit is not something this can undo. A crystal with no solvent-accessible volume needs no special case: its mask is empty, so the solvent term is identically zero whatever the parameters say. There is a test for that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 10:34:21 +02:00
leonarski_fandClaude Opus 5	2ae8db6623	model validation: the scale is fitted on the working reflections, so R-free is free of it ... The model's overall scale, its anisotropic B and both bulk-solvent parameters were fitted over every reflection, free ones included, and R-free was then computed from the resulting scaled Fmodel. About eleven parameters, minimised against the very sum R is made of, chosen with the help of the reflections that are supposed to be held out of it. The documentation said so and argued it was harmless - the parameters are far too few to absorb individual reflections - and on the numbers that argument is right: the standard optimism bound puts the effect at a few hundredths of a percentage point on an R-free of twenty percent, below the free set's own sampling noise. It is still wrong. R-free means the test reflections took no part in choosing anything the model is scored on, and a reader who greps R_FREE= is owed that without a footnote. The fit now sees the working set; the parameters it returns are applied to every reflection, so the free ones are scored against a scale they had no hand in. Refmac fits scale and B on the working set and says so in its documentation; phenix.model_vs_data, which is the direct analogue of this mode, uses its work-only scaler. Two things still consult the free set, and the documentation now names them instead of leaving them to be discovered: the rigid-body step is committed only if R-free improves, and the alternative indexing is the candidate with the lowest R-free. Both are single discrete decisions rather than fitted parameters, both are the conventional use of a test set, and both leave R-free very slightly optimistic where they fire. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 10:31:24 +02:00
leonarski_fandClaude Opus 5	2574c2e587	docs: two changelog entries that are not a user's business ... One described a fix to a defect this same release introduced - the two-pass guard learning to weigh axial reflections, which exists because the indexed-frame floor moved in this release and exposed it. A reader upgrading from the last release never saw the defect, so the entry above it already says the only thing that changed for them. The other announced a regression-testing script. It is a repository tool, not part of what a user installs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 07:30:51 +02:00
leonarski_fandClaude Opus 5	24dcaf4e67	rugnux_vs_pdb.py: measure a corpus against the models its depositors published ... A companion to rugnux_vs_xds.py, asking a different question. That one compares merging statistics against another program's reduction of the same images; this one asks what an independent party already decided the images were worth - the deposited model, its published resolution and its published R-free - and compares against that. The runs are deliberately bare: --model and nothing else. No space group, no cell, no resolution limit. Anything told to the program is a thumb on the scale, and the question is what a bare invocation reaches on its own. The R-free it reports is a validation number rather than a refinement result: the deposited model is placed rigid-body against our merge and the free residual of that placement is what is compared. The depositor's published R-free came from full refinement against their own reduction, so it is the harder number, and the comparison is deliberately unfair in their favour. Entries are identified by directory name, which must be the PDB entry ID; metadata and coordinates come from the RCSB API. The sweep is resumable, since a hundred datasets is hours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 07:21:06 +02:00
leonarski_fandClaude Opus 5	e02dd2e924	rugnux: the two-pass guard weighs the reflections a space group is read from ... The guard that chooses between the file's geometry and the refined one judged the two passes on CC1/2 alone. CC1/2 is an average over tens of thousands of reflections and the principal axial rows hold a few dozen, so a pass can win it while measuring half the axial reflections the other one did - and the systematic absences that decide a screw axis are read from nothing else. An axial reflection crosses the Ewald sphere over a narrow range of the sweep, so a pass that loses frames loses whole rows rather than a scattering of reflections. Measured on a crystal that was losing a screw exactly this way: the refined pass held 7 of the 16 low-order axial reflections and won the gate by 0.0114 in CC1/2, which is inside its own noise. The screw evidence fell from 24.2 nats to 10.5, under the bar, and the run reported the symmorphic group. A third arm, beside the two the guard already had: refuse the refined pass when it holds under three quarters of pass 1's axial reflections AND its CC1/2 does not say it is better. It can only decide cases the gate was deciding on a difference inside its own noise, and it needs pass 1 to hold enough of the row for a screw to be claimable at all. Over the corpus it fires on three crystals: the one it was built for, which recovers its screw; a second that gains 0.057 in CC1/2 and was being handed the wrong geometry for reasons nobody was looking for; and a third that loses 0.009. The other 145 are untouched. This closes a regression the six-spot floor exposed. The floor did not cause it - at the old floor the refined pass was broken outright and the CC1/2 arm rejected it for unrelated reasons, which rescued this crystal by accident. What the floor removed was the accident. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 07:18:59 +02:00
leonarski_fandClaude Opus 5	2c4b804f70	docs: the scaling model carries all three deterministic corrections, as the code does ... The observation model said its L term was the prescaling correction, and that predictions carry its reciprocal as prescaling_corr. That was true when that one field held everything. It has since been narrowed to Lorentz x polarization, with the sensor's efficiency and the flight path carried beside it, and every site that corrects an intensity multiplies all three - the scaling fits, the merge ingest, the anisotropy analysis and the unmerged export alike. So the model term is the product, not the first factor of it, and a reader implementing against that page would have applied a third of the correction. The header comment on the efficiency factor had the same fossil: it said the total correction is prescaling_corr * qe_corr, written before the flight path existed. The comment two fields below it already said all three, so the file disagreed with itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 22:59:40 +02:00
leonarski_fandClaude Opus 5	fc41b1c34c	release: version 1.0.0-rc.167 ... VERSION bumped and update_version.sh run, which rewrites the version string everywhere it is recorded: the OpenAPI specification and the three clients generated from it, the frontend package and its version module, the Sphinx configuration, the Redoc page, and the FPGA HDL and PCIe driver strings. Every hunk is a version string and nothing else, checked rather than assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 22:45:37 +02:00
leonarski_fandClaude Opus 5	2538f5a956	docs: the manual describes the program this branch actually built ... A sweep of docs/ against the code, and a changelog a user can read. The report reference still described the section layout from before the report was rewritten, claimed anomalous keys were withheld from Friedel-merged runs when 144 of 148 stored reports carry them, and listed as default a set of keys that --developer now selects. The tutorial described an enantiomorph message that no longer exists; the option tables were missing --developer and --finalist-ledger and carried a short option the program does not have; the integration page's unmerged-MTZ column list predated the two new columns and asserted the absence of one of them. The analysis pages had the efficiency and flight-path corrections but none of the symmetry work: the pseudo-translation detector, the absence test that divides it out, the glide test, the evidence-keyed alternatives and the metric re-ask are now written up where the methods are described. The usage message omitted FLIGHT from the formula that turns a written intensity back into a raw count, while the writer has been emitting the column. The usage message is the authority on what the program does, so it says so now. The changelog had thirty-one entries filed under the previous release, most of them written from the inside: what a change did to the code rather than what it does for a reader. This release gets its own section with sixteen, each one capability. Every key, flag and column a user could grep for survives; what goes is the seam between one person's work and the next's, which is not something a user can act on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 22:44:09 +02:00
leonarski_fandClaude Opus 5	e8fbae879a	rugnux: the two offline instruments go, and one release states one report version ... The finalist-ledger and glide instruments were built to measure a question each, over stored merges, and both questions have been answered. They were never asked for and they are not part of the product: neither is installed, both duplicate what rugnux itself does with a flag, and each is one more executable to keep linking. The --finalist-ledger mode stays where it belongs, in rugnux. Their removal also retires the CUDA device-link property they would have needed in the portable build, which is the shape of problem an unused target creates: a configuration nobody builds, breaking for a target nobody runs. The changelog claimed four different REPORT_VERSION values inside one unreleased section, which is development history rather than something a user can act on. A release states its report version once; the entries keep what they changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 21:53:17 +02:00
leonarski_fandClaude Opus 5	ac1e8b8603	tests: the report version pin and the screw-gate section follow the code ... Two test expectations that the changes under them made stale, and one overstated sentence in the docs. REPORT_VERSION was bumped to 12 without touching the line that pins it - the very line whose comment says a key added to the report is a contract change and this is where it has to be acknowledged. Acknowledged, and the absence of SOHNCKE_SPACE_GROUP on a fixture that is GIVEN its group rather than searching for one is now asserted too, since that is what makes the key's contract honest. The section documenting that the E^2 gate is what saves a screw from fabricated violations no longer holds, because there are now two independent defences: with the gate off, the absent class still sits at 2% of its own row, so the zone is dead per reflection and licenses the absence evidence to override the count. The section is kept rather than deleted - it now pins the second defence, and a future P2 there would mean the deferral has stopped licensing a zone that is genuinely extinct. The docs said SOHNCKE_SPACE_GROUP is written on every run. It is written on every run whose group came from the search; a run given its group with -S has no Sohncke candidate to name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 21:12:42 +02:00
leonarski_fandClaude Opus 5	485bbeaf71	rugnux: a cell given with -C is matched in any equivalent description ... Every candidate the indexer forms is Niggli-reduced and primitive; the reference cell was compared against them exactly as typed. For a centred lattice those are different numbers entirely - a conventional F-cubic cell reduces to a rhombohedron with 60-degree angles, a centred monoclinic cell to axes it shares none of - so the true candidate failed the 5% test and the run reported that it had found no lattice. A user who supplied the cell as it is published was worse off than one who supplied nothing. The reference is now expanded into the primitive lattices its six numbers could stand for, one per centring, each reduced the way a candidate is; a candidate matching any of them is kept. The cell as typed stays in the set, so the indexer that returns the basis it was handed is unaffected. The cost is real and stated: a wrong cell now gets several chances to match rather than one. The controls that hold are on data - deliberately wrong cells are still refused, and every cell that worked before gives the same answer to the digit - but a synthetic lattice can be made to match through a centring hypothesis it has no right to, and no volume test tightens that without excluding a true candidate this corpus already contains. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:26:38 +02:00
leonarski_fandClaude Opus 5	78a941156b	symmetry: absence evidence overrides the violation count, one zone at a time ... A screw candidate was refused whenever more than a tenth of its predicted-absent reflections read as present, whatever the absences themselves said. On a strong axial row near the spindle those readings need not be structure factors at all: measured on one crystal, the same reflection reads +268 at one Ewald crossing and -44 at the other, a directional smear tail of the monsters beside it. Three such reflections refused a group whose absence evidence was the highest of every candidate scored, on exactly the axes the deposition names. So a class whose evidence clears the claim bar may now override the count. The likelihood has already priced those reflections in - they are inside sum_u - and still reads the class as extinct, so the count adds nothing it has not weighed, and adds it unweighted. The deferral is licensed ZONE BY ZONE, which is the part that makes it safe. The evidence is a group-level number while the count indicts particular zones, so an overwhelming genuine zone would otherwise pay another zone's debts: a four-fold screw row at 194 nats lifted the veto for two claimed two-fold rows reading 9 nats each with half their class individually present. Each zone carrying violations must now hold the deferral itself - dead at a nat per predicted absence rather than merely accumulating one over a long row, and not judged against a measured pseudo-translation, where the corrected evidence is a residue of the cone's own under-measure and the corrected count is the only instrument the modulation does not reach. Measured over the corpus: two crystals gain the screw their absences support, no crystal loses one, and every merged statistic is unchanged - a screw and its parent share a Laue class, so nothing about the merge moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:26:23 +02:00
leonarski_fandClaude Opus 5	118746d8da	symmetry: indistinguishable groups are keyed on the evidence, not on a count of absences ... The alternatives a run reports were the candidates whose absent AND violation counts matched the winner's exactly. Two settings of one point group routinely predict a different NUMBER of absences on a zone the sweep never measured - one extra reflection on a row with no control class of its own, contributing nothing either way - and that one reflection was enough to drop a candidate the data support exactly as well as the one adopted. Measured on an orthorhombic wedge: four candidates at the same 100.7 nats with no violations, of which the two predicting one more unmeasured absence were excluded, and one of those two was the right answer. The report then said the space group followed from the systematic absences, on an axis whose row was never collected. Keyed on the evidence, a super-centering that nets the same count but over-claims still scores differently and is still not offered. The adopted group does not change - it is the same candidate either way - so this only widens what the run admits it cannot separate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:25:51 +02:00
leonarski_fandClaude Opus 5	f98cbf5373	symmetry: a reference correlation with too little to correlate is refused, not scored zero ... PickBestReindex discards only non-finite scores, so returning 0.0 when fewer than the minimum number of reflections matched made that a SCORE rather than an abstention. An operator that mapped almost everything outside the reference's coverage - and therefore had nothing to be judged on - beat a measured identity that merely correlated a little negatively, and the whole run was reindexed on no data. NaN is what the per-image resolver's ReferenceCC already returns in the same situation, against the same minimum. This makes the two agree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:25:39 +02:00
leonarski_fandClaude Opus 5	de1b9299cb	rugnux: the report names the best Sohncke group on every run ... A crystal built from chiral molecules - any protein - has no glide plane and no inversion centre, so where the search names a group carrying a glide, the group that crystal could actually have is a different one. The run knows both and printed only the adopted one as a key. SOHNCKE_SPACE_GROUP= is written whenever a space group was determined, and repeats SPACE_GROUP_NAME= where no glide was found. That repetition is the point: a key that appears only sometimes has to be tested for before it can be read, and the reader this is for - someone who knows their sample is a protein and wants the answer their sample can have - should be able to grep one name without first deciding what the run concluded. REPORT_VERSION is 12. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:33:57 +02:00
leonarski_fandClaude Opus 5	6773e8516d	symmetry: a metric two-fold the lattice search refused is asked of the intensities ... The Niggli character walk takes the first character its tolerance matches, and on a lattice that is nearly but not exactly hexagonal it matches hexagonal. Under a hexagonal holohedry no candidate point group can carry the two strongest two-folds the data actually have, so the search lands on the weakest one and the crystal is processed in a group of order two where it should have been eight. The metric group knows better than the character walk: it holds every rotation the cell can host, including the ones the named lattice system has no room for. Each of those is now put to the intensities directly - one operator, scored the way Stage A scores its own, on the same reflection population, with the same strong-reflection gate and the same E^2 cap. That last part is what makes the answer usable: normalising over the full resolution range of a merge whose outer shells are correlated junk reads a genuine two-fold at CC 0.05, and the same operator over the population the pipeline itself pairs reads 0.88. The flattening and normalising the search does at its start is now one function, so an operator can be asked about without enumerating a point group around it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:21:41 +02:00
leonarski_fandClaude Opus 5	77390a847b	rugnux: a frame counts as indexed from six spots, the floor the setter already allowed ... The bar a frame had to clear to validate a lattice was 9, while the setter's own minimum was 6. On a crystal whose frames cannot carry 9 the difference is not a standard of evidence, it is arithmetic: the true lattice is found, indexes every frame that has the spots for it, and is then refused because too few frames could clear a count they never had spots to reach. A 7 A crystal with a median of eleven in-range spots per frame, a third of whose frames cannot reach 9 at all, validated on six frames of sixty and was thrown away - the lattice it named merges to completeness at its deposited resolution. The gate that carries the meaning is the fraction of frames that must validate, which is untouched: a wrong lattice does not index 20% of frames on six spots either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:10:07 +02:00
leonarski_fandClaude Opus 5	0d56a6219b	symmetry: a glide plane named from its own zone, and the Sohncke answer kept beside it ... A screw axis extinguishes a row; a glide plane extinguishes a zone. The search scored the first and had no candidate that could carry the second, so a small molecule whose cell and whose rotations were both found correctly stopped one glide short of its group. The zone is keyed by the rotation part of the improper operator, so a glide and its centring partner - the c and the n of the same plane - are one zone rather than two scored twice. A Sohncke group has no improper operator at all, so on chiral data the zone loop never runs and every Sohncke candidate scores what it scored before, bit for bit. That is why this is offered inline rather than behind a flag; it was still measured, on 140 protein datasets, and no protein answer moves and no non-Sohncke candidate reaches the top of the ranking. A glide is judged per reflection rather than on the zone's sum. The statistic is linear in the number of absences at fixed deadness, and a plane holds hundreds to thousands of reflections where a row holds tens, so a sum reaches hundreds of nats on a class that is merely a few times weak; per reflection the false and the genuine zones separate by a factor of nine. An unmeasurable zone refuses its candidate rather than abstaining, because a glide is an extra claim on top of a group that already fits without it. Both readings are reported on every run. A crystal of chiral molecules cannot have a glide, so a reader who knows the sample is a protein must be able to see the Sohncke group without processing the images again. A centre of symmetry is not offered at all: a candidate is enumerated only when its absences differ from a Sohncke candidate's, and an inversion centre predicts exactly what its Sohncke subgroup predicts. It cannot be claimed from intensities and this does not claim it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:09:26 +02:00
leonarski_fandClaude Opus 5	49938d2a24	symmetry: an axial absence is judged after the pseudo-translation modulating it is divided out ... A translational pseudo-symmetry splits every reflection into two classes by the parity of its index along the translation, one systematically strong and the other systematically weak. Where the translation is a half-integer along an axis, those two classes are exactly the absent class and the control class of a screw along that same axis - so the absence test divides one by the other and pays the suppression twice, reading a class that is present but suppressed as extinct, and buying a screw the crystal does not have. Measure the modulation along each axis from the intensities and divide it out before scoring. Per axis, not pooled: on the crystal this was found on it is 0.034 along the row carrying the false screw against 0.287 and 0.159 along the two rows whose screws are real, and a pooled estimate under-corrects the row that needs it while over-correcting the rows that do not. The violation counter is corrected with the evidence, because candidates rank on the sum of their zones and a correction that only ever lowers a score can never unseat a screw that has already been added. The false zone goes from 75.7 nats for its screw to 56.3 against it and from five violations in ninety to fourteen; the run adopts the deposited group. The two genuine screws on the same crystal pay 2.4 and 1.2 nats and gain no violations. On a crystal whose deposit puts a real screw on a row that also carries a half-integer translation, the correction fires, that screw pays 18.4 nats and still carries 59.5, and the verdict does not move. Across the corpus nine crystals engage it and one zone changes state. Only order-two screws are treated: a three-fold with a one-third translation is left unanswered rather than answered no. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 23:56:11 +02:00
leonarski_fandClaude Opus 5	8b9598c1c4	twinning: the L-test partners are chosen so a pseudo-translation cannot silence it ... The L-test compares a reflection with a partner a fixed step away, and the step it used preserves the parity class of a half-integer pseudo-translation - so a pseudo-body-centring was invisible to it by luck rather than by design. A translation of one third is not so lucky: it puts the two members of a pair in different modulation classes, inflates the statistic past the bound that is read as evidence AGAINST twinning, and the twin call is lost silently on a crystal that has one. Choose the partners so every pair stays inside one modulation class, which is what the half-integer case was already getting by accident. The obvious alternative - lengthening the step until it clears a third as well - was measured and rejected: it puts the statistic past that bound on more than two thirds of the corpus against a seventh today. Measured over a hundred and thirty-seven crystals: a hundred and twenty-nine unchanged, seven repaired, and exactly one verdict moves - a crystal whose twinning was being denied by its own pseudo-translation. On a synthetic perfect twin carrying a one-third translation, both indicators are destroyed before the change and both return after it. Nothing branches on this verdict, so no merged intensity moves; the whole effect is what the report says about the crystal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 20:09:00 +02:00
leonarski_fandClaude Opus 5	a4be5e6cd0	diagnostics: a translational pseudo-symmetry is detected and named ... A pseudo-translation makes one class of reflections systematically strong and the complementary class weak. It is the classic predictor of a failed molecular replacement, it raises the second moment where twinning lowers it so each masks the other's test, and rugnux did not look for it at all. Look for it: a native Patterson from the merged intensities, the largest off-origin peak taken as a fraction of the origin, and the vector refined against the data. Both halves are scored against a null computed for that crystal rather than a bound taken from elsewhere - the peak against intensities permuted within resolution shells, the modulation depth against the same search started from random vectors - because the noise floor of the first runs from 1.35 to 17.7 per cent across the corpus, so no fixed threshold could mean the same thing twice. Requiring both is what keeps the false-positive rate down: either alone calls one crystal in ten. A translation the data are exactly invariant under is a lattice vector, not a pseudo-symmetry. Recognising that removes eleven false calls from ninety-five merges folded in P1 - each reading 83 to 102 per cent of origin against a genuine maximum of 62 - and adds one true positive whose pseudo-translation lies underneath its own centring and is unreachable without it. Report-only: it gates nothing and changes no reflection, no scale and no group. It fires on eight of a hundred and thirty-seven crystals, and on those the verdict moves from OK to WARNINGS, which is the point of it. Median cost 55 ms, worst 1.9 s on the largest merge here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 20:08:40 +02:00
leonarski_fandClaude Opus 5	5b8ce26c83	integration: the flight path between the sample and the detector is corrected for, and named ... A reflection arriving at an angle to the detector normal crosses D/cos(alpha) of whatever lies between the sample and the sensor, not D, so it is attenuated more than one arriving head-on and reads low. That is the same geometry as the sensor crossing already corrected here and the opposite sign, and it was missing. The factor is exp(D/L*(1/cos(alpha)-1)) from the NIST attenuation coefficient of the medium, the stated distance and the stated wavelength. Nothing in it is fitted, and it is not justified by any measured amplitude: the flight path and the sensor crossing are collinear to better than 0.998 over the angular range any single experiment samples, so no fit of one can be evidence for the other. It is the tabulated absorption of a known thickness of a known material over a known path. The medium cannot be detected. No field of the NXmx application definition describes it, none of the masters this program reads carries one, and it cannot be inferred from the implied transmission either - in this corpus a station confirmed to use helium sits at 51% implied air transmission and one confirmed to use air at 63%, so any rule separating them is a threshold fitted between two points. It is therefore assumed, stated, and overridable: --flight-path air|helium|vacuum, defaulting to air. Helium is its own material rather than an alias for vacuum, attenuating about a six hundredth of air rather than nothing. On an untilted detector the correction is a function of resolution alone, so its entire effect on merged data is a shift in the Wilson B - which is what the report now prints beside the assumption, accurate to better than a tenth of an angstrom squared against measurement from 0.05 up to 28. Where that shift is large the report warns, because a wrong medium is then the largest number in the run: applied to data from the confirmed helium station it returns a B of 14 A^2 at 3.0 A resolution, which is not a value a crystal can have. The corpus contains its own control. One crystal, one station, three collections a quarter of an hour apart at falling energy through the same air: corrected, the Wilson B rises monotonically with the dose, as it must. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 17:55:43 +02:00
leonarski_fandClaude Opus 5	7c10d62dab	integration: the sensor efficiency is carried as its own quantity, not folded into the Lorentz-polarization factor ... It was multiplied into the per-reflection factor at prediction, so that factor held Lorentz, polarization and efficiency at once and the two spellings that reach a file - the wire key and the reflection dataset - meant something different from what they had meant the day before. The unmerged MTZ had to divide the two apart again at write time to fill its own columns, which is a good sign the wrong thing was being carried. Carry them separately. The prescaling factor is Lorentz and polarization again, what its name and both reference implementations mean by it, and the efficiency is its own field through prediction, integration, serialization and storage. Fifteen sites that want the total now multiply the two - once per reflection, not once per pixel. The efficiency is stored rather than recomputed on read, because the writer has no geometry to recompute it from, and because a file written before the correction existed would have had a radial trend invented for it. Sixty stored files were checked for the one combination that would be ambiguous - the old meaning of the factor beside a stored efficiency - and none carries it. Output does not move. Re-scaling a file written before the efficiency existed is byte-identical, which is a proof rather than a sample, since the stored factor is exactly one there. Where the efficiency is live, one product is reassociated - (L*Q)/P becomes (L/P)*Q - and about a third of the values differ in the last bit or two: every structural column is identical, so no reflection is gained, lost or reindexed, and no intensity in 1.4 million observations moves by as much as 1e-4 of its own sigma. The parity tests now compare the efficiency as well, and their non-vacuity guard watches it rather than the factor it left - which is the same guard that went blind when the efficiency was added to a field it was not watching. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 16:28:15 +02:00
leonarski_fandClaude Opus 5	e44ce069a2	docs: the polarization factor is credited, and the sign that sets its plane is documented ... The correction is Kahn's, and the paper was not cited anywhere. It is now, at the function and in the acknowledgements. The plane was already settable and only the API said so: a negative factor is a vertically polarized beam. Measured on the corpus, dividing by the factor as given collapses the azimuthal modulation of a real background from 27.6% to 0.77%, and asking for the wrong plane doubles it to 55.7% - so the convention is right and the sign is load-bearing. The usage text now says which sign means which plane. No dataset in the corpus declares a plane and every one that can be measured is horizontally polarized, so nothing here changes what any current run computes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:30:04 +02:00
leonarski_fandClaude Opus 5	6140de7249	rugnux: the unmerged MTZ carries the sensor efficiency in its own column, not inside LP ... LP is a column other programs read and un-apply. It is documented as the Lorentz-polarization factor, and until the sensor efficiency was folded into the same product that is what it held. Measured on our own unmerged file, LP spanned a factor of 1.1343 across the detector where pure L/P spans nothing of the sort - the excess is the efficiency, 13% end to end at 13 keV and 34% at 18 keV. Both reference implementations keep it out. Recomputing pure L/P from a stored XDS file's own geometry over its 124k reflections leaves RLP flat to 0.1% from 8.6 to 33.7 degrees, where a folded-in efficiency would have shown a 7% trend - and XDS has the sensor numbers in hand. DIALS fills LP from lorentz and polarization alone and writes QE as a separate column, even a column of ones. Split them: LP is L/P again, QE is the efficiency, and the intensity is unchanged, so a reader that un-applies LP recovers what it expects and one that wants raw counts divides by LP and multiplies by QE. Only the unmerged file moves - every other column is bit-identical and the file grows by exactly one float per reflection plus one header record. The merged files are byte for byte what they were. The process file gains an optional qe dataset beside the existing one rather than changing what that one means, so a file written before the efficiency existed still loads, and reads back as a correction of exactly one - which is what it was. It is stored rather than recomputed on read because the writer has no geometry to recompute it from, and because recomputing would have written a radial trend into every stored file that never had one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:29:45 +02:00
leonarski_fandClaude Opus 5	db4af06a2b	integration: the sensor efficiency reaches the stills GPU predictor too ... The efficiency correction was added to the rotation predictor on both the CPU and the GPU, and to the stills predictor on the CPU, but not to the stills predictor on the GPU - which the factory selects for stills whenever a card is present. So the correction was applied on a machine without a GPU and silently skipped on every machine with one: 6.6% apart on the geometry the tests use, 23.9% at 18 keV through a 450 micron sensor. The parity test that should have caught this is the reason it was not caught: it compares the predicted position and resolution of every reflection and nothing else, so a factor carried alongside them is invisible to it. It now compares the prescaling factor as well, with an assertion that the factor is not uniformly one so the comparison cannot pass vacuously, and the rotation path - which had no parity test at all - gets the same one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:29:22 +02:00
leonarski_fandClaude Opus 5	db9cc9106f	rugnux: the per-reflection correction factor is named for what it is, not for what it once held ... The factor multiplied into each integrated intensity was called rlp, for reciprocal Lorentz-polarization, and until this week that is all it held. It now also carries the sensor efficiency at the angle the beam arrives, and on the stills path it holds that efficiency and the polarization with no Lorentz term at all - correctly, since the Lorentz factor of a still is one. Three different products under one name that promises exactly one of them, in code where the neighbouring member is the total correction. Rename it prescaling_corr: multiplicative, applied before scaling, therefore not a scale, and silent about its contents - which is the point, since the contents have now grown twice. It is also what DIALS calls the same product. The stills refinement member spelled "1 / rlp" becomes inv_corr, and the comments and usage text that promised "the Lorentz-polarization factor and nothing else" now say what is actually there. The Lorentz term keeps its own name where it is computed, because that name is correct. The two external spellings are untouched: the CBOR key and the reflection dataset are a published format, and a reader that meets an unknown key would take the factor as zero, which both the merge key and the ingest treat as a reflection to drop - so every reflection would vanish and the run would still exit zero. No output changes: the merged and unmerged files of two full runs are byte for byte what the previous binary wrote, four stored files from before the efficiency correction still re-scale identically, and the reflection datasets of the process file are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 13:41:16 +02:00
leonarski_fandClaude Opus 5	d6ccdbdb61	rugnux: the report says what it found before it says how it found it ... The report was written in the order the pipeline runs, so a user reading one had to reach line 380 before meeting the first evaluative statement, and the warnings were last. Every run was 314 to 407 lines whether it succeeded or failed, about 200 of them fixed prose. The anisotropy section printed 21 keys and announced DETECTED (strong) on 54% of all runs - "strong" is the statistical confidence, which a reader takes as the severity - and reported a censored fit, which is survival-analysis vocabulary for good news. Assemble the report into a document and render from it, rather than streaming it out as the pipeline goes. That is what allows a verdict to sit above the evidence it was drawn from: SUMMARY carries VERDICT, one sentence of plain text, the warnings and ten facts, and it is composed after the sections that produce them. A clean run is 188 lines, a failed one 93, and the verdict is on line 25 in both. Warnings now carry a closed pathology vocabulary alongside their free text, so a consumer can switch on the code and a reader still gets the sentence. Everything removed from the default report is still written under --developer, which also carries the internals worth having when diagnosing the program rather than the crystal: the gate keys behind the anisotropy verdict, the operator and candidate tables, the sweep and spindle internals, and the essays. Nine statements the report made that were not true are fixed here as well. Among them: --mode scale printed a detector tilt of exactly zero on tilted data, which is worse than printing nothing because nothing about it looks wrong; every --mode scale run carried a "No image indexed" warning, because a key that is absent and a key measured to be zero were the same value; and a fitted resolution was asserted past the point where the run's own table shows CC1/2 at zero. REPORT_VERSION is 8. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 13:40:59 +02:00
leonarski_fandClaude Opus 5	9b96906150	integration: the parallax variance takes its attenuation length from the tables, not from lambda^3 ... The radial parallax broadening is the variance of the depth at which a photon converts, so it scales with the attenuation length. That length was approximated as photoelectric-dominated and scaled by lambda^3 from a single 13 keV reference per material. For silicon above 10 keV that is within 0.2%, but for CdTe it overstates the attenuation length by up to a factor of two, and by six above the cadmium K edge - which made this variance term 1.9x too large on 750 um CdTe data. Take the length from the same tabulated coefficients the efficiency correction uses. Silicon data is unaffected to within the approximation's own error; CdTe data gets a spot-width variance that matches the sensor it was recorded on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 11:07:17 +02:00
leonarski_fandClaude Opus 5	efa8ab1221	integration: each reflection is corrected for the sensor's efficiency at the angle it arrives ... A photon entering a flat sensor at an angle alpha to its normal crosses t/cos(alpha) of material instead of t, so the absorbed fraction rises toward the detector edge. The correction is QE(0)/QE(alpha) taken on the diffracted-beam direction against the detector normal, not on the scattering angle, so it follows a tilted or swung-out detector rather than assuming the two coincide. On an untilted detector this is a function of |s| alone: it is 99.7% a Wilson B offset and cancels exactly within a resolution shell, so merged protein data barely moves and no gain is claimed. It stops cancelling the moment the detector is tilted, because the incidence angle then acquires an azimuthal dependence: on a 30 degree swung-out geometry at 18 keV the within-shell spread reaches 21% median and 31% peak, and the anisotropy tensor moves with it. Attenuation lengths are the tabulated NIST coefficients rather than a wavelength-cubed approximation, which is within 0.2% for silicon above 10 keV but wrong for CdTe by a factor of two, and by six above the cadmium K edge. Photoelectric branching cancels in the ratio; K-fluorescence escape is not modelled, and the header says so. The correction self-disables where the physics makes it meaningless - an opaque sensor - so it needs no flag and is exactly neutral on all long-wavelength data and on thick CdTe. That also makes it a no-op on a file that stores its sensor thickness in the wrong unit, of which the corpus holds one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 11:07:01 +02:00