414 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	d4f280047f	grid scan: a raster reaches the spot engine, and the crystal cap says none rather than zero ... Two defects the merge created and one the API carried. Rugnux gated the per-image spot engine on AnalysisModeIsMX, so AnalysisMode::Grid fell through to the azimuthal-integration-only path: a raster ran, scored nothing, and reported no crystals. The gate now asks the stages table whether the mode does spot finding, which is the actual question - three modes need that engine for three different reasons, and a fourth would otherwise have to be remembered here too. max_crystals was a required integer defaulting to 10, with 0 meaning "all". Zero reads as "report no crystals", the opposite of what it did. It is now optional, and absent means no cap; a crystal found and then dropped is information the caller cannot get back. grow_score_threshold was missing from the schema entirely. Measured over the labelled corpus after these fixes: 34 of 34 confirmed-protein rasters yield a crystal, 0 of 8 water, 0 of 6 ice, 19 of 19 heldout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 08:22:14 +02:00
leonarski_f	de3fdf9534	Merge branch 'worktree-agent-ae76aa0c66b33a4db' into grid-scan-integration	2026-09-08 07:55:31 +02:00
leonarski_f	cb1d88dd4d	Merge branch 'worktree-agent-a4c58d76a0be29dd6' into grid-scan-integration ... # Conflicts: # rugnux/rugnux_cli.cpp	2026-09-08 07:55:31 +02:00
leonarski_f	36a082c2f8	Merge branch 'worktree-agent-aa55c81220288558b' into grid-scan-integration ... # Conflicts: # common/ScanResultGenerator.cpp # docs/CBOR.md # docs/HDF5.md # frame_serialize/CBORStream2Deserializer.cpp # frame_serialize/CBORStream2Serializer.cpp # writer/HDF5DataFilePluginMX.cpp # writer/HDF5DataFilePluginMX.h # writer/HDF5NXmx.cpp	2026-09-08 07:54:46 +02:00
leonarski_fandClaude Opus 5	748d14d2cd	file format: the ice ring ratio is stored and sent under its own name, and the retired one is still read ... Completes the rename the C++ identifiers already carried. Producers now emit ice_ring_ratio / ice_ring_ratio_mean over CBOR and write /entry/MX/iceRingRatio and iceRingRatioMean; both readers accept the retired ice_ring_score / iceRingScore spellings as well, so nothing that exists on disk or on the wire stops being readable. The REST plot_type gains ice_ring_ratio and KEEPS ice_ring_score, both mapping to the same plot, so no client breaks. --ice-min-score is deliberately unchanged: it is a threshold on the ratio, it is the one surface a user types, and this program has 91 long options and no aliases. The fallback is not a formality. rugnux --mode scale reads the stored per-image value to reproduce the ice gate the writing run applied, and an unread dataset does not fail loudly - ice_n == 0 sends the gate to its fail-safe branch, "ice present". Demonstrated on a clean crystal whose stored file uses the old names: with the fallback the gate reads 1.06 against a 1.50 threshold and skips ice handling; with the fallback removed the same file has 272056 of 1032493 reflections (26%) excluded from the scale fit, and ISa, R_meas, I/sigma, SIGANO and both error-model terms all move. That is a silent change to merged intensities on files already written, which is why the two new CBOR test cases were each checked to FAIL when their fallback is taken out rather than merely to pass. Verified: same binary on an old-name and a new-name copy of one file gives byte-identical .hkl, .mtz and unmerged .mtz and an identical report; a file written after the rename scales to the same bytes again; the master and the data-file read paths were each exercised with both spellings; and the live API serves the same plot under either name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:42:23 +02:00
leonarski_fandClaude Opus 5	273591af11	grid scan: a crystal is grown out of its seeds, so one weak grid point no longer splits it ... A crystal in the corpus was reported as two because the grid point joining its halves scored 0.498 - two thousandths under the threshold. That is an artefact of putting a hard edge through a continuous quantity, not a gap in a crystal. The patch search is now hysteresis. A patch is labelled at a lower grow threshold (0.35) and kept only if it holds a cell above the seed threshold (0.50), which is the same thing as growing out of the seeds in one pass of the labeller rather than two. A cell between the two levels joins a crystal that already exists but can never start one, so no lowering of the grow threshold can turn a background into a crystal - and the negatives are safe by construction, since water peaks at 0.146 and ice at 0.490, both under the seed level. Both halves of the small-patch rule read the GROWN patch - the count includes the cells growth added and the peak is the patch's best cell wherever it lies - which is stated at the test, because a reader will otherwise wonder whether a grown cell can rescue a patch no seed would have admitted. It cannot: the patch is discarded before that test unless it holds a seed. The reported extents are those of the grown patch, so they reach the 0.35 contour. Over the corpus that is major_um x1.04 and n_images x1.07 at the median (x1.11 and x1.14 at the mean, +7% cells overall), and the cells it adds are spread fairly evenly over 0.35-0.50 rather than piled at the bottom - a crystal edge sampled at one grid step, not bleed into a neighbour that never diffracted. The patch mean falls with them, and can now sit below the seed threshold; the peak beside it is what the admission was decided on. RASTER_REPORT_VERSION is 3, because N_CELLS and the extents are a different measurement under the same names. The corpus table is unchanged: 17/17 protein, 0/4 water, 0/3 ice, 10/10 heldout. The split heals into one crystal, and one raster's five patches become three. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:34:36 +02:00
leonarski_fandClaude Opus 5	98132d0f83	analysis: every analysis method carries its own settings, and a raster's indexing is one of them ... AnalysisSettings had begun collecting per-method parameters - the calibrant was already in it, and the grid thresholds were about to be. That makes the structure every method reads grow whenever any one method gains a knob, and it puts a field in front of readers for whom it means nothing. So: AnalysisSettings keeps what all methods share, which for now is the mode, and each method gets a class of its own bound the same way. GridScanAnalysisSettings holds the protein-score threshold, the minimum cells per crystal, the decisive single-cell score, the maximum crystals reported and the indexing switch. CalibrationSettings holds the calibrant and the ring source. Both sit on DiffractionExperiment outside the per-run dataset member, both have an Import/Get pair, and both have their own endpoint - /config/grid_scan_analysis and /config/calibration - which is how every other settings group in this API is already reached. Grid indexing is no longer fixed in the stages table. It was turned off there on cost grounds, and that reasoning does not hold: a raster runs at up to 100 Hz, which the FFT indexer keeps up with, and a fixed-target serial experiment with a known cell wants ffbidx on every cell, where a raster that indexes is most of the measurement. So it is a setting, and DEFAULTS ON. It is additive rather than a change of answer - blobs are still found on the protein score, so indexing alters nothing about which cells are called crystals and only adds what was found in them, including the per-cell lattice count, which is the cheapest multi-lattice or cracked-crystal signal there is. That makes indexing the one stage a mode does not decide. AnalysisModeStages still carries a value for it, but only as the setting's default, and DiffractionExperiment::GetAnalysisStages - which is what every gate reads - substitutes the configured one. The table row is marked so nobody reads it as the mode's answer. The calibration knobs stay coupled to the mode but the rule now lives with them: CalibrationSettings::ApplyToAzimuthalIntegration moves azimuthal integration onto the CPU and supplies sectors where fewer than four were asked for, carrying the reason with it - the FPGA integration core holds 2048 bins in total, so 32 sectors leave 64 q bins, which cannot locate a ring. Stated there because it will otherwise read as an FPGA defect to be fixed back onto that path, and it is not one: the core is sized for a detector at full rate, and a calibration exposure is a few images at a few Hz. Both imports apply it, so the order the mode and the calibration settings are set in does not matter. CalibrationMethod moves from image_analysis/geom_refinement/PowderCalibration.h into common/CalibrationSettings.h, which that header now includes. One enum, so the setting and the code consuming it are not two vocabularies; every existing user sees it unchanged. The grid thresholds have one home and it is this class. The raster work owns AnalyzeGridScan's parameter surface and carries PROTEIN_SCORE_THRESHOLD_DEFAULT / MIN_BLOB_CELLS_DEFAULT beside that header today; the header here states the signature that replaces them, so the two do not become competing defaults. The beam size deliberately stays a separate argument to AnalyzeGridScan: it is measured, not configured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:27:41 +02:00
leonarski_fandClaude Opus 5	0f00b76a9a	image analysis: the ice score takes the pipeline's own band width, and the ice quantities are named for what kind of number they are ... Two things, both about telling one ice quantity from another. The ice score's spot channel had its own band half-width of 0.02 A^-1 while the spot finder marks ice rings at 0.03 (ice_ring_width_Q_recipA). The 0.02 was justified by a 5 pp specificity gain measured on the PYTHON PROTOTYPE, which used a fitted beam centre and a mask-derived coverage table; the shipped port, which takes the geometry's centre and the azimuthal profile's own live pixel count, does not reproduce it. Measured over the corpus by truth class rather than by directory label, at 0.02 vs 0.03 on the combined score: ice loops 62.13/62.19%, _icy protein 89.03/89.79%, _clean protein 16.51/16.31%, water 17.19/20.03%. The widths are indistinguishable except on water, where one of the four loops is independently known to carry a full hexagonal pattern. So the width is now a parameter and the pipeline's own value is passed in - one band width, not two. The 0.012 tolerance in the radial channel is NOT a second band width, and is renamed CENTRE_SMEAR_Q to say so: it is how far either side the channel looks for the bin a mis-set beam centre moved the ring to. The rest is naming. Three kinds of number were all called score, or built from things called count, and a reader could not tell from the name whether 1 meant "none" or "certain" - which are opposite. The convention, now stated in docs/CPU_DATA_ANALYSIS.md: *_score is bounded [0,1] and 1 is certainty, *_ratio is unbounded and 1 is nothing, *_count is a count. The C++ identifiers for the ice ring ratio follow it (ice_ring_score -> ice_ring_ratio, GetIceRingScore -> GetIceRingRatio, PlotType::IceRingScore -> IceRingRatio), and the local in the scaling gate that shadowed the new ice_score while meaning the ring ratio is renamed with them. Nothing outside the source moved: the CBOR keys ice_ring_score and ice_ring_score_mean, the datasets /entry/MX/iceRingScore and iceRingScoreMean, the ice_ring_score plot type and the --ice-min-score flag are all unchanged, and were checked to be after the rename. Renaming those changes stored files, the stream format, the REST API and a CLI flag, and is a separate decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:19:03 +02:00
leonarski_fandClaude Opus 5	cdb60c2916	grid scan: one cell is a crystal when that cell is decisive, and the list can be capped ... The minimum patch size was a flat floor, so a crystal that covers one or two grid points was thrown away however strong its diffraction. It is now a floor OR a piece of evidence: a patch is reported when it has min_blob_cells cells, or when its best cell reaches decisive_protein_score. One condition, and both halves are parameters. The bar is the patch PEAK, not the patch mean. A two-cell patch with one strong cell and one marginal one is the case this exists for, and the mean averages exactly that evidence away. Over the 67 labelled rasters the two populations do not overlap: no water raster reaches a peak of 0.15 and no ice raster reaches 0.50, while the weakest protein raster peaks at 0.67 - so 0.6, the middle of that gap, is the default. The peak is reported beside the mean, in the table and in the JSON, so an admission can be checked against the number that decided it. max_crystals caps the returned list after the sort, best first; 0, the default, is all of them, because a crystal that was found and then dropped is information the caller cannot get back. The four parameters now travel as one GridScanAnalysisParameters, which is the shape the GridScanAnalysisSettings class in common/ is to take: when it lands it replaces this struct in the signature and nothing else changes. On the corpus this reaches 17/17 protein loops (it was 16/17 - the miss was a crystal covering two grid points of a 4x4 raster) with water still 0/4 and ice still 0/3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:15:30 +02:00
leonarski_f	cd9cc4800e	Merge branch 'rc167' into worktree-agent-ab3a3e173261a95d8 ... # Conflicts: # common/JFJochMessages.h # common/ScanResultGenerator.cpp # docs/CBOR.md # docs/HDF5.md # frame_serialize/CBORStream2Deserializer.cpp # frame_serialize/CBORStream2Serializer.cpp # frontend/src/components/DataProcessingPlot.tsx # tests/CBORTest.cpp # writer/HDF5DataFilePluginMX.cpp # writer/HDF5DataFilePluginMX.h # writer/HDF5NXmx.cpp	2026-09-08 00:24:42 +02:00
leonarski_fandClaude Opus 5	550f7c7d7b	scan result: a grid scan's crystals and a calibration's geometry reach the stream, the file and the API ... A grid scan and a powder calibration each produce an answer that is not a per-image number, and neither had anywhere to go. ScanResult gains two top-level optionals, grid and calibration, in the same idiom the three rotation_* members already use: a mode's payload is present when that mode ran and absent otherwise. images stays required, so every existing /result/scan client - the python client and the beamline scripts on it - is untouched. No oneOf or discriminator. The spec contains none today, so polymorphism is unexercised across all three generators at once, and the only thing it would buy is making "exactly one payload" structurally true rather than true by convention. powder_calibration_output has been fully specified and referenced by nothing since it was written; the calibration member reuses it. The crystal list travels the same three hops a per-image quantity does. In the CBOR end message grid_crystals is an array of maps keyed by field name - the shape a spot already uses - so a reader that does not know a field skips it and the record survives a field being added; the deserializer steps over an unknown key rather than reading its payload as the next one. In the HDF5 master it is parallel 1-D datasets under /entry/MX/crystals, one per field, because crystals are numbered rather than named and a group per crystal would put the index in the path. The reader reads them back so a stored raster re-opens with what it found. Nothing caps the list at one. A raster can find several crystals and the ordering carries the answer: sorted by score, best first. angle_deg is an axis, not a direction. It lies in [0, 180) and wraps, so 179 and 0 are adjacent and two of these do not average arithmetically - a pair of nearly parallel needles averaged across the wrap comes out as a right angle. That is stated at every layer it passes through, since it is the kind of field a consumer will reach for without reading the producer. The producer side is not wired yet: nothing fills ScanResult::grid or EndMessage::grid_crystals here. The seam is receiver/JFJochReceiver.cpp, where rotation_lattice is filled in EndDataCollection and GetFinalStatistics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:17:10 +02:00
leonarski_fandClaude Opus 5	a7d3ada3ab	analysis: what runs over the images is one stated mode, shared by broker, rugnux and viewer ... Until now nothing in the tree said what analysis a run performed. The answer was composed at each site out of four independent scalars - the detector type, two separate "spot finding off" switches, an indexing flag and a rotation flag - so what was configured and what actually ran were different things, and no single place could be read to find out which. AnalysisMode {None, MXRotation, MXStills, Azint, Grid, PowderCalibration} is that statement, in common/ because all three programs configure the DiffractionExperiment that carries it. AnalysisSettings sits on the experiment beside IndexingSettings, outside the DatasetSettings member, which is the one thing a /start replaces wholesale - so the mode is persistent by construction rather than by a rule someone has to remember. The mode does not label a run, it decides it. AnalysisModeStages() is a table - modes as rows, pipeline stages as columns - and every gate reads that table instead of testing the mode: spot finding in DiffractionExperiment::IsSpotFindingEnabled, indexing (and with it prediction and integration, which never run without a lattice) in one gate inside IndexAndRefine that serves all three front ends, azimuthal integration where the CPU engine is built. Two rows carry a judgement worth reviewing: powder calibration keeps spot finding, because --calibration spots fits the pooled spots; grid does not index, because a raster is thousands of frames and the per-image scoring it ranks on deliberately avoids an indexer that fires on ice. There is deliberately no Auto value. GetIndexingAlgorithm() resolves Auto at read time, which is exactly why an indexing setting cannot be read back off the configuration; removing that kind of implicitness is the point here, so the mode getter stays a plain accessor. MXStills is the default because None would silently switch analysis off on every deployment whose configuration predates the field. Rotation MX is absent from the OpenAPI schema rather than present and refused: jfjoch_broker has no rotation analysis path, so the REST and configuration-file routes cannot express it at all. The shared enum can still carry the value from elsewhere, so CheckAnalysisSettingsOnline refuses it on both routes with a message naming rugnux. A sweep collected under an MX mode is not refused - collecting rotation data online is normal and live spot counts are useful - but it is said out loud in the log, since the mistake worth preventing is the silence about what was done to it, not the acquisition. Powder calibration forces azimuthal integration onto the CPU and supplies 32 sectors where fewer than four were asked for. The FPGA integration core holds 2048 bins in total, so 32 sectors would leave 64 q bins - far too coarse to fit a ring. Frame rate is what this costs and a calibration exposure does not need it. The two existing "no analysis" switches, per-dataset dataset_settings.spot_finding and persistent SpotFindingSettings::enable, are interfaces in too many places to remove now. They are marked deprecated in the schema and in both headers, and the mode takes precedence over them: a mode that analyses no spots wins outright, while under a mode that does find spots they remain the finer control. The precedence is written where it is enforced. rugnux's ProcessMode is gone, replaced by the shared enum; RugnuxMode stays as the CLI spelling layer and no existing spelling changes. --mode gains mx_rotation and mx_stills, which are spellings of -R and --force-still rather than new switches; plain mx still chooses between them from the goniometer. scale keeps no shared counterpart, since it runs no analysis over images at all. The mode reaches the CBOR start message and /entry/MX/analysis_mode in the HDF5 master, so a written file records which analysis produced it. It is read back as provenance only - what a stored file was produced by is not what the next run should do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:16:26 +02:00
leonarski_fandClaude Opus 5	14e2bf3c0f	Merge rc167 into the detection-score branch ... Both lanes added a per-image scalar to the same eleven files, so every conflict was two additions competing for one line. All were resolved by keeping both, with three that needed more than that: - ScanResultGenerator: rc167 changed the per-image float vectors to resize(n, NAN) so a frame that never arrived does not read back as a real 0. v_protein_score and v_ice_score are exactly that case - 0 is a real answer ("nothing detected here") - so they take the NAN default too. - HDF5MetadataSource: rc167 established that NaN in a stored per-image array means "no value" and the optional must come back absent. The two detection scores now follow it, which they did not before the merge; without the guard a missing score would come back as a NaN that a threshold would silently compare against. - CBORTest: designated initialisers must follow member declaration order, so spindle_blind_fraction precedes the two scores in the DataMessage aggregate. Verified after the merge that every CBOR key that is encoded is also decoded (198 encoded keys, one intentional exception: the "type" discriminator), that both lanes' fields survive in the writer, the reader, the plots and the API, and that a stored file still round-trips. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:13:51 +02:00
leonarski_fandClaude Opus 5	21eca107fa	rugnux: --mode raster reads a stored grid scan and reports the crystals in it ... The offline entry point for a raster. It scores every image of a stored grid scan - spot finding for the per-cell resolution, no indexing, because a raster answers where the crystal is and not what its lattice is - hands the per-image protein and ice scores to AnalyzeGridScan, and writes what came back as two files beside the usual output: <prefix>_raster_report.txt in the idiom of the results report, and <prefix>_raster.json with the same content typed, so a battery can aggregate a sweep without scraping prose. --beam-size states the beam at the sample, defaulting to the file's incident_beam_size. It matters more than it looks: the reported crystal sizes are measured and still contain the beam, and taking an anisotropic beam back out is a subtraction of two covariance matrices, so a beam given as square when it is not rotates the reported crystal axis. --raster-protein-threshold and --raster-min-cells expose the two constants AnalyzeGridScan held at file scope, so a sweep can vary them without a rebuild; they are now defaulted parameters with the old values, and every existing call is unchanged. The observer feeding the analysis reads the file's own image number (DataMessage::original_number), not the loop's ordinal, so -s/-e/--stride cannot silently shift the grid mapping. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:01:39 +02:00
leonarski_f	3be28a99ad	Merge branch 'worktree-agent-aa55c81220288558b' into worktree-agent-ab3a3e173261a95d8	2026-09-08 00:01:07 +02:00
leonarski_f	294eabdb4b	Merge branch 'rc167' into worktree-agent-ae76aa0c66b33a4db ... # Conflicts: # viewer/JFJochViewerDatasetInfo.cpp	2026-09-08 00:00:55 +02:00
leonarski_fandClaude Opus 5	0c6167b7c5	viewer: a grid scan shows protein and ice over the shape of the loop, with a frame around each crystal ... Two scalar maps side by side never answered the question a raster is taken to answer. The grid-scan view gains a composite: the normalised background gives the loop its shape in grey, ice paints it cyan, protein orange, and where both are present the two mix in proportion. Protein and ice go in as they come - they are saturating scores already in [0,1], and stretching one would turn "no protein anywhere" into "protein everywhere". Only the background is normalised, with its range floored, so a scan of pure air stays flat instead of having its noise stretched into a convincing loop. Each crystal found in the raster gets a frame, drawn oriented: centred on the crystal, sized along its own principal axes and rotated by its angle. An axis-aligned box would hide the case the axes exist to expose - a 100 x 10 um needle at 45 degrees reads as a compact blob. The extent is in micrometres and goes through both step sizes separately, which a real 20 x 16 um raster needs. A blob within 1.2 of round is drawn as a circle, since its angle means nothing. The single-scalar path is untouched; the composite is one more entry in the metric combo, offered first for a grid scan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 23:59:56 +02:00
leonarski_fandClaude Opus 5	62ca2b24f6	image analysis: two per-image detection scores say whether there is protein and whether there is ice ... proteinScore and iceScore are saturating scores in [0,1] that answer presence, not quality: a superb crystal and a barely-diffracting one both read near 1, and neither the spot count nor the resolution enters either of them as a term. iceRingScore already reports a magnitude - a ratio, unbounded - which is not a number that can be thresholded; these are. The protein score counts distinct d SHELLS above 5 A rather than spots, so a parasitic ring in the low-resolution band cannot accumulate evidence, and weights each spot against the frame's own median so a scattering of the weakest detections cannot fill a shell either. The ice score carries two channels and takes the stronger: a radial one over the azimuthal profile, which runs the hexagonal and the CUBIC phase as separate hypotheses and decides between them at the end (flash-cooled loops show cubic or stacking-disordered ice at least as often as hexagonal, the two share only three lines, and dropping the cubic hypothesis costs about 5 pp on iced loops), and a spot one that reads an excess on the ice radii against the same band slid to every ice-free offset, which is what catches ice arriving as discrete crystallites and leaving the radial profile flat. Both read d out of the geometry, so both move with a beam-centre error; the centre is not fitted here, and the one they were computed with is written beside them as scoreBeamCenterX/Y so a later rescoring can tell an algorithm disagreement from a geometry one. Ported from validated prototypes and checked against them frame by frame on stored data: mean absolute difference 2.7e-5 (protein), 1.3e-8 (ice radial) and 3.4e-4 (ice spots). On a 41-loop battery the protein score reaches 98.4% of confirmed-protein frames and 0.00% of water frames, and finds no cluster on any water or ice raster. Cost is 0.01 ms/frame for the protein score and 0.08-0.32 ms/frame for the ice score. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 23:59:46 +02:00
leonarski_fandClaude Opus 5	ee0ad8d149	grid scan: a completed raster is reduced to the crystals in it, oriented and ranked ... AnalyzeGridScan takes a finished ScanResult plus its GridScanSettings and returns the list of crystals the raster hit, sorted by score so element 0 is the one to collect. Pure function - no I/O, no FPGA, no JSON. Today the list holds nought or one entry; N is the point of the shape. The per-image protein score is scattered back onto the display grid through Rearrange, which already knows the snake order, the vertical flag and the step signs, thresholded, and labelled into blobs. Each blob is then measured: - Centre is a weighted centroid, pulled towards the cells that diffract best. The pull uses the RANK of the resolution inside the blob, never its value, so a salt grain reporting an absurd 0.8 A weighs exactly what a genuine best cell weighs and cannot drag the centre however extreme its number. A cell with no resolution gets the lowest weight rather than being dropped. The centroid of a banana- or L-shaped blob can land outside the blob, where no image exists, so image_number is snapped to the nearest cell that was actually collected. - Second moments are taken in micrometres, not in cells. A 20 x 16 um raster is ordinary and moments in cell units give a wrong angle - eight degrees wrong on the staircase in the tests. The angle is an axis, so it lives in [0,180) and wraps there. - The axis DIRECTION comes from the eigenvector but the LENGTH from the projected extent, because "how far do I scan" is an extent question and the constant taking a second moment to a length assumes a shape a blob of five cells does not have. Where the two disagree about which axis is longer - a moment dominated by clumps at the ends - the extents are swapped and the angle turned a quarter turn, so major_um >= minor_um with angle_deg along it is an invariant a consumer can draw a frame from. - score is the MEAN protein score over the blob, not the peak: the score saturates, so the peak is 1.0 for every real crystal and ranks nothing. res_A is the 25th percentile, not the minimum, the minimum being precisely where a salt spot or a hot pixel shows up; it is NaN when nothing in the blob measured a resolution. Sizes are measured and the beam is left in them. The beam is already in the file as incident_beam_size, so a consumer can deconvolve reproducibly and reversibly instead of inheriting ours; the result carries the beam size so it says what the extents contain. The header records that removing an anisotropic beam is a covariance-matrix subtraction followed by re-diagonalisation, not a per-axis quadrature removal, which is silently wrong whenever the crystal is not aligned with the grid - the needle case this design exists for. Labelling is a small dense flood fill in common/, 8-connected. StrongPixelSet::sparseccl is the wrong abstraction for a dense grid map: sparse union-find over raster-ordered strong pixels, hardcoded module dimensions, a 4000-pixel cap, spot-shape acceptance, and an FPGA header. 8-connected rather than 4 because where the step is coarser than the beam a needle at 45 degrees lands as corner-touching cells; under 4-connectivity that breaks into single cells and the minimum-size rule then discards the crystal entirely, which is the case oriented axes exist to catch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 23:39:16 +02:00
leonarski_fandClaude Opus 5	b650cda037	Revert "integration: the background is fitted over the ring that survives, not averaged over it" ... This reverts commit 6a3ae9c65. The rotation battery refuses it. The median is clean - zero movement in R_meas and CC1/2, most crystals untouched or moving in the third decimal, which is what was predicted - but the tail is not a cost, it is a failure: one crystal's CC1/2 falls from 91.3% to 6.8% with its reflection and observation counts essentially unchanged, so the intensities are being corrupted rather than the weak ones discarded. Another loses seven points of R_meas and half its ISa; a third has its high-shell CC1/2 go negative; a fourth's ISa doubles, which is an error model measuring something different rather than measuring better. The change was predicted to cost one to two points of high-shell CC1/2 on crowded patterns, and that prediction was made on six geometries. Fifty-one say otherwise. The defect it fixes is real and stays on the record: within the outer background radius of the array edge the ring is truncated inward while the disk is not, so the surviving ring sits where the radial background is higher and the reflection reads low - measured at 1.79 counts per pixel four pixels from a border, and confirmed by a masked gap producing the same bias signed by the direction of its displacement. Fitting the background over the surviving ring is the right shape of answer. Doing it per reflection, from a ring that a neighbour mask has often already truncated, is evidently not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 16:30:22 +02:00
leonarski_fandClaude Opus 5	6a3ae9c658	integration: the background is fitted over the ring that survives, not averaged over it ... The signal disk and the background ring are concentric, which is the whole reason a linear background cancels between them. Within the outer ring radius of the edge of the sensor array that concentricity is gone: the ring loses its outer part while the disk barely loses anything, so what is left of the ring sits further into the detector, where the radial background is higher, and the reflection reads low. Measured at signal-free positions four pixels from a border: the ring reads 162.35 counts per pixel against a true background over the disk of 160.57, which over a hundred disk pixels is 182 counts of deficit, against 216 to 239 observed. <I/sigma> runs -1.83, -2.34 and -1.18 at nought to three, three to six and six to nine pixels from the border, and recovers exactly at the outer ring radius. The same reflection measured at a border reads 179 counts lower than in the interior over seven thousand matched pairs. A masked module gap does the same thing but signed by the direction of the displacement, which is why nothing has caught this: at a gap the two populations cancel in the mean, while at the sensor border the truncation is always inward, so the bias is always negative. The background is now the intercept of a straight line in radial offset over whatever ring pixels survive, read at the reflection's centre. Three extra sums per ring pixel and no extra reads; the radial distance was already computed there. It is exact under any truncation and reduces to the mean when the ring is whole, so it is unconditional rather than a mode: a badly truncated ring pays in sigma, through the variance the fit honestly reports, rather than in a rejection. On the crystal where this surfaced the outermost shell's correlation with a deposited model goes from -0.234 to +0.004, and the shell above it from -0.091 to +0.179. Correcting beats discarding: dropping every observation within fifteen pixels of a border reached only -0.019 and +0.127, because the corrected observations still carry signal. Interior reflections do not move. The cost is one geometry: where a neighbour mask has already truncated the ring almost everywhere, the fit roughly doubles the variance of the background estimate while finding no gradient worth removing, and a crowded small detector loses one to two points of CC1/2 in its finest shells. Also: the MINPK denominator counted only profile mass that lands on the detector, so a reflection whose peak is off the sensor scored a perfect one and no guard could fire. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 15:46:32 +02:00
leonarski_fandClaude Opus 5	e5c96cd41b	model validation: the placement target carries a bulk solvent that means something ... The rigid-body target refits the scale at every evaluation, deliberately - held fixed, the target would measure the scale as much as the placement, and the body would translate to repair a scale error instead of moving to where the density is. That refit was gemmi's unbounded fit, the one already replaced for the reported scale, and here it was worse: measured over a corpus of deposited models, 40% of the evaluations that decide where the model goes came out with a b_sol outside 10-80 A^2, on 57 of 89 datasets, ranging from -8072 to +1721. A negative b_sol is a solvent term that GROWS with resolution. One crystal ran its entire committed placement between 290 and 430 A^2, and that placement went into the reported maps. The bulk solvent is fitted once per zone instead, inside the same physical box the reported fit searches, and then held while the overall scale and the anisotropic B keep following the body. That split is the point: k_sol and b_sol describe the crystal's disordered solvent, not the fit of one placement, and measured across a whole zone they drift by a single grid step. Fitting them at every evaluation costs three times the wall clock, makes the scaler 82% of the run, and moves discontinuously under a forward difference - which is poison for a numerical Jacobian. This costs 4%, leaves no evaluation outside the box, and lands the body within 0.37 degrees of the expensive version, against 4.33 degrees for what it replaces. Placements change on fourteen crystals in eighty-nine. R-free is a wash on the mean; the step buys more total R-free from fewer commits, and the gain sits where the runaway actually bit. Two things found while auditing the file and left as they were, because they are right: the refinement sees working reflections only, end to end, and the gauge that removes the origin-free directions of a polar group carries more than half the whole refined translation in ninety-four of a hundred and thirty-nine polar zones. It had no test. It has one now. An observation with no calculated amplitude gets a zero residual, which drops it from the target rather than scoring it as a perfect fit, and is now counted and reported - a large count says the model's reflection conditions do not match the data's, which is a statement about the model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 15:27:30 +02:00
leonarski_fandClaude Opus 5	004fa5a781	model validation: the model says, shell by shell, where the data still carry signal ... A placed and scaled model gives an expected intensity for every reflection, and comparing that with the merged intensity is a measurement nothing else in the report can make: CC1/2 and R_meas compare the data with themselves, and can only say whether a shell is reproducible, not whether it is right. CC(model, data) is reported per shell, on the merge's own shells so the rows line up, with the reflection count and a significance beside each. Read it in one direction only. A correlation significantly above zero proves signal is present, because a model cannot invent agreement with data it never saw. A correlation near zero proves nothing, since the model may be the thing at fault - one crystal in this corpus reads negative across its outer shells against a deposited model of a different crystal form. So this can support keeping a shell and must never be used to discard one, and nothing in the pipeline acts on it. Intensities rather than amplitudes, matching CC1/2 and CCref beside it. The raw merged intensity rather than the French-Wilson amplitude: that amplitude is a posterior mean under a Wilson prior which pulls weak reflections towards their shell mean, and the shells this number is read in are exactly the weak ones, so it would correlate the prior with itself. The free flag is ignored on purpose. Nothing has been refined against these reflections, so the correlation is unbiased over all of a shell rather than over the test set alone - which is where the outer-shell reflection counts are, and with them the difference between a statistic that can resolve something and one that cannot. REPORT_VERSION is 7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 13:12:32 +02:00
leonarski_fandClaude Opus 5	cb08f63a52	model validation: the bulk solvent is searched inside its physical range, not fitted without bounds ... gemmi offers two scalers and we were using the one without bounds. Its Levenberg-Marquardt path has nothing stopping the flat-solvent parameters from leaving the range the model means anything in; the alternative path that does declare bounds is behind a compile guard we have never enabled. On this corpus six datasets in fifty-one fitted a b_sol outside it, the worst at 1707 A^2. What that does is subtler than a bad scale, and worth recording because it is why nobody noticed: a b_sol that large does not corrupt the solvent term, it switches it off - 1.4% of it survives at 10 A - so the model is simply scaled without a solvent contribution and the R-factors look unremarkable. k_sol and b_sol now come from a grid search over the physical box, with the scale and the anisotropic B refitted at each candidate pair, following Afonine et al. Refitting at each point is what makes it work: clamping the parameters after an unbounded fit costs up to 0.044 in R-free, because it leaves the scale and B where the rejected fit put them. Non-physical fits go from six in fifty-one to none, and both R-work and R-free come out slightly but significantly better rather than merely no worse. Which reflections are fitted remains the caller's business - the function scales whatever it is handed - so the working-set restriction of the previous commit is not something this can undo. A crystal with no solvent-accessible volume needs no special case: its mask is empty, so the solvent term is identically zero whatever the parameters say. There is a test for that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 10:34:21 +02:00
leonarski_fandClaude Opus 5	ac1e8b8603	tests: the report version pin and the screw-gate section follow the code ... Two test expectations that the changes under them made stale, and one overstated sentence in the docs. REPORT_VERSION was bumped to 12 without touching the line that pins it - the very line whose comment says a key added to the report is a contract change and this is where it has to be acknowledged. Acknowledged, and the absence of SOHNCKE_SPACE_GROUP on a fixture that is GIVEN its group rather than searching for one is now asserted too, since that is what makes the key's contract honest. The section documenting that the E^2 gate is what saves a screw from fabricated violations no longer holds, because there are now two independent defences: with the gate off, the absent class still sits at 2% of its own row, so the zone is dead per reflection and licenses the absence evidence to override the count. The section is kept rather than deleted - it now pins the second defence, and a future P2 there would mean the deferral has stopped licensing a zone that is genuinely extinct. The docs said SOHNCKE_SPACE_GROUP is written on every run. It is written on every run whose group came from the search; a run given its group with -S has no Sohncke candidate to name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 21:12:42 +02:00
leonarski_fandClaude Opus 5	485bbeaf71	rugnux: a cell given with -C is matched in any equivalent description ... Every candidate the indexer forms is Niggli-reduced and primitive; the reference cell was compared against them exactly as typed. For a centred lattice those are different numbers entirely - a conventional F-cubic cell reduces to a rhombohedron with 60-degree angles, a centred monoclinic cell to axes it shares none of - so the true candidate failed the 5% test and the run reported that it had found no lattice. A user who supplied the cell as it is published was worse off than one who supplied nothing. The reference is now expanded into the primitive lattices its six numbers could stand for, one per centring, each reduced the way a candidate is; a candidate matching any of them is kept. The cell as typed stays in the set, so the indexer that returns the basis it was handed is unaffected. The cost is real and stated: a wrong cell now gets several chances to match rather than one. The controls that hold are on data - deliberately wrong cells are still refused, and every cell that worked before gives the same answer to the digit - but a synthetic lattice can be made to match through a centring hypothesis it has no right to, and no volume test tightens that without excluding a true candidate this corpus already contains. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 20:26:38 +02:00
leonarski_fandClaude Opus 5	6773e8516d	symmetry: a metric two-fold the lattice search refused is asked of the intensities ... The Niggli character walk takes the first character its tolerance matches, and on a lattice that is nearly but not exactly hexagonal it matches hexagonal. Under a hexagonal holohedry no candidate point group can carry the two strongest two-folds the data actually have, so the search lands on the weakest one and the crystal is processed in a group of order two where it should have been eight. The metric group knows better than the character walk: it holds every rotation the cell can host, including the ones the named lattice system has no room for. Each of those is now put to the intensities directly - one operator, scored the way Stage A scores its own, on the same reflection population, with the same strong-reflection gate and the same E^2 cap. That last part is what makes the answer usable: normalising over the full resolution range of a merge whose outer shells are correlated junk reads a genuine two-fold at CC 0.05, and the same operator over the population the pipeline itself pairs reads 0.88. The flattening and normalising the search does at its start is now one function, so an operator can be asked about without enumerating a point group around it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:21:41 +02:00
leonarski_fandClaude Opus 5	0d56a6219b	symmetry: a glide plane named from its own zone, and the Sohncke answer kept beside it ... A screw axis extinguishes a row; a glide plane extinguishes a zone. The search scored the first and had no candidate that could carry the second, so a small molecule whose cell and whose rotations were both found correctly stopped one glide short of its group. The zone is keyed by the rotation part of the improper operator, so a glide and its centring partner - the c and the n of the same plane - are one zone rather than two scored twice. A Sohncke group has no improper operator at all, so on chiral data the zone loop never runs and every Sohncke candidate scores what it scored before, bit for bit. That is why this is offered inline rather than behind a flag; it was still measured, on 140 protein datasets, and no protein answer moves and no non-Sohncke candidate reaches the top of the ranking. A glide is judged per reflection rather than on the zone's sum. The statistic is linear in the number of absences at fixed deadness, and a plane holds hundreds to thousands of reflections where a row holds tens, so a sum reaches hundreds of nats on a class that is merely a few times weak; per reflection the false and the genuine zones separate by a factor of nine. An unmeasurable zone refuses its candidate rather than abstaining, because a glide is an extra claim on top of a group that already fits without it. Both readings are reported on every run. A crystal of chiral molecules cannot have a glide, so a reader who knows the sample is a protein must be able to see the Sohncke group without processing the images again. A centre of symmetry is not offered at all: a candidate is enumerated only when its absences differ from a Sohncke candidate's, and an inversion centre predicts exactly what its Sohncke subgroup predicts. It cannot be claimed from intensities and this does not claim it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-06 07:09:26 +02:00
leonarski_fandClaude Opus 5	64169bb04d	rugnux: a measured spot budget below the input floor degrades the run instead of killing it ... The rotation first pass measures how deep each image's intensity-ordered spot list still lies on the lattice and adopts that depth as the spot budget. The measurement can honestly land below ten - seen where a wrong header detector distance left indexing a compensating, uniformly scaled cell that matches only the few brightest, most central spots - but the setter rejects anything under ten, so the run died on its own measurement, reporting a parameter the user never set. The floor predates the estimator by a year and was written to guard user input; by provenance it is the stills fitting bound, one more than a viable cell needs, which a rotation pass that only tests a lattice already in hand does not owe anything to. Exempting the measurement entirely would be worse than the crash: the validation gate still needs that many indexed spots per frame, so a shorter list can never index anything and the pass would abort one confusing error later. So the floor gets a name beside its companion, keeps its job for input, and the measured budget is clamped to it before adoption - with a warning saying what a sub-floor measurement means: the geometry in the file or the lattice is wrong, not that the crystal is weak. The dataset that died now completes like its siblings with the cause named; the same data at the reference-refined distance indexes 99% of images and measures no budget cut at all; and runs whose budgets sit at or above the floor are report-identical, checked at the two lowest in the corpus. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 23:33:43 +02:00
leonarski_fandClaude Opus 5	8b9598c1c4	twinning: the L-test partners are chosen so a pseudo-translation cannot silence it ... The L-test compares a reflection with a partner a fixed step away, and the step it used preserves the parity class of a half-integer pseudo-translation - so a pseudo-body-centring was invisible to it by luck rather than by design. A translation of one third is not so lucky: it puts the two members of a pair in different modulation classes, inflates the statistic past the bound that is read as evidence AGAINST twinning, and the twin call is lost silently on a crystal that has one. Choose the partners so every pair stays inside one modulation class, which is what the half-integer case was already getting by accident. The obvious alternative - lengthening the step until it clears a third as well - was measured and rejected: it puts the statistic past that bound on more than two thirds of the corpus against a seventh today. Measured over a hundred and thirty-seven crystals: a hundred and twenty-nine unchanged, seven repaired, and exactly one verdict moves - a crystal whose twinning was being denied by its own pseudo-translation. On a synthetic perfect twin carrying a one-third translation, both indicators are destroyed before the change and both return after it. Nothing branches on this verdict, so no merged intensity moves; the whole effect is what the report says about the crystal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 20:09:00 +02:00
leonarski_fandClaude Opus 5	a4be5e6cd0	diagnostics: a translational pseudo-symmetry is detected and named ... A pseudo-translation makes one class of reflections systematically strong and the complementary class weak. It is the classic predictor of a failed molecular replacement, it raises the second moment where twinning lowers it so each masks the other's test, and rugnux did not look for it at all. Look for it: a native Patterson from the merged intensities, the largest off-origin peak taken as a fraction of the origin, and the vector refined against the data. Both halves are scored against a null computed for that crystal rather than a bound taken from elsewhere - the peak against intensities permuted within resolution shells, the modulation depth against the same search started from random vectors - because the noise floor of the first runs from 1.35 to 17.7 per cent across the corpus, so no fixed threshold could mean the same thing twice. Requiring both is what keeps the false-positive rate down: either alone calls one crystal in ten. A translation the data are exactly invariant under is a lattice vector, not a pseudo-symmetry. Recognising that removes eleven false calls from ninety-five merges folded in P1 - each reading 83 to 102 per cent of origin against a genuine maximum of 62 - and adds one true positive whose pseudo-translation lies underneath its own centring and is unreachable without it. Report-only: it gates nothing and changes no reflection, no scale and no group. It fires on eight of a hundred and thirty-seven crystals, and on those the verdict moves from OK to WARNINGS, which is the point of it. Median cost 55 ms, worst 1.9 s on the largest merge here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 20:08:40 +02:00
leonarski_fandClaude Opus 5	5b8ce26c83	integration: the flight path between the sample and the detector is corrected for, and named ... A reflection arriving at an angle to the detector normal crosses D/cos(alpha) of whatever lies between the sample and the sensor, not D, so it is attenuated more than one arriving head-on and reads low. That is the same geometry as the sensor crossing already corrected here and the opposite sign, and it was missing. The factor is exp(D/L*(1/cos(alpha)-1)) from the NIST attenuation coefficient of the medium, the stated distance and the stated wavelength. Nothing in it is fitted, and it is not justified by any measured amplitude: the flight path and the sensor crossing are collinear to better than 0.998 over the angular range any single experiment samples, so no fit of one can be evidence for the other. It is the tabulated absorption of a known thickness of a known material over a known path. The medium cannot be detected. No field of the NXmx application definition describes it, none of the masters this program reads carries one, and it cannot be inferred from the implied transmission either - in this corpus a station confirmed to use helium sits at 51% implied air transmission and one confirmed to use air at 63%, so any rule separating them is a threshold fitted between two points. It is therefore assumed, stated, and overridable: --flight-path air|helium|vacuum, defaulting to air. Helium is its own material rather than an alias for vacuum, attenuating about a six hundredth of air rather than nothing. On an untilted detector the correction is a function of resolution alone, so its entire effect on merged data is a shift in the Wilson B - which is what the report now prints beside the assumption, accurate to better than a tenth of an angstrom squared against measurement from 0.05 up to 28. Where that shift is large the report warns, because a wrong medium is then the largest number in the run: applied to data from the confirmed helium station it returns a B of 14 A^2 at 3.0 A resolution, which is not a value a crystal can have. The corpus contains its own control. One crystal, one station, three collections a quarter of an hour apart at falling energy through the same air: corrected, the Wilson B rises monotonically with the dose, as it must. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 17:55:43 +02:00
leonarski_fandClaude Opus 5	7c10d62dab	integration: the sensor efficiency is carried as its own quantity, not folded into the Lorentz-polarization factor ... It was multiplied into the per-reflection factor at prediction, so that factor held Lorentz, polarization and efficiency at once and the two spellings that reach a file - the wire key and the reflection dataset - meant something different from what they had meant the day before. The unmerged MTZ had to divide the two apart again at write time to fill its own columns, which is a good sign the wrong thing was being carried. Carry them separately. The prescaling factor is Lorentz and polarization again, what its name and both reference implementations mean by it, and the efficiency is its own field through prediction, integration, serialization and storage. Fifteen sites that want the total now multiply the two - once per reflection, not once per pixel. The efficiency is stored rather than recomputed on read, because the writer has no geometry to recompute it from, and because a file written before the correction existed would have had a radial trend invented for it. Sixty stored files were checked for the one combination that would be ambiguous - the old meaning of the factor beside a stored efficiency - and none carries it. Output does not move. Re-scaling a file written before the efficiency existed is byte-identical, which is a proof rather than a sample, since the stored factor is exactly one there. Where the efficiency is live, one product is reassociated - (L*Q)/P becomes (L/P)*Q - and about a third of the values differ in the last bit or two: every structural column is identical, so no reflection is gained, lost or reindexed, and no intensity in 1.4 million observations moves by as much as 1e-4 of its own sigma. The parity tests now compare the efficiency as well, and their non-vacuity guard watches it rather than the factor it left - which is the same guard that went blind when the efficiency was added to a field it was not watching. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 16:28:15 +02:00
leonarski_fandClaude Opus 5	6140de7249	rugnux: the unmerged MTZ carries the sensor efficiency in its own column, not inside LP ... LP is a column other programs read and un-apply. It is documented as the Lorentz-polarization factor, and until the sensor efficiency was folded into the same product that is what it held. Measured on our own unmerged file, LP spanned a factor of 1.1343 across the detector where pure L/P spans nothing of the sort - the excess is the efficiency, 13% end to end at 13 keV and 34% at 18 keV. Both reference implementations keep it out. Recomputing pure L/P from a stored XDS file's own geometry over its 124k reflections leaves RLP flat to 0.1% from 8.6 to 33.7 degrees, where a folded-in efficiency would have shown a 7% trend - and XDS has the sensor numbers in hand. DIALS fills LP from lorentz and polarization alone and writes QE as a separate column, even a column of ones. Split them: LP is L/P again, QE is the efficiency, and the intensity is unchanged, so a reader that un-applies LP recovers what it expects and one that wants raw counts divides by LP and multiplies by QE. Only the unmerged file moves - every other column is bit-identical and the file grows by exactly one float per reflection plus one header record. The merged files are byte for byte what they were. The process file gains an optional qe dataset beside the existing one rather than changing what that one means, so a file written before the efficiency existed still loads, and reads back as a correction of exactly one - which is what it was. It is stored rather than recomputed on read because the writer has no geometry to recompute it from, and because recomputing would have written a radial trend into every stored file that never had one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:29:45 +02:00
leonarski_fandClaude Opus 5	db4af06a2b	integration: the sensor efficiency reaches the stills GPU predictor too ... The efficiency correction was added to the rotation predictor on both the CPU and the GPU, and to the stills predictor on the CPU, but not to the stills predictor on the GPU - which the factory selects for stills whenever a card is present. So the correction was applied on a machine without a GPU and silently skipped on every machine with one: 6.6% apart on the geometry the tests use, 23.9% at 18 keV through a 450 micron sensor. The parity test that should have caught this is the reason it was not caught: it compares the predicted position and resolution of every reflection and nothing else, so a factor carried alongside them is invisible to it. It now compares the prescaling factor as well, with an assertion that the factor is not uniformly one so the comparison cannot pass vacuously, and the rotation path - which had no parity test at all - gets the same one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:29:22 +02:00
leonarski_fandClaude Opus 5	db9cc9106f	rugnux: the per-reflection correction factor is named for what it is, not for what it once held ... The factor multiplied into each integrated intensity was called rlp, for reciprocal Lorentz-polarization, and until this week that is all it held. It now also carries the sensor efficiency at the angle the beam arrives, and on the stills path it holds that efficiency and the polarization with no Lorentz term at all - correctly, since the Lorentz factor of a still is one. Three different products under one name that promises exactly one of them, in code where the neighbouring member is the total correction. Rename it prescaling_corr: multiplicative, applied before scaling, therefore not a scale, and silent about its contents - which is the point, since the contents have now grown twice. It is also what DIALS calls the same product. The stills refinement member spelled "1 / rlp" becomes inv_corr, and the comments and usage text that promised "the Lorentz-polarization factor and nothing else" now say what is actually there. The Lorentz term keeps its own name where it is computed, because that name is correct. The two external spellings are untouched: the CBOR key and the reflection dataset are a published format, and a reader that meets an unknown key would take the factor as zero, which both the merge key and the ingest treat as a reflection to drop - so every reflection would vanish and the run would still exit zero. No output changes: the merged and unmerged files of two full runs are byte for byte what the previous binary wrote, four stored files from before the efficiency correction still re-scale identically, and the reflection datasets of the process file are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 13:41:16 +02:00
leonarski_fandClaude Opus 5	d6ccdbdb61	rugnux: the report says what it found before it says how it found it ... The report was written in the order the pipeline runs, so a user reading one had to reach line 380 before meeting the first evaluative statement, and the warnings were last. Every run was 314 to 407 lines whether it succeeded or failed, about 200 of them fixed prose. The anisotropy section printed 21 keys and announced DETECTED (strong) on 54% of all runs - "strong" is the statistical confidence, which a reader takes as the severity - and reported a censored fit, which is survival-analysis vocabulary for good news. Assemble the report into a document and render from it, rather than streaming it out as the pipeline goes. That is what allows a verdict to sit above the evidence it was drawn from: SUMMARY carries VERDICT, one sentence of plain text, the warnings and ten facts, and it is composed after the sections that produce them. A clean run is 188 lines, a failed one 93, and the verdict is on line 25 in both. Warnings now carry a closed pathology vocabulary alongside their free text, so a consumer can switch on the code and a reader still gets the sentence. Everything removed from the default report is still written under --developer, which also carries the internals worth having when diagnosing the program rather than the crystal: the gate keys behind the anisotropy verdict, the operator and candidate tables, the sweep and spindle internals, and the essays. Nine statements the report made that were not true are fixed here as well. Among them: --mode scale printed a detector tilt of exactly zero on tilted data, which is worse than printing nothing because nothing about it looks wrong; every --mode scale run carried a "No image indexed" warning, because a key that is absent and a key measured to be zero were the same value; and a fitted resolution was asserted past the point where the run's own table shows CC1/2 at zero. REPORT_VERSION is 8. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 13:40:59 +02:00
leonarski_fandClaude Opus 5	7e7103f49f	spindle: the blind-cone score compiles on MSVC, and the report version test says 7 ... Two CI failures from the spindle series, both self-inflicted. The severity used M_PI, which is not standard and which MSVC does not define, in a file under image_analysis - a tree the Windows viewer builds. common/JFJochMath.h exists for exactly this and is now included. The same defect was found and fixed in the post-refinement residual earlier in the day; the grep that confirmed it looked at a worktree that did not yet carry this file, so the second instance survived. The report-version test pins the version deliberately - adding a key to the report is a contract change and that line is where it has to be acknowledged - but the key added with the spindle report keys bumped the constant to 7 without the test following. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 20:00:20 +02:00
leonarski_fandClaude Opus 5	307987c865	rugnux: the mounting's cost is computed exactly from the measured group, not guessed from an angle ... Offline, a merged rotation run has what a still lacks - a determined point group and an exact indexed orientation - so the run-level number no longer needs the pessimistic presumed-diad bound, and it no longer uses the nearest-axis angle either. That 15-deg warning heuristic was wrong in both directions: an aligned in-plane 2-fold of a dihedral group is repaired by the principal axis, a cubic group is never severe in any orientation, and a lone diad perpendicular to the spindle is severe with no axis anywhere near the spindle at all. The group's proper rotations are applied to the sweep's blind double cone in the crystal's actual orientation, and what no operator maps onto measured territory is counted, weighted by each shell's own cone width so the result is a fraction of unique reflections to this run's resolution limit. Friedel and the improper operators need no separate handling - the cone and the measured region are both inversion-symmetric. The number is machine-readable on purpose: SPINDLE_LOST_UNIQUE_FRACTION in the report (0-1, a bare number a pipeline can act on) and /entry/MX/spindleLostUniqueFraction in the master, with the warning prose only on top of it, fired when the group recovers less than half the cone's content. REPORT_VERSION stays 7: the format's own rule is that adding a key does not move it. This also settles what the nearest-axis keys hedged: with the measured group the mounting is cleared or convicted exactly, so their documentation now calls them descriptive and points at the new key for the verdict. Verified against Monte Carlo: P1 loses 2.0% of unique reflections at theta_max = 15 deg with nothing repaired; a lone diad on or perpendicular to the spindle repairs nothing; an axis of order >= 3 perpendicular to the spindle repairs everything; 622 with an in-plane diad on the spindle loses nothing; cubic loses nothing in any orientation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	26a82ddc72	spindle: the per-image severity travels the whole data path, and absence travels with it ... The score was computed and then thrown away - carried on the per-image message but transported nowhere - so the automation it exists for could not read it. It now flows like bkg_estimate at every layer: CBOR (per-image key, END-block run mean and per-image array), HDF5 (per-image /entry/MX/spindleBlindFraction in the data files, the array and spindleBlindFractionMean in the master), read-back into a re-opened dataset, the scan result, the receiver plots, the REST plot and scan_result schemas, and the viewer and frontend plot menus. Absence is load-bearing and every transport keeps it distinguishable from a measured zero: the CBOR key is simply missing, the HDF5 array holds NaN, and read-back turns NaN back into an absent optional rather than a value. A pipeline that read 0 where the truth is "no value" would take exactly the wrong action - a measured 0 says one sweep loses nothing, absence says nobody could look, and the second must engage the recovery protocol while the first must not. The round-trip tests pin all three states through CBOR and through a written-and-reopened file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	bc3d693721	spindle: three trigger states, and the threshold is geometry rather than tuning ... The score exists so beamline automation can engage a recovery protocol - a two-sweep collection, a goniometer reorientation - with no human in the loop, so its canonical reading is fixed in one place with nothing for a beamline to tune: engage at 0.5, do not below, and NO VALUE is a third state that automation must treat as engage. The error costs are asymmetric - a false negative leaves the data permanently short, a false positive costs minutes of beamtime - and a frame nobody could measure must not be read as a frame measured safe. The 0.5 is derived, not tuned: the score is monotone in the folded miss-angle, so any threshold is a fold-angle gate, and 0.5 gates at fold <= 0.4040 * theta_max (verified root). Engaging on any overlap at all would gate at fold < theta_max, whose perpendicular band alone spans sin(theta_max) per row - 26% of orientation space at 15 deg - and unions over a frame's rows to well over half of all mountings, degenerating the trigger into "always"; at 0.5 the residual missed loss stays below half the cone. Also correct the message-field comment that said the whole cone means "1 - cos theta_max of every shell": an inner shell loses 1 - cos theta(d) at its own, smaller theta(d). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	89962574ef	spindle: the severity no longer rides on the indexing seed or on which indexer is configured ... The score gated on 60 spots, but the seed escalation stops at the leanest seed that indexes - 30 spots on precisely the clean frames a grid scan produces - so the value was absent exactly where beamline automation most needs it, and absence maps to "engage": the protocol would have fired on every good frame, which degenerates the trigger into "always". The floor itself stays where it was calibrated; what changes is what it gates. When no escalation pass could answer, one severity-only pass runs over the full spot list - the row search alone, no reduction, no refinement - purely to produce the number. The same was true of the indexer choice: only the FFT family computes a row shortlist, so a deployment configured with the known-cell indexer - the ordinary online stills path - never produced the score at all. Where the severity-only pass has no row search to run, the severity is read off the rows of the winning lattice instead, which any indexer produces: the lattice's shortest few distinct directions, as many as the FFT shortlist resolves in practice, fed through the same window and scoring with equal magnitudes. The count parity is load-bearing - a worst case over every enumerable lattice direction fires on 100% of harmless mounts of a generic triclinic cell against 74% for this selection at theta_max = 15 deg, and an always-firing trigger decides nothing - while the diad-detection rate stays 1.00 on the monoclinic classes either way, a dropped axis row being recovered by the pair normals exactly as an invisible one is. A frame that neither indexed nor reached the spot floor still reports nothing, which is the honest answer and maps to the recoverable error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	0ee1e5e070	spindle: an axis too long to see still shows its direction in the normals of the rows that are not ... The length window is deliberate - a row 2.5x the crystal's shortest is not a plausible symmetry axis in a crystal that small - but it made the score blind to a lone 2-fold on an axis LONGER than the window: a monoclinic crystal with a long unique axis, mounted near-perpendicular at an unlucky azimuth, returned a confident 0 rather than a refusal. The axis is recoverable without ever seeing its row: the normal to two direct-lattice rows is itself a reciprocal-lattice row, and a symmetry axis is parallel in the direct and reciprocal bases, so cross(a, c) is the unique-axis direction whatever the length of b. The normals of the strong in-window row pairs are now scored alongside the rows themselves, with row_length_A = 0 marking a direction the frame inferred rather than measured. Measured on a synthetic lone-diad crystal with a 300 A unique axis over random mounts, the fraction of severe mounts reported severe at the 0.5 trigger rises from 0.60 to 1.00, the engagement rate on harmless mounts of that class does not move, and the recovered direction reproduces the true axis exactly (every visible row is perpendicular to it). Also state the shortlist-consistency calibration on its per-crystal basis - 22 independent mounts, not the several hundred frames they contributed - and carry the conditioning the perpendicular case needs: an axis of order >= 3 there fully repairs the cone (measured 0.000 unrepaired for orders 3, 4, 6 against 1.000 for a diad), which a still cannot know, so the lone diad stays the operative worst case and the bound stays deliberately pessimistic on higher-symmetry crystals. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	c477cf0be3	spindle: a row perpendicular to the axis is as blind as a row on it ... The blind region a sweep leaves is a double cone, so a 2-fold sends it to two places: 2*beta away, and 180-2*beta away. The severity took only the first, and scored a row perpendicular to the spindle as 0 - "symmetry repairs everything" - when such a 2-fold in fact carries the cone onto its opposite lobe, which the sweep leaves equally unmeasured. Folding the miss-angle to min(beta, 90-beta) covers both images and reproduces a Monte-Carlo of the true overlap to 0.002. The failure was silent and in the dangerous direction, and it fired on the more common geometry: for a random axis the perpendicular band is several times wider than the aligned one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	28a98e1a14	stills: a per-image number for how much of a sweep's blind cone this orientation loses ... A rotation sweep never records a double cone of half-angle asin(lambda/2d) about the spindle. That loss is normally repaired by the point group; it is not repaired when an operator's axis lies inside the cone, because the cone then maps onto itself. A still cannot know the point group, but it can see where the crystal's short lattice rows are, and a symmetry axis is always one of them - measured over 107 solved cells its length is 1.4x the shortest row at the median. The number is the fraction of the blind cone that a 2-fold about the nearest short row carries back into the blind cone: two equal caps of angular radius theta_max whose centres are 2*beta apart, which is the circle-lens area in beta/theta_max and within 0.035 of the spherical value even for a 55 deg cone. 0 means one sweep reaches everything symmetry could give; 1 means the row is on the spindle and the whole cone is lost coherently. No goniometer geometry enters, so the number describes the problem and leaves the remedy to the beamline. Free: it rides on the FilterFFTResults shortlist the indexer already builds, needing only the spindle, the wavelength and the frame's own resolution. FilterFFTResults gains an optional out-parameter for each row's peak prominence, which the length window is taken over. Measured on 384 stills of 22 solved crystals against the frames' own symmetry axes, with the same procedure re-run along five decoy directions per frame as the null: 0.93 of severe orientations reported severe, 0.010 of harmless ones reported severe, AUC 0.948, and the value tracks the true severity to 0.10 at the 90th percentile. Below 60 spots the misses triple, so that is where it stops answering. It is blind to a symmetry axis much longer than the crystal's shortest row: measured on synthetic stills the search grid finds a 150 A row every time, a 200 A row half the time and a 300 A row once in sixteen. Two cheap consistency tests refuse to answer for part of that regime and cost nothing on real frames, but they do not cover it - the reach is a documented property of the number, not something every frame can detect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-03 23:57:12 +02:00
leonarski_fandClaude Opus 5	491167c90d	rugnux: a Bravais class the reduction decided by rounding is re-asked on the metric's own cell ... The class is named by Niggli-reducing the indexed cell and looking the reduced cell up in the 44 lattice characters, and that lookup is a coin flip for any lattice whose Buerger cells straddle the Niggli type-I/type-II boundary. An F-centred cubic lattice does so by construction: it has reduced forms on both sides, the two sides carry different characters, and which side the reduction lands on is set by the last digits of whatever refinement produced the cell. Measured over 600 perturbations of one such lattice: 43% cubic F, 36% tetragonal I, 21% orthorhombic I, and the split is flat over a factor of ten in the noise. The class then caps the point-group search, so from the body-centred sub-cell the cubic three-fold is never enumerated and the run reports that nothing was refused - which is accurate, because nothing was asked. Le Page's two-fold search has no such key: it measures each rotation's obliquity on the lattice itself, in a primitive basis. LePageLattice turns the rotation group it finds into a conventional cell, a centring letter and an integral change of basis, and where that group is larger than the adopted class's holohedry the merge is reindexed into that cell and the space-group search is run again there, on both merges, with every gate live. Nothing here decides: the reindex is committed only where the search in the new setting confirms a strictly higher point group AND the centring the new cell describes, so a pseudo-symmetric metric leaves the answer already in hand standing. Measured as a paired battery over 113 rotation datasets: the re-ask fires on 7 and adopts on 1, and that one crystal - an F-centred cubic lattice the reduction had named body-centred tetragonal - moves to its deposited group, gaining 0.10 A of resolution and 2.8x the multiplicity at R_meas 0.117 -> 0.120. Nothing else moves, in space group, resolution, CC1/2, R_meas, multiplicity, I/sigma or completeness. A second such crystal, named body-centred orthorhombic, is offered the same cubic cell and confirms all 23 added operators at CC 0.96 with an H ratio of 1.00, and is still refused, on the merge chi^2 ratio at 2.50x a bound of 1.85. Letting H rescue that refusal is the one-line change an earlier round measured and rejected - it promotes the synthetic P 4_3 2_1 2 in the test suite to point group 432 - so it is not here, and that crystal is left where it was. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-03 13:53:14 +02:00
leonarski_fandClaude Opus 5	c77ebe56c3	reader: a one-element string array is a scalar ... Masters exist that store sensor_material, description and the compression name as shape (1,) rather than as true scalars - JUNGFRAU files from an early beamline deployment do, and the application definition permits it. ReadString() required rank 0 and threw on anything else, so the whole file was lost: the metadata parse never finished and no image was ever examined. Accept any shape holding exactly one element, and keep throwing for a string dataset that genuinely holds several, which is a different quantity and cannot be read into one std::string. The numeric path already worked this way (HDF5DataSet_scalar_stored_rank1); this is the string half. Measured: three long-wavelength rotation datasets that could not be opened at all now process end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-03 10:58:38 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00