1174 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	bddbec7aff	rugnux: measure the canonical pass's geometry probe before the scaling engine is built ... The post-refinement probe on a canonical pass only measures, and it reads nothing the scaling engine produces - the reflections' hkl, I, sigma and positions are final once the images are integrated, and the engine writes back per-frame fields only. It ran after the first merge, beside the engine's arrays, where its gathered observations and rocking events set the pass's memory high-water mark. Run it before the engine is built instead; the pre-pass keeps its original place (it consumes the smoothed mosaicity the merge writes back). Measured on a 3600-frame long-axis rotation set: the canonical pass's high-water drops from 14.8 GB to under 12.8 GB (sampled RSS, -N 6); merged MTZ/HKL/CIF, P1 MTZ, per-image table and the report are byte-identical, the probe's own result included. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 18:27:14 +02:00
leonarski_fandClaude Opus 5	c58510edba	RotationScaleMerge: ingest without the index-order copy of the sort keys ... Ingest wrote every observation's 24-byte SortKey into an index-order array and then scattered that array into its h buckets - two key arrays alive together at the top of the ingest. Count the buckets per chunk of frames straight from the source reflections and scatter from them instead: a frame's reflections are consecutive in the key numbering, so the chunks are consecutive index ranges and each bucket lands in the same index order as before. The key array is byte-identical. BuildInRangeObservations also hands back the remap temporaries (old runs, per-run maps, new_idx) before the observation array is built rather than at the end of the function. Measured on a 3600-frame long-axis rotation set (95.7 M partials in the geometry pre-pass): the pre-pass ingest high-water drops 15.45 -> 14.1 GB (sampled RSS, -N 6); merged MTZ/HKL/CIF, the P1 cross-check MTZ and the per-image table are byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 18:27:14 +02:00
leonarski_fandClaude Opus 5	e6e506809f	Bragg integration: size each image's reflection list to what is kept ... Finalize reserved npredicted reflections and kept only the ones whose fit succeeded. On a dense long-axis pattern more than half the predicted reflections lose their background ring, so each retained per-image vector carried more dead capacity than data for the rest of the pass. Count the kept ones first and reserve exactly that. No result changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 18:27:14 +02:00
leonarski_fandClaude Opus 5	bd386ba0f2	tools/battery: a set with a known-wrong reference can be left unscored ... Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 18:27:14 +02:00
leonarski_fandClaude Opus 5	aaca5cc481	rugnux: with Friedel mates merged separately, a thin hand takes its outlier median from the pair ... The rot3d merge tests each observation against the weighted median of its reflection, and only from three observations up: at two the median is one of them. With -A each hand of a Friedel pair is its own reflection, so a P1 anomalous merge sits at about two observations a hand and almost nothing is tested - on one P1 sweep the merge rejected 0 observations with -A against tens of thousands without it. A single observation a hundred times its partner's intensity (300 sigma apart, a 3.9 A reflection) then survived into the merge, carried 94% of the weighted variance of its CC1/2 bin, took that bin from 0.80 to 0.18, and the automatic cut with it: 3.46 A and UNUSABLE, where the same data without -A cut at 1.40 A. A hand with fewer than three observations now takes the weighted median of both hands together, where the pair has three. I(+) and I(-) differ by the anomalous signal, a few percent of I and far inside the six-sigma test, so the mate supplies the observations the hand is missing. A hand with three of its own keeps its own median exactly as before, and a Friedel-merged run is unchanged. The medians are formed on the host, so the device merge reads the same ones. Measured with -A: the P1 sweep 3.46 A / UNUSABLE -> 1.49 A (1.63 A XDS reference; 1.40 A without -A, unchanged), 0 -> 3,699 rejections. Two lysozyme and one insulin set: same cut, space group, CC1/2 and ISa. On one of the lysozyme sets 16 more observations are rejected, every shell's CCanom is unchanged, and the overall CCanom goes 0.31 -> -0.02: the whole-range figure was being carried by a handful of wild pairs, while the shells ranged -18% to +12% and XDS's overall anomalous correlation is 1%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 18:27:10 +02:00
leonarski_fandClaude Opus 5	a4d1b8eb69	rugnux --model: MapToFPhi returns the conjugate, as gemmi's transform does ... gemmi::transform_map_to_f_phi conjugates the forward FFT at the end (a structure factor is the sum over exp(+2 pi i h.x), a forward FFT the sum over exp(-2 pi i h.x)); MapToFPhi did not, so every model-path Fcalc and Fmask came out with its phase negated. Amplitudes, and so the R-factors and the scale, were unaffected; the phases of the maps and everything read off them (map coefficients, density at atom centres) were not. Caught by ModelValidation_MapToFPhiMatchesGemmi, which now passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 16:08:23 +02:00
leonarski_fandClaude Opus 5	600497cc67	One process-wide lock for FFTW planning ... FFTW's planner (every fftwf_plan_* and fftwf_destroy_plan) shares global state and is not thread-safe; executing a plan is. FFTIndexerCPU and BeamCenterFFTCPU each guarded their planning with a lock of their own, TranslationalNCS and the viewer's spectrum with none, and ModelFFT with a third - which does not stop two of them planning at once. common/FFTWPlannerLock.h holds the one mutex they all now take. No numerical change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 16:04:23 +02:00
leonarski_fandClaude Opus 5	5d931222d7	rugnux --model: evaluate the rigid-body Jacobian's columns in parallel ... The rigid-body target is an Fcalc + solvent mask + scale re-fit per evaluation, and its forward-difference Jacobian made six of them one after the other after the central one - 7 of the ~8 evaluations per LM iteration. The six shifted evaluations now run in parallel, column j on its own Evaluator over its own copy of the model (an evaluation moves every atom, so two cannot share one), set to the same zone and given the central evaluation's bulk-solvent pair - the pair the serial loop's shifted evaluations used, since the zone's solvent is fitted on the zone's first evaluation and every Evaluate starts with the central one. Each column's arithmetic is the serial one and the Jacobian is assembled in column order, so the refinement is bit-identical; the evaluation count is kept as the serial loop kept it (up to and including a failing column). The per-zone solvent fit (FitModelScale) gets the thread count as well. Inside a null replicate (a pool worker) the columns run inline, as before. To check: RIGID_BODY_* and MODEL_* keys and md5 of maps/.mtz/_model.cif identical with and without this commit on the audit set; rigid-body seconds on a large model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 13:03:37 +02:00
leonarski_fandClaude Opus 5	47bc15c62c	rugnux --model: drop two fits nothing needed ... A rejected rigid-body placement put the atoms back and then recomputed the full Fcalc and solvent mask at the restored positions. Nothing reads them: the real model's ModelState is only fitted again after its own compute_model_factors (the re-placement on the merged indexing, the null-failure fallback), and a null replicate's state is discarded. The atoms are still put back; the structure factors are left as they are. Where the lattice has twin laws, the indexing probe fits the identity labelling first, and place_and_fit then made the identical fit again - same ModelState (nothing moves the model between the two), same reflections. The probe now hands that Fit over, for the real model when the identity wins and always for a null replicate (placed against the merged indexing). fit_model is deterministic, so the reused Fit is the one the second call produced. Bit-identical by construction: no value that is read changes. To check: MODEL_* keys and md5 of maps/.mtz/_model.cif identical with and without this commit on the audit set, including sets with twin laws and a null. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 13:03:22 +02:00
leonarski_fandClaude Opus 5	2f727b1975	rugnux --model: fit the bulk-solvent grid points in parallel ... FitModelScale fits ~113 (k_sol, b_sol) grid points, each a full Levenberg-Marquardt over every working reflection, and it is the largest single cost of --model (every fit_model call: the first fit, the twin-law probe, the fit after the rigid body, each null replicate). It ran on one thread. Each point starts from fit_isotropic_b_approximately(), which sets k_overall and b_star from the data and the point's solvent pair alone, so no point depends on the one before it. The points of each pass (coarse, then the refinement around the coarse winner) now run in contiguous chunks, each on its own copy of the Scaling, and the winner is read off afterwards in grid order with the serial rule (lowest finite R, first on a tie). Each point's arithmetic is unchanged, so the result is the serial loop's bit for bit. Where fit_isotropic_b_approximately() has five or fewer reflections to fit on it returns without setting anything and points would chain, so there the grid is still walked serially on the caller's Scaling. Inside a null replicate (a pool worker) the chunks run inline, as before. To check: MODEL_* keys and md5 of the maps/.mtz/_model.cif identical with and without this commit on the audit set; model-phase time on a fit-dominated set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 13:03:22 +02:00
leonarski_fandClaude Opus 5	e6adf8d650	rugnux --model: model structure factors with FFTW instead of pocketfft ... Every map -> structure-factor transform on the model path - Fcalc from the model density and Fmask from the bulk-solvent mask, in the fit, the rigid-body evaluations (two per evaluation, ~120 evaluations per placement), the frame probe and the stills model reference - went through gemmi's vendored pocketfft, single-threaded. They now go through MapToFPhi (rugnux/ModelFFT.{h,cpp}): an FFTW r2c (fftw3f, already fetched and linked) planned with the guru interface on the same layout gemmi uses (u fastest, w halved), scaled by V/N, into the same FPhiGrid, so gemmi's prepare_asu_data() extracts the reflections exactly as before. gemmi's vendored code is not modified. The F -> map transforms of the output maps are left on gemmi. Plans are FFTW_ESTIMATE only (MEASURE times candidates and could choose differently between runs), made under a mutex - FFTW's planner is not thread-safe, and the null replicates and the parallel Jacobian call this concurrently - cached per grid size for the life of the process, and executed with fftwf_execute_dft_r2c on fftwf_malloc buffers (new-array execution needs the planning arrays' alignment, which a std::vector does not promise). NOT bit-identical with pocketfft: the two libraries order their arithmetic differently, so structure factors move in the last float bits and everything downstream (scale fit, rigid body, R-factors, maps) can move in the last printed digit. Verify by tolerance, separately from the parallelisation commits: MODEL_* keys equal to printed precision and the same MODEL_FIT / hand / indexing decisions on the audit set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT	2026-09-19 13:03:22 +02:00
leonarski_fandClaude Opus 5	848de4b7dc	rugnux: the geometry walk continues only while re-integration realises a gain ... The rotation geometry walk kept re-integrating while the post-refinement at the adopted geometry still committed a move, up to four rounds. It never ended on its own: every walking run hit the cap, and with the cap lifted to twelve every one hit twelve. Each round's fit predicted a held-out gain the re-integration at its geometry then did not realise - the realised residual stayed flat from the first round, or passed a minimum and rose again while the distance crept on along the distance/cell-scale degeneracy - so the cap decided where on the drift the run stopped. A round is now kept only when the held-out residual measured by the pass re-integrated at its geometry falls below the best round's by more than the standard error of the difference. The first round that does not ends the walk, and the run goes back to the best round (one more pass, since the canonical files were written by the last). The walk also stops when re-indexing pushes the fitted cell back on the same length in the same direction on two rounds running: the fit keeps asking for a cell nothing ratifies. The cap is 8; a walk still paying at the cap keeps its last (= best) round, sets geometry_not_converged, and the report flags GEOMETRY_NOT_CONVERGED. Either way the kept round is judged against the header-geometry pass by RefinedPassIsWorse, as before. PASS_DECISION lists distance and realised residual per round and why the walk ended. Runs whose post-refinement does not commit a move of more than one step do not walk and are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:37:55 +02:00
leonarski_fandClaude Opus 5	c6f97460dd	PostRefine: report the realised held-out residual, its noise and the starting cell ... Each post-refinement now records, beside what it commits, what the integration it was handed realises at the geometry it started from: the held-out residual at nominal (the same value the log prints on the left of "held-out ... ->"), its standard error over the held-out residual values, and the cell the pass's own indexing refined. Nothing here changes a decision. Two helpers read them: HeldOutResidualFell (a pass at a new geometry realises a lower residual than another by more than the standard error of the difference) and ReindexPushesCellBack (re-indexing at a committed geometry returns, on the length the fit moved most, a cell on the side the fit moved away from). The geometry walk in RunAllPasses uses both in the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:37:55 +02:00
leonarski_fandClaude Opus 5	0a7308c604	rugnux: the device merge before the correction surfaces sees the collapsed-scale drop ... DropCollapsedFullScales zeroes the corr of fulls whose frame scale collapsed on the host copy of the fulls; on the resident-fulls (GPU) path that zero reached the device only with the corrected corr pushed after the surfaces. The merge that measures CC1/2 before the correction surfaces runs in between, so on the GPU it still merged the dropped frames' fulls while the host path excluded them - the two builds measured different observations. The corr is now pushed to the device right after the drop, and the later push covers only the surfaces and the frame rejection. This changes the input of the two-pass quality guard (cc_half_before_corrections) on GPU runs where a full scale collapsed; the final merge is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:32:01 +02:00
leonarski_fandClaude Opus 5	bcdaac3b14	rugnux: integrate a sparse rotation lattice the pooled spots accept ... A weak crystal inside a powder of its own microcrystals plus ice puts only ~5% of each frame's spots on its lattice. The pooled first-pass test accepted that lattice (5.5% of validation spots against 0.6% at a wrong spindle angle), but then: - every frame failed the 20% per-frame floor (LATTICE_MIN_INDEXED_FRACTION), so nothing was integrated and the merge was skipped (indexing rate 0); - the background-measured beam centre, ~3 px from the file's and correct, was not adopted because the arbiter counts validation frames, which are 0/60 at every centre. Changes: - When fewer than 1/6 of the validation frames clear the per-frame floor AND the sweep's pooled on-lattice fraction is itself below that floor, integrate every frame from the sweep's lattice (as XDS/DIALS do) and leave frame selection to scaling. Sweeps sparse only in spots per frame keep the floor. - Merge when a rotation lattice was found even if no frame "indexed" on its own. - When neither centre indexes a validation frame, adopt the measured centre if its pooled excess over chance beats the file's by 3.29 sigma. On the case above: P2 at the XDS cell, merged to 1.77 A, CC1/2 0.95, ISa 3.4 (XDS with the same lattice: ISa 3.3-5.5). Three healthy/partially-indexing rotation sets are bit-identical; a two-wavelength CBF set that currently merges 42 frames on a wrong cell moves (beam centre adopted, 755 frames). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:31:14 +02:00
leonarski_fandClaude Opus 5	d3cd834f01	rugnux: print ISa as undetermined when the error model's b is not resolved from zero ... The (a, b) error-model fit already refuses to report ISa when its top intensity bin has no leverage on b. It still printed 1/b when b had leverage but came out at zero within its own error - the strongest bins scatter no more than counting statistics say (the fitted b^2 is negative and clamps to 0), or there are too few samples for the bin medians to mean anything. ISa then reports the noise in b as an I/sigma: on one low-resolution sweep consecutive merges of the same data gave ISa 0, 28, 46, 113 and 130. The standard error of b^2 is taken from the 16 bins' own scatter about the fitted line, and when b^2 is less than two standard errors above zero the merge result carries isa_resolved = false. Only what is printed follows it: the report's ISA key reads "undetermined", the summary line says so, the asymptote is not printed and the mmCIF carries "?". The fitted ISa itself is unchanged and is still what the space-group search's present-reflection cut and the refused-point-group arbitration read, so no decision moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:30:52 +02:00
leonarski_fandClaude Opus 5	0a342fcb13	rugnux: weight every CC1/2 by the information a reflection carries ... The resolution cutoff, the shell table and the overall CC1/2 counted every unique reflection equally. The merge itself is inverse-variance weighted, so an observation from a frame the crystal barely diffracted on enters it at 1/G^2 of a good one - honestly, with its sigma - but a reflection measured only on such frames is scaled-up noise that then counts as much as a well-measured pair in every Pearson CC1/2 read off the merge. On a sweep where half the frames are weak the curve collapses at every resolution: the cut lands at 3.7 A on a P1 crystal whose good frames reach 1.5 A, and the UNUSABLE verdict fires (CC1/2 0.40 beside I/sigma 10). Each merged reflection now carries cc_weight: the precision its half-sets would have had with every observation at the run's typical frame scale, over the precision they have. G_ref = sum G^3 / sum G^2 over the usable observations is the precision-weighted typical scale, which the dead frames cannot drag down however many there are; the factor per observation is max(1, (G_ref/G)^2), with G the frame's total scale (partial scale, flux and the fulls' own G) taken before the correction surfaces and before collapsed frames are dropped, so nothing intensity- or resolution-dependent enters it. On a sweep without a weak stretch every weight is 1 and the CC1/2 is the plain Pearson it was. The cutoff fit, the shell table and the overall CC1/2 (and so the UNUSABLE verdict and the report's shell checks) all read the same weighted statistic. The two extra per-group sums are accumulated on both merge paths, the host loop and MergeAccumKernel, from one per-frame factor array; a host recompute of the device sums agrees to 1e-15 relative on every merge after the corrected corr is uploaded, and a host-merge run gives the same cut, space group, CC1/2 and ISa on five sets. Weighting by the half-set error variance alone (1/(v0+v1)) is not this: v grows with the intensity, so it weights the weak end of the intensity distribution and biases homogeneous data coarser. Measured (written resolution, together with the weighted outlier median): a P1 sweep with a long weak stretch 3.73 -> 1.40 A against a 1.63 A XDS reference, UNUSABLE withdrawn (overall CC1/2 0.40 -> 0.97); a second 3.61 -> 3.00 A; one with most of the sweep out of beam 7.35 -> 5.20 A. Lysozyme, thaumatin and two insulin sets unchanged to 0.01 A (one lysozyme sweep with a weak wedge 1.13 -> 1.16 A, from the median), same space groups throughout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:30:52 +02:00
leonarski_fandClaude Opus 5	aff6492119	rugnux: weighted median for the rot3d merge outlier test ... The per-reflection median the merge rejects outliers against was an unweighted median of the scaled intensities. On a sweep where the crystal barely diffracts over a long stretch, observations from those frames are scaled up by 1/G together with their sigmas, and in P1 at multiplicity ~3 two of them outvote one well-measured observation: the median becomes noise and the well-measured observation is rejected against its own small sigma. On one P1 sweep with ~44% such frames this rejected 52,574 observations (2,131 once those frames are dropped) and took the correlation of the merged intensities with an external reference from 0.92 to 0.68 at low resolution. The median is now weighted by 1/(sigma*corr)^2, which leaves it unchanged where the observations are comparably precise: rejections 3,932, correlation 0.92. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:30:20 +02:00
leonarski_fandClaude Fable 5.1	34834d00ba	rugnux: the report's shell check compares the fit with the shell the crossing lies in ... FITTED_RESOLUTION was withheld whenever the fit was finer than the d_min of the last shell whose CC1/2 still reached the target. A shell's CC1/2 is an average over the shell, so on every ordinary fall-off the crossing lies inside the first shell whose average is below the target - 0.01-0.1 A finer than the number the check allowed - and the report said "no resolution could be fitted" on data whose fall-off is textbook: 68 of 100 battery datasets, with the fit 0.01-0.02 A inside the first sub-target shell. The comparison is now against that shell's d_min, and the RESOLUTION_FIT flag fires only where the fit is finer than the shell the crossing can lie in. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-19 11:30:20 +02:00
leonarski_fandClaude Opus 5	49541101ad	Twin-immune zone evidence for the operators a promotion adds (report-only) ... Under a twin law T a reflection whose twin mate is itself (up to the true group and Friedel) is untouched at every twin fraction. For each index-2 subgroup H of the adopted point group, those are the reflections centric in the group and acentric in H: centric if the operators the group adds over H are real, acentric if they are a twin law or a pseudo-symmetry - the one intensity statistic that still separates the two at fraction 0.5, where every operator statistic reads "real". Read on the P1 cross-check merge: epsilon-1 reflections in shells with <I/sigma> >= 5 (noise inflates every class towards centric), each class normalised against its own mean in bins of ~100 reflections and within the two phase classes of a detected pseudo-translation, Wilson outliers above E^2 = 20 dropped. Reported per subgroup: the added operators, n, <|E^2-1|> +- SE read absolutely against 0.968 / 0.736, and the centric-over- acentric Wilson log-likelihood in nats with both densities convolved with each reflection's error (flooring E^2 at its sigma instead read a genuine 1.2 A lysozyme zone as acentric), with the acentric control beside it. Log, report prose and TWIN_ZONE_n keys. Nothing reads it back; no decision changes. On the reference sets: 6toc P4222, all three subgroups centric (1.04-1.07, +253 to +534 nats); 6iu9 P3121 over P31 acentric (0.789, -294 nats); 5j23 R32 over R3 acentric (0.805, -304 nats, tNCS-class normalised); lysozyme P41212 centric (0.90, +476 to +1297); a P21 myoglobin centric (0.930, +135). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:29:57 +02:00
leonarski_fandClaude Opus 5	c657204a72	R-free flags keyed on the lattice holohedry ... The free-set hash was keyed on the Laue ASU of the merging group, so a twin law - a lattice symmetry the crystal lacks - put nearly every free reflection's twin mate in the working set (measured: 97-98% of the free-touching twin pairs mixed, on the main P3121 output of 6iu9 and on every law of the 6toc P1 cross-check), and each file of one crystal carried a different free set (P1 vs merged agreed on 88-92% of reflections). With the cell given, the key is now the reflection's orbit, Friedel mate included, under the lattice holohedry: the metric point group of the cell (gemmi Le Page two-folds, 3 deg obliquity, lattice of the cell's own basis vectors so every file gets the same group; this contains the merging group). Twin mates share a flag, and the merged MTZ/mmCIF, the P1 cross-check and any subgroup re-merge carry one free set (nested where the small-data floor lifts the fraction of one file more than another). This is phenix.refine's default (use_lattice_symmetry). Where the cell does not carry the merging group, the merging group's key is used as before. The small-data floor still counts reflections of the merging group, so the free fraction is unchanged. Reference free sets are untouched. On 6toc, 6iu9, 5j23: mixed twin pairs 0 for every law; P1 and merged flags agree on 100% (6iu9, 5j23) and nested on 6toc; free fraction 0.050-0.051 as before (6toc merged 0.086, floor unchanged). Intensities and space groups identical to rc171. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:29:57 +02:00
leonarski_fandClaude Opus 5	7ae1da277a	Space-group search: log the refusal the two-merge rule overrides ... When the all-observation and Lorentz-filtered search merges disagree, the higher point group is adopted. The losing arm's answer was dropped, so a promotion one arm refused on its twin gates (H ratio, added-operator R) went through with nothing in the output saying a gate had fired - and the report printed the adopted group's failing H ratio ("2.02 against a bound of 1.85") as if it were a margin. Seen on an H3 partial twin (5j23), refused on both twin gates by the all-observation merge and promoted to R32 by the filtered one. Now the losing arm's refusal (group and reason), or the fact that it never confirmed the operators, is logged as a warning and written to the report (SPACE_GROUP_ARM_OVERRIDE). A bound the adopted group exceeds is printed as "EXCEEDS its bound - that gate fired and was overridden", for the H ratio and for the best-operator R. The comment that claimed both arms keep their vetoes now describes the rule as implemented. The rule itself is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:29:57 +02:00
leonarski_fandClaude Opus 5	e715d3999e	Twinning: normalised L-test, and one verdict read in every Laue class ... The L-test now divides each intensity by its resolution-shell mean before forming pairs, over the shells whose <I/sigma> reaches 1 (the floor the second moment already used). Two index steps are not the same resolution, and on a small cell with a steep fall-off the raw pairs read <|L|> up to 0.08 high: an untwinned crystal 0.568 and a partial twin 0.427 where phenix.xtriage reads 0.486 and 0.360. Normalised, the four reference merges (6toc, 6iu9, 5j23, a lysozyme) agree with xtriage within 0.02 (0.487, 0.376, 0.374, 0.480). Selection stays by shell, never by the reflection's own I/sigma, which biases <|L|> down. Equal-count shells were tried for the second moment and widen its gap to xtriage, so the shells are kept. The L-test is no longer switched off in holohedral Laue classes. Merging I(h) with I(Th) under a false operator gives (I(h)+I(Th))/2 for every twin fraction, the perfect-twin distribution, so <|L|> < 0.42 there is reported as an adopted operator averaging unequal intensities (promotion suspect, or a twin law absorbed into the point group) - a warning, not a veto. Over 42 holohedral merges of the corpus the genuine ones read 0.436-0.513, the over-promoted H3 twin 0.374, and one ~490 A-axis crystal 0.365. The verdict is one line (TwinningVerdictLine) used by the stats text, the report prose, the summary row, the warning and the viewer; a new TWINNING_VERDICT key names it. The twin fraction comes from the statistic that carries the verdict (the L-test unless the call rests on the second moment alone), is not quoted from a second moment under a detected pseudo-translation, and is not quoted at all on a merge under a suspect operator. The pre-search numbers (measured on the P1 search merge) are user-visible, are measured with that merge's own pseudo-translation declared, and leave out the reflections the lattice centring extinguishes - an R lattice in its hexagonal cell otherwise pairs present with absent reflections (5j23: 0.633 with them, 0.40 without). Report-only: no space-group decision, merge or intensity changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:29:57 +02:00
leonarski_fandClaude Opus 5	2c5f26bf7a	tools/battery: the 16 keV thaumatin reference obeys Friedel's law ... At 16 keV a native crystal has no usable anomalous signal, so XDS's CORRECT was rerun with FRIEDEL'S_LAW=TRUE and the set no longer runs with -A. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:24:32 +02:00
leonarski_fandClaude Opus 5	a20e27a69f	tools/battery: the like-for-like table covers XDS's own range ... --report-resolution now gets XDS's own d_min, the range its CORRECT.LP totals cover, so the comparison is exact; shells past rugnux's cut print as not merged. The derived reference d_min stays what rugnux's own cut is scored against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:22:49 +02:00
leonarski_fandClaude Opus 5	9b3eea72fe	tools/battery: one rugnux run per set ... --model and --report-resolution are both processing-inert (merged reflections bit-identical with and without them), so the bare/model and bare/xds variant pairs collapse into one command per arm: open: rugnux -o p --no-export-unmerged --model <deposited coords> <input> (no --model where there is no deposited model) inhouse/private: rugnux -o p --no-export-unmerged [-A] --report-resolution <reference d_min>,<XDS d_max> <input> Every row is rugnux's own result (own cut vs the reference, as `bare` was), and on the XDS arms also carries the REFRES_* table, scored against CORRECT.LP's completeness, multiplicity, R_meas, CC1/2 and ISa, with ISa and R_meas ratio plots and a like-for-like table; REFRES_SHELLS_PAST_LIMIT > 0 marks the row as coverage (rugnux's cut coarser than the reference). Nothing is forced on the processing any more. On the open arm the space group is scored on the data's own determination (SOHNCKE_SPACE_GROUP where SPACE_GROUP_ENANTIOMORPH is ASSUMED_FROM_MODEL), the model's label kept as sg_label; R_FREE/R_WORK/CC_MODEL /MODEL_FIT are trend fields (placement-only, own free set); REFMAC stays opt-in. results.json schema 3: one row per set, no variant/first_read; older runs are read through their bare rows (open-arm model R-factors folded in), so compare and report keep working against them. Every time is now a first read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:18:10 +02:00
leonarski_fandClaude Opus 5	16bb4da2bf	rugnux: the run's own N_obs and R_meas stop at the same cut as N_uniq ... The own statistics table's grid bottoms 0.1% below the finest kept reflection, and the observation-level counts (N_obs, R_meas, the Bijvoet split behind SigAno and CCanom) re-walk the fulls, which still hold every ingested group. Groups in that band just below the auto cut were erased from the merge, so they were not in N_uniq, but their observations still counted in N_obs and R_meas of the finest shell. The own table now floors those counts at the cut by group d, the rule the erase applies - the same floor the reference-range table already used (now passed as a double, so the comparison is exactly the erase's). Report-only: on an auto-cut cubic in-house set the written MTZ, HKL, P1 MTZ, unmerged MTZ and image table are byte-identical; TOTAL_OBSERVATIONS 378754 -> 377933, MULTIPLICITY 40.31 -> 40.22 (now equal to the reference-range table's), finest shell N_obs 60873 -> 60052, R_meas 1136.9% -> 1129.2%, CCanom -0.6% -> -0.3%; overall R_MEAS, CC1/2, SigAno unchanged at printed precision; the mmCIF's pdbx_number_measured_all / pdbx_redundancy / last shell row follow. A detector-limited lysozyme run (no auto cut) is byte-identical throughout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:18:10 +02:00
leonarski_fandClaude Opus 5	f9c5adbfb0	rugnux: the indexing-ambiguity warning is withdrawn only where the model settled it ... The INDEXING_AMBIGUITY warning was suppressed whenever --model was given on rotation data, decided before the model was read. A model that then decided nothing - not tested, rejected, or a probe winner that did not beat its own null - still silenced it, so WARNING_COUNT, PATHOLOGY_FLAGS and possibly VERDICT differed from a run without the model although the written reflections were identical. The warning is now issued exactly as without a model, and withdrawn (from the warnings and from the statistics text) after model validation only where the indexing probe decided the indexing: the model fits and the winner's R-free margin beats the random-placement null (ModelValidationResult::indexing_decided, set where the decision is taken). A reference MTZ, or the model reference on serial stills with -C and -S, suppresses it up front as before. Verified bare vs --model on three open-arm sets: merged MTZ data identical in all three; warnings identical where the model decided nothing (identity probe without null; no twin law); withdrawn where the probe decided (+33 sigma). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 11:18:00 +02:00
leonarski_fandClaude Fable 5.1	8a0da57941	rugnux: --report-resolution, a second statistics table at a reference range ... Comparing a run with another program's table has meant running rugnux AT that program's resolution range (--scaling-high-resolution), which is a different run: the range moves the cut, the space-group decision and everything after them, so the comparison buys itself a different answer. --report-resolution <dmin>[,<dmax>] instead leaves the run alone and adds a second table to section 3 of the report - the REFRES_* keys and a shell table - binned from the same merged reflections over the range given, with the completeness denominator enumerated over that range and the shells in equal steps of 1/d^2 so they read row for row against a CORRECT.LP at the same range. Report-only: the merged files and every decision are byte-identical with and without it. The table holds only what the run kept. Where the reference range is finer than the run's own limit, the shells past it are printed as not merged (with their possible count) rather than as zeros, REFRES_SHELLS_PAST_LIMIT counts them so a consumer can tell "not merged" from a measured zero, REFRES_ COMPLETENESS counts their reflections as missing, and the other overall numbers are over the shells the run reached; nothing is read from the observations the run judged to carry no signal. REFRES_ISA is the error model refitted on the reflections of the table alone, in XDS's convention (rotation only; the stills model is fitted over the whole range already). On the rotation path the statistics block of MergeAndStats becomes a lambda over a shell grid, called once for the run's own grid and once for the reference one; the reference call floors every observation-level count at the cut by group d, the rule the erase applied. The stills MergeStats takes a declared range, whose bounds are the grid's whether or not any reflection reaches them. Both --mode mx and --mode scale report it, the viewer's command line echoes it, and the docs describe the keys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-19 10:58:42 +02:00
leonarski_fandClaude Fable 5.1	26a80caac4	rugnux: overall SigAno and CCanom over the written range only ... The per-shell SigAno and CCanom of the rotation merge were accumulated on the report grid, but the overall pair of numbers - the SIGANO / CC_ANOM keys and the mmCIF's pdbx_absDiff_over_sigma_anomalous - were summed over every Bijvoet pair the fulls hold, including the ones past the automatic resolution cut that the written reflections do not contain. On a run the cut trims, the overall line therefore described more data than the shells above it add up to. Both overall numbers now count only the pairs that land in a shell, as the rest of the table does; a run with a manual limit, whose ingest already ends at the limit, is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-19 10:58:41 +02:00
leonarski_fandClaude Fable 5.1	8242773938	CUDA: allocation streams are handed back, not leaked; a handled pool failure leaves no error ... A long-running broker began cancelling every data collection with Device decoding failed (CUDA (GPU) error (out of memory)), falling back to host decompression CUDA (GPU) error (out of memory) while nvidia-smi showed the cards less than a fifth full. Two defects, both in CUDAMemHelpers.h and both from the pooled allocator (cudaMallocAsync) that came with rc.162. The leak. cuda_allocation_stream() kept one stream per (thread, device) in a thread_local map of raw cudaStream_t and never destroyed them. That was written against rugnux, where the worker threads live as long as the process. The broker starts fresh std::async threads for every data collection - 16 or 64 of them - so every collection left that many streams behind. Measured: 0.56 MB of device memory per leaked stream, linear to 4928 streams, never returned, with nothing on the host side growing. The RAII wrapper (CudaStream) was there but not used at this site, and using it as-is - a stream destroyed when its thread exits - would not have been safe: ShadowFinder builds its GPU accumulator on a throw-away std::async thread and frees it from another thread long after, and that free is ordered on the allocating thread's stream. So the streams are still never destroyed, but a thread now only borrows one: CudaStream objects live in a process-wide per-device idle list, a thread takes one on first use and hands it back when it exits. Their number is bounded by the threads that were ever alive at once instead of by the threads ever started. The list itself is deliberately leaked, so that nothing calls into CUDA during static destruction. Replaying the broker's pattern against the real header, 60 collections of 64 threads: before 3840 streams, 260 -> 2424 MB of device memory after 64 streams, 260 -> 358 MB The stale error. Every helper here throws a named message, yet the log carried the raw CUDA string, so the failure came through a cuda_err() and not from an allocation. CudaDevicePtr falls back to cudaMalloc when cudaMallocAsync fails, silently - but the failed call stays behind as the thread's last error, and the cudaGetLastError() that follows the next kernel launch reports it. The buffers were all allocated; the frame was lost anyway, once on the device-decode route (caught, hence the warning) and once more on the host fallback (fatal). The pooled attempt failing, and a stream that cannot be created, are both handled by falling back, so both now clear the error they leave. What finite resource the production cards ran out of at under 4 GB used was not established - no cap on the number of streams was found up to 4928 on the card this was measured on. The leak is the only thing on this path that grows with uptime. tests/CUDAMemHelpersTest.cpp: later threads end up on the same stream, concurrent threads on different ones, a buffer is freed cleanly after its allocating thread has exited (and another has borrowed its stream), and a pool that cannot serve a request leaves no error behind - the last by capping a memory pool at 4 MB so that the pooled attempt fails and the fallback succeeds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-19 10:39:39 +02:00
leonarski_fandClaude Opus 5	aa47b6bbbd	tools/battery: the sparse myoglobin set gets a real XDS reference ... The crystal does have a single P2 lattice, holding ~6% of each frame's spots on top of ice and protein powder; XDS indexes it with the ice rings excluded. The reference is now that P2 run to 2.0 A instead of a failed P1 run, and the set is renamed from aggregate to sparse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 10:02:08 +02:00
leonarski_fandClaude Opus 5	6e2e2fcaf9	tools/battery: a working XDS reference for the half-shadowed lysozyme set ... XDS was rerun with the pin shadow masked and the frames where the crystal leaves the beam excluded; the reference is now P43212 to 1.65 A instead of a failed P1 run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:47:36 +02:00
leonarski_fandClaude Opus 5	ee85ad1abd	tools/battery: run every variant of a set back to back ... The battery is bound by reading images from disk, so instead of one battery run per setting each set now runs in all its variants in a row while its images are still in the page cache: open arm bare, then model (rugnux --model <deposited coordinates>) XDS arms bare, then xds (XDS's resolution range forced, -A where XDS was anomalous) bare (plain rugnux <input>) runs first on every arm, so its time always carries the set's disk read whichever variants are selected; each row records first_read and the report's timing table says which variant's times are cold. --variants runs a subset; --unforced is gone (--variants bare). Rows, work dirs (work/<arm>/<set>/<variant>/), compare and the report are per variant; compare pairs (arm, set, variant) on the variants both runs have on an arm, and a schema-1 run is read as one variant. The model variant takes the coordinates and published R-free from model_check's RCSB cache (site key pdb_cache) and records rugnux's R-free/R-work and the ratio; the REFMAC check is now opt-in (--model-check) and its keys moved to refmac_*. results_schema 2. model_sweep.py is retired: the model variant replaces it (its --spot/--scaling-low-resolution 50 were rugnux's defaults, so the command is the same). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:42:27 +02:00
leonarski_fandClaude Opus 5	f6f955e19b	tools/battery: -A where XDS kept Friedel mates apart ... On the XDS-referenced arms rugnux merges Friedel mates separately whenever XDS ran with FRIEDEL'S_LAW=FALSE, in forced and unforced mode alike, so the merging statistics compare like with like. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:25:42 +02:00
leonarski_fandClaude Opus 5	17eec06023	tools/battery: the R-free ratio compares like with like ... Our merge and the depositor's structure factors are both scored on the depositor's free set by the same protocol, and the report plots that ratio. R-free on our own free set reads low, because those reflections were mostly work reflections in the deposited refinement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:22:20 +02:00
leonarski_fandClaude Opus 5	81bd42095b	tools/battery: model_check.py, R-factors of the deposited model against the merged data ... Run after rugnux has written its MTZ, so processing never sees the model. Fetches the entry (cached), matches the data to the model's setting by trying every Niggli-cell change of basis and keeping the one that best fits the model's Fcalc on a mid-resolution shell, re-expresses the data in the deposited space group and cell, and has REFMAC compute R-work/R-free of the unmodified model (rigid-body mode read at its first cycle: no dictionaries, no ligand stripped, no refinement). Baseline: the same protocol on the depositor's own structure factors and free set, plus rugnux's data on the depositor's flags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:20:51 +02:00
leonarski_fandClaude Opus 5	26e256513c	tools/battery: in-house sets named by protein, beamline and feature ... Directories follow <protein>_<beamline>_<feature> (e.g. lyso_x10sa_90deg_1, thau_bl1a_6p5keV, thau_micromax_pink); previous ids stay as aliases so older runs still pair up in compare. The ice-only control moved out of the in-house arm. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:18:19 +02:00
leonarski_fandClaude Opus 5	13d78e5bc3	CLAUDE.md: the battery is tools/battery, with its rules ... Replaces the mention of the retired rugnux_vs_xds.py with a short Battery section: the canonical commands, never a full run without the owner's go-ahead, judge against a persisted baseline, the private arm stays out of the repository. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:15:45 +02:00
leonarski_fandClaude Opus 5	2cfdc939a7	Retire the scattered battery scripts in favour of tools/battery ... - rugnux_vs_xds.py: deleted. The inhouse and private arms of tools/battery run the same comparison against CORRECT.LP, from a fixed manifest instead of a directory scan, with persisted runs and a standard report. - lyso_test/anomalous_scoreboard.sh: deleted. A one-dataset SHELXC/ANODE scoreboard with hard-coded personal paths, superseded by rugnux_anomalous.py. - rugnux_vs_pdb.py -> tools/battery/model_sweep.py. The rugnux --model sweep is not superseded, but it now takes its inputs from the open arm's manifest and the battery site config, so both tools process the same files. - docs/TESTS.md and rugnux_stills_ab.py point at the battery. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:15:45 +02:00
leonarski_fandClaude Opus 5	a7a3c2a00c	tools/battery: one canonical runner, derived XDS d_min, clean inhouse layout ... - Site config: site.example.json is committed; the local site.json (git-ignored) or $JFJOCH_BATTERY_SITE / --site names this machine's paths and the private manifest. Data roots are the arms' own directories, so copying them and editing the site config moves the battery. - Run states: manifest records the runner's pid; a run is complete, running, aborted (Ctrl-C/SIGTERM, or `abort RUN`) or unfinished. compare and report refuse a run that did not finish unless --allow-incomplete, and the report says so. - `list` shows runs and their state; `run` defaults its label to the rugnux version and its baseline to the site's persisted one; a baseline must be complete and of the same privacy as the run; a private report cannot be written into the repo. - XDS reference d_min: when the finest shell of XDS's table has no significant CC1/2, XDS merged past its own signal (no resolution cut, or a cut at the detector edge) and its limit is the detector's reach. The reference is then where XDS's CC1/2 falls through 0.30 (rugnux's own target), and that is also the range forced on rugnux. The manifest keeps XDS's limit (dmin_xds) and the rule (dmin_rule). Forcing the old 1.08 A on the weak insulin set turned rugnux's I23 into I222 (ISa 12.5); forced at the derived 1.81 A it is I23 again (ISa 18.0). Changed in the inhouse arm: insu_I_weak 1.08 -> 1.81, lyso_half_image 0.80 -> 2.60, cytc_eiger 2.04 -> 2.27, lyso_strong 1.18 -> 1.24; every other inhouse limit stands. - inhouse.json follows the directory rename (ids and inputs, data root /home/data/inhouse); `remap` applies such a rename and keeps the old ids as aliases, so runs from before it still compare set by set. - Optional open-arm model check (model_check.py, when present): R-free of the deposited model against each merge, recorded and plotted. - results.json has a fixed, documented row schema (results_schema 1); the HTML report gains inline-SVG plots (verdicts per arm, d_min and R-free ratios per set). - README.md: arms, data layout and provenance, prerequisites, running, run directory, results schema, report, comparing, protocol, adding datasets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 09:15:37 +02:00
leonarski_fandClaude Opus 5	8afab32d38	tools/battery: 6h2p runs as two sets, one per wavelength ... The crystal was measured at two wavelengths; both sweeps are kept. The native sweep carries the deposited resolution, the 1.89 A sweep is scored on symmetry and cell only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 08:51:07 +02:00
leonarski_fandClaude Opus 5	e24ee4d4b5	CLAUDE.md: the standard test proteins and the open battery arm may be named ... Extends the lysozyme carve-out to thaumatin, insulin, cytochrome C and myoglobin (plus the in-house no-crystal controls), and allows the public PDB depositions of the open battery arm. The private arm stays out of the repository entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 08:50:54 +02:00
leonarski_fandClaude Opus 5	e6baa4f153	tools/battery: committed, reproducible validation battery ... One runner for the open (vs PDB deposition) and in-house (vs XDS) arms, replacing the scattered scripts outside the repository. Each run writes a new, read-only directory named <date>_<rugnux git>_<label> holding a copy of the binary with its sha256 and build flags, manifest.json (inputs, references, options, host, GPU sharing) and results.json in one schema for all arms, plus per-set work directories and a Markdown and a self-contained HTML report (per-arm and per-population pass rates, resolution gain, ISa/R_meas distributions, timing, failures, delta vs a baseline run). - Dataset lists and references are data files: open.json (151 public sets, deposited group/cell/resolution, pinned inputs) and inhouse.json (standard test crystals and no-crystal controls, XDS references). The private arm's manifest stays outside the repository; a private run gets its own run directory and report. - Scoring is setting-invariant: lattice by Niggli-reduced primitive cell (beta vs 180-beta, C2 vs I2), symmetry via sgequiv on the open arm and by point group against XDS, then merge usability. Re-scoring the rc169 and rc170 battery outputs reproduces their verdicts. - Input discovery (`discover`) ranks sweeps by frame count, never by file size or sort order; a run never searches, it runs what the manifest names. - `compare A B` flags per-set changes beyond noise; `--rerun-changed` reruns them with A's saved binary to separate real changes from run-to-run noise. - `--tier smoke` selects a 22-set subset spanning the populations; a full-arm run takes a lock; `--gpulock` queues on the shared GPU and marks the timing as not a reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-19 08:13:46 +02:00
leonarski_fandClaude Opus 5	d6a2b9d19e	broker: error responses carry the content type the OpenAPI spec declares ... The 500 handlers built a correct error_message object but sent it through send_plain(), which labels the body text/plain. The spec declares 500 as application/json + error_message, and the generated clients key their deserialization off the content type: the Python client kept the body as a raw str, handed it to ErrorMessage.from_dict(), and pydantic raised - the exception swallowed by the finally: in response_deserialize, so the caller got a ServiceException with data=None and only the raw text in .body. send_plain becomes send_error, taking the content type alongside the code and the body; the two exception handlers now return it. 400 stays a plain-text exception string, exactly as the spec says; 500 is JSON. Two further 500s did not carry an error_message object at all and now do: the generic std::exception branch of handleParsingException, which returned a bare what(), and ProcessOutput's output-validation failure, which returned the validation dump. A catch-all set_exception_handler covers anything that escapes a route's own handler - httplib would otherwise answer with a bodyless 500, which fails to deserialize the same way. Verified against a running broker: POST /pedestal in the wrong state returns application/json and the generated Python client parses it into ErrorMessage(msg=..., reason='WrongDAQState'); a malformed body still returns 400 text/plain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 13:59:57 +02:00
leonarski_fandClaude Opus 5	51f51ec5bf	tests: the FFT indexers on a frame of pure noise ... The frame that works the indexer hardest is junk, not a crystal: a real lattice prunes the FFT shortlist to four or five distinct directions, while noise leaves dozens, which is what drives the length sort and the degenerate-plane fallback - where a production broker segfaulted twice. There was no test in that regime. Two cases, both over every FFT back-end the build has (GPU FFT under CUDA, CPU FFTW always), so a non-CUDA job covers them as well: - a cloud of 1500 reciprocal-space vectors of random direction and length, with no periodicity in it. Both back-ends return 70 candidate lattices from such a frame, which is the designed behaviour - Run() offers candidates and the caller scores them - so what the test pins is that none of them looks like a crystal: each takes about 1% of the cloud against the ~100% a real lattice takes, and the assertion allows 5%. - the same noise flattened onto a tilted plane. A coplanar shortlist cannot close a cell, so this is the branch that goes looking for the missing row in a 3 deg cap (SearchCap), verified with a temporary probe to enter it on both back-ends and not to enter it on the isotropic frame. The SAME indexer object then has to index a clean lattice correctly, which is the regression test for the cap search putting the direction grid back. The cloud is drawn from a fixed seed, and from the engine scaled by hand rather than through uniform_real_distribution, whose output is not specified to be the same in every standard library: a crash that needs one particular junk frame is no use as a regression test if the frame is redrawn on each machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 13:03:06 +02:00
leonarski_fandClaude Opus 5	4b5c6e234d	indexing: the cap search always puts the direction grid back, and every FFT upload is checked ... A production broker on rc.170 segfaulted again in the FilterFFTResults length sort, in a binary that does contain the rc.170 non-finite guard (verified from its disassembly: the two isfinite tests are compiled in and skip the insert). With magnitude and length guaranteed finite, the only remaining way that sort sees a NaN key is a NaN direction vector - and the one place that writes the direction grid is SearchCap. SearchCap swaps in a temporary cap grid but restored it around ExecuteFFT only. The fill can run out of memory, the upload can fail, and the .at() that reads the peak back throws on exactly the corrupted result row this search has to survive - any of those left the cap grid installed on an indexer the pool hands the next image. If the axis was ever not a direction, axis.Normalize() is NaN and the leaked grid is 16384 NaN directions, which is that crash on every image afterwards. The restore now covers the whole body, and an axis that is not finite and non-zero is refused instead of normalized into NaN. The peak is also tested against the minimum length before the .at() rather than after, so an unusable peak with a corrupted index no longer throws at all. DirectionsChanged uploaded the grid with three unchecked cudaMemcpy calls, and the spot upload in ExecuteFFT had three more: a silently failed upload leaves the device holding the previous grid, so the directions the host reads the results against are not the ones the kernel used. Checked like the rest of the file. Tests: [Indexing] (9 cases, 115 assertions) and RotationIndexer pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 12:25:09 +02:00
leonarski_fandClaude Opus 5	20977e89bc	viewer: draw a live grid scan as a grid, and the compact axis names upright ... Two fixes in the dataset-info panel. The live (HTTP) reader took the goniometer axis and the grid scan as alternatives, so a raster collected at a head position - both fields set, the axis with step 0 - lost its grid and was drawn as a line plot, with the Grid button disabled. The broker had the same bug and lost it in rc.162 (OpenAPIConvert.cpp); the HDF5 read-back path already read both. The reader now sets both, like the other two. Qt Charts always draws a vertical axis title rotated, which reads awkwardly beside the two numbers it names in a half-height plot and is what forced that plot's minimum height (Qt elided "Spots" to "Sp..." once the font outgrew it). The title is switched off and the view paints the name itself: upright, one letter under the next, in the series' colour, centred on the plot area, with the left margin reserving its width. It is re-measured on every rebuild, so a font change still fixes itself, and plotAreaChanged re-centres it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 11:34:57 +02:00
leonarski_fandClaude Opus 5	4bff5c2b01	v1.0.0-rc.171 ... VERSION bumped and update_version.sh run: the OpenAPI spec and the three clients it generates, the frontend and docs version strings, the Redoc page, and the FPGA and PCIe-driver version strings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 08:41:29 +02:00
leonarski_fandClaude Opus 5	29a90416b7	docs: condense the rc.171 changelog ... Three lines at the level the rest of the file keeps: what a user gains, not how it works. The detail lives in RUGNUX_FORMATS.md and in the commits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 08:38:42 +02:00