17 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	23812b902c	Rigid body: no JFJOCH_RIGID_BODY_CPU; GPU/CPU parity tests tightened ... The GPU rigid body is chosen by whether a card is there, nothing else. A run that must stay on the CPU says so at the process level (CUDA_VISIBLE_DEVICES=), which RigidBodyGPUPool::Create already honours through get_gpu_count(). Parity tests ([ModelValidation][gpu], compiled into jfjoch_test on a CUDA build): - every GPU case now SKIPs without a card, instead of passing silently; - ModelMaskGPU_MatchesGemmi also checks that the CPU path's PutMaskOnGrid is gemmi's mask bit for bit on the same grid, so the GPU mask is held against the CPU path itself; - RigidBodyGPU_MatchesCPU: Jacobian columns 2e-3 -> 5e-4 relative (measured up to 5e-5; the floor is the 1e-4 the scale's near-tie can move every column by). Residuals stay at 2e-4 of <Fobs> (measured up to 3.9e-5); - RigidBodyGPU_FitAgreesWithCPU: fit endpoint 2e-3 -> 1e-5 A rmsd (measured up to 5e-7 A). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 22:46:00 +02:00
leonarski_fandClaude Opus 5.5	15843eb49d	Rigid body: backend seam, GPU engine for density/composition/Jacobian, engine pool ... RigidBodyTarget becomes the CPU backend behind RigidBodyTargetBase, numerics unchanged. RigidBodyTargetGPU evaluates the same target on a GPU engine: atoms placed on the device, density by a deterministic brick gather (atoms in model order per point), cuFFT R2C, the SymmetryComposition gather for Fcalc and dF/dt in double, the numeric rotation columns as a batch-3 transform, the Jacobian rows and the Kaufman sums by fixed-order reductions. The mask and the scale fit are still computed on the host (stubs until ModelMaskGPU and ModelScaleGPU land). A pool of 1-4 engines is reserved once per validation, capped at min(25% of the card, free - 1 GB); GPU or CPU is decided once for the whole validation, and a CUDA failure restarts it on the CPU. JFJOCH_RIGID_BODY_CPU forces the CPU. PutMaskOnGrid skips gemmi's shrink where its stencil is empty (every rigid-body zone): exact, it cannot change a point. GPU vs CPU on the five test groups: residuals within 7e-6 of <F>, Jacobian columns within 4e-5 relative, fits 4e-7 A apart. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 16:51:16 +02:00
leonarski_fandClaude Opus 5.5	4ce36935b9	rugnux --model null: redraw a replicate the rigid body carried onto the model's own orientation ... The pre-draw guard keeps a null replicate from STARTING within RigidBodyReachDeg of an orientation equivalent to the model's (space-group rotations x {identity + twin laws}), but the analytic-Jacobian rigid body walks further than that reach: on 9yzk one replicate drawn outside the 10.6 deg reach was walked 23.9 deg onto an equivalent orientation (R-free 0.5578 -> 0.3917), the null SD went 0.016 -> 0.066 and MODEL_FIT flipped ACCEPTED -> REJECTED. Now where the replicate ENDS is checked too: RefineRigidBody reports the rotation it applied, the final orientation is that times the drawn one, and a replicate that ended within reach is drawn again (through the same pre-draw filter) and placed again, capped at NULL_MAX_REDRAWS. The redraws come from a generator per replicate (NULL_SEED + 1 + i), so they do not depend on the order the concurrent replicates finish in; the count is logged once per run. The real fit's procedure is unchanged. Validation, 12 open-arm sets against 20260928-0459_24ae26_r6-pooled: post-placement redraws on 9yzk (1) and 5epe (1), none elsewhere. 9yzk back to ACCEPTED, +14.15 sigma, null 0.5880 +- 0.0138 (base +12.21, 0.5864 +- 0.0158), p.mtz byte-identical to base. 5epe: MODEL_FIT +41.62 -> +46.39, indexing margin +40.94 -> +31.97 sigma, same decision. Every other set identical to the previous rigid-bc commit. -N 1 and the default thread count give identical outputs and logs on 9yzk and 6toc. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 12:45:38 +02:00
leonarski_fandClaude Opus 5.5	a594758238	Merge branch 'model-null-cheaper' into rc173 (model validation null scored to 3.5 A, at least 1000 free reflections) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 04:29:36 +02:00
leonarski_fandClaude Opus 5.5	1fd28672bb	ModelValidation: draw the null's placements away from every orientation equivalent to the model's ... The null is nine fixed-seed random orientations of the model, each placed by the same rigid body as the real fit and fitted to the data AS MERGED. Where the indexing probe reindexed the data to the model's, the model's solution against the merged data is its twin-related orientation - and more generally the crystal looks the same from every orientation the lattice's rotations (the space group's, composed with the twin laws) take the model to. A draw within the rigid body's reach of one of those is walked onto it and is not a sample of the null. On 5epe (F 2 3, twin law y,x,-z, 24 equivalent orientations) replicate 8 of 9 was drawn 7.9 deg from one of them; the rigid body moved it 7.9 deg, held-out R-free 0.53 -> 0.20, R-work 0.197 against 0.56-0.58 for the other eight. The null's sd became 0.125 and the crystal's own model read +0.01 sigma, DOES NOT FIT, so the indexing it had decided at +25.8 sigma was refused and the files kept the merged indexing (R-free 0.52). This happens on every binary whose indexer picks the other hand. The fix is at the draw, not the score: a draw whose angle to the nearest equivalent orientation is under the rigid body's reach is drawn again from the same generator, so everywhere no draw falls there the nine rotations - and every output - are exactly the ones before. The reach is the rotation about the centroid that moves the model's rms-radius atom by the ladder's first zone (6 A, or d_min where coarser): 18.7 deg on 5epe, whose other draws sat at 21.7 deg and beyond and did not converge. Twin laws come from the lattice (ReindexAmbiguityOperators on the data cell and group) regardless of whether the probe ran. Redraws are capped at 1000 so that a model small enough for the reach to cover the whole of SO(3) takes its draws as they come instead of looping. Scoring the real model against the best of both hands instead was not taken: the real fit is already scored in the hand that wins, and the contamination is in the null, not in the real fit. Measured, battery commands with --model, rc173 27e154e5d vs this: 5epe 1 redraw; DOES NOT FIT +0.01 -> FITS +55.86 sigma (null 0.5730 +- 0.0072); indexing decided at +40.4 sigma and reindexed y,x,-z; R-free 0.5246 -> 0.1749 (deposited 0.1671) 9fhc 1 redraw (reach 8.8 deg; the replaced draw had not converged); files byte-identical, MODEL_FIT_SIGMA +201.27 -> +199.39, verdict unchanged 9hnc, 8sa8, 8xtg, 9bn8: no redraw; p.hkl, p.mtz, p.cif (bar the audit date), p_maps.mtz, p_model.cif and the model-validation report byte-identical [ModelValidation] tests pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-28 02:23:04 +02:00
leonarski_fandClaude Opus 5.5	3783d388ed	ModelValidation: score the null, and the real fit against it, to 3.5 A [not exact - changes sigmas] ... QUALITY-AFFECTING, separate from the exact commits before it. The nine null replicates were fitted at the data's full resolution: a full-resolution Fcalc and mask, 1 + n(indexing candidates) scale fits over every reflection, the rigid body (which already stops at 3.5 A), then another Fcalc and fit. Those full-resolution scale fits were most of the null's CPU. Now the replicates, the real model's R-work and the real indexing margin they are compared with are all computed to max(d_min, 3.5 A) - the same fits on the same reflections on both sides - but never on fewer than 1000 free reflections, since the margin is read in R-free: on a small cell the limit moves out to where 1000 of them are, or to the data's own. The reported R-factors, maps, placement and files are still made at full resolution; only the null test moves. Why the free-reflection floor: without it the [ModelValidation] reindexing test (a 34x34x38 A cell, 1553 reflections) lost its decision - at 3.5 A its margin was read on ~30 free reflections, the null margin sd went to 0.053 and the real margin to +2.6 sigma, under the gate of 3. Measured against the floor-fixed build before this commit (battery commands, --model; CPU seconds in the null - wall clock on this shared box moved with other jobs, 2-4x less where it was quiet): set CPU s R-work sigma indexing margin sigma verdict / decision / files 9hnc 2307 -> 583 +10.4 -> +15.3 +106 -> +21 (reindexed) same / same / identical 8sa8 1382 -> 313 +328 -> +203 - same / same / identical 8xtg 968 -> 496 +215 -> +172 +77 -> +27 (kept) same / same / identical 9fhc 1042 -> 737 +201 -> +139 +76 -> +41 (reindexed) same / same / identical 9bn8 393 -> 153 +378 -> +235 +137 -> +56 (reindexed) same / same / identical 5epe 631 -> 600 +0.01 -> -0.03 +26 -> +25 (not decided) same / same / identical Sets where the null does not run (6oel, 5reo) are untouched by construction. The sigmas drop because the null's sd at 3.5 A is about twice as wide; the smallest margin here is still 8x the gate, but a borderline indexing decision would be decided less often, not more. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-27 23:12:27 +02:00
leonarski_fandClaude Opus 5.5	7eaf0f21b0	ModelValidation: floor the null's spread instead of zeroing the sigma below it [changes verdicts] ... record_fit set r_work_sigma to 0 whenever the null's sd fell under NULL_SD_FLOOR = 1e-3, and the indexing margin had the same guard at 1e-4. The floor was there so that sd == 0 and sd == 1e-7 do not land on opposite verdicts, but a zero sigma is itself a verdict: DOES NOT FIT. A large, well-measured crystal has a null that narrow for real - the replicates of a random placement agree to 0.0006-0.0010 in R-work over hundreds of thousands of reflections - so the crystal's own model, at R-work 0.17-0.29 against a null of 0.50-0.55, read "+0.00 sigma, the model DOES NOT FIT", and the decisions it gates (setting, indexing, enantiomorph label) were refused. The spread is now floored in the denominator, (mean - R) / max(sd, floor): continuous at sd -> 0, and a null with no spread because the model is too small or symmetric for a rotation to matter still reads near zero, since the real fit is then no better than its own random placements. Changed, measured on the battery commands with --model (every set in the last battery whose sigma was 0.00 for this reason; all three were REJECTED): 8sa8 +0.00 -> +328 sigma, FITS; the model's setting is adopted, R-free 0.2273 -> 0.1747 (deposited 0.1525) 9bn8 +0.00 -> +378 sigma, FITS; reindexed to the model's indexing (+137 sigma), R-free 0.5462 -> 0.1721 (deposited 0.1575) 8xtg +0.00 -> +215 sigma, FITS; the indexing it gates was already the model's ("kept"), so the reflection files are byte-identical and only the verdict changes Unchanged (files byte-identical, sigmas identical): 9hnc, 9fhc, 5epe; no margin null in the battery was under its own 1e-4 floor. [ModelValidation] tests pass. 5epe stays DOES NOT FIT for a different reason, not touched here: in F23 one of the nine random placements lies within the rigid body's reach of the twin-related orientation, walks there (held-out R-free 0.51 -> 0.18), and the null's sd becomes 0.13. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-27 23:12:25 +02:00
leonarski_fandClaude Opus 5.5	ce87b98280	ModelValidation: run the null's replicates on threads of their own, so their inner passes go parallel ... The nine null replicates ran under ParallelFor, i.e. on pool workers, and a parallel pass reached from a pool worker runs inline. So the scale fit (FitModelScale's grid) and the rigid-body Jacobian of every replicate but the one on the calling thread ran serially, and the null lasted as long as its slowest serial replicate. Sampled from /proc every 0.5 s through the null of a 1.63 A P2 set (36793 atoms, 4 indexing candidates): CPU use sat at exactly 8.0 cores of 32 for most of its 220 s, falling to 1-4 at the tail; on every set measured the null averaged 6-8 busy cores. The replicates now run on std::async threads outside the pool, so their passes queue on the pool as the real fit's do. Exact: every pass splits its work the same way wherever it runs (ParallelChunks is split on the thread count, not on the caller), so p.hkl, p.mtz, p.cif, p_maps.mtz, p_model.cif are byte-identical and the model-validation section of p_report.txt unchanged on 9hnc, 8sa8, 8xtg, 9fhc, 5epe and 6oel (battery commands, --model). (6oel's report also differs in SUPERCELL_DOUBLED_CELL, 106.695/119.998 vs the equivalent 73.305/60.002 - that line flips between runs of the same code already, and 6oel never reaches the null.) Null wall clock / mean busy cores (shared 16-core/32-thread box with other jobs, so indicative): 9hnc 220 s / 7.8 -> 152 s / 15.9 8sa8 167 s / 8.2 -> 60 s / 23.7 8xtg 88 s / 7.3 -> 46 s / 20.3 9fhc 77 s / 6.7 -> 57 s / 17.7 5epe 45 s / 6.1 -> 36 s / 17.0 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-27 21:11:14 +02:00
leonarski_fandClaude Opus 5.5	02bc0b7e0e	Rugnux: test angle promotions of the metric symmetry; score model frames at the data's best indexing ... The lattice walk admits an angle up to 3 deg from 90 and the constrained refinement then holds it there. The existing two-arm test (promoted vs demoted class, judged on the held-out residual) only covered length equalities (a = b). It now also covers monoclinic/orthorhombic classes: pass 1 reads how far the free refinement leaves a held angle from 90 (AngleEqualityDeparture_deg), turns it into pixels at the far corner, and where that exceeds the integration disc runs the demoted arm on the indexer's free (triclinic) cell. A primitive monoclinic crystal with beta 1.4 deg from 90 was indexed and integrated as primitive orthorhombic on 60/60 validation frames, so the per-frame guard never fired; demoted, the held-out residual halves (1.54e-6 -> 8.16e-7), ISa 5.5 -> 6.1, d_min 3.70 -> 3.37 A, CC1/2 at 3.9 A 0.54 -> 0.76. Unchanged on five controls (three oP/oI, two near-90 monoclinic). Model validation scored alternative cell frames against the data in the merged indexing, before the indexing-ambiguity probe. On a pseudo-orthorhombic P2 crystal whose data need h,-k,-l to match the model, every frame read random (R 0.77-0.84) and noise picked an axis swap into P1121; each frame is now scored at its best over the data's alternative indexings. CC_model 0.23 -> 0.45, R-free 0.66 -> 0.44 on that set. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-26 13:43:58 +02:00
leonarski_fandClaude Opus 5.5	19f399a973	Rugnux: write the standard setting, or the user's; match a reference MTZ on its own axes ... The files were written on the axes the space-group search named, which for P2221/P21212 puts the unique axis wherever the a<b<c indexing put it (a 52.51 87.87 137.72 crystal came out P 2 21 21 where XDS writes 87.87 137.72 52.51 P 21 21 2), and a reference MTZ was matched in the data's frame: on permuted axes its free-R flags landed on unrelated reflections while the log reported a high matched count. - New CrystalSetting (scale_merge): changes of basis between settings of one lattice (cell, group, index operator, basis matrix), the {-1,0,1} det +1 candidates (CellMappingOperators, moved from ModelValidation), MetricViolation (moved from Rugnux), ChooseOutputSetting and SeatGroupByAbsences. - Output setting: after every decision the merge, the integrated reflections (unmerged MTZ), the P1 cross-check, the lattice and the _process.h5 reindex matrix are relabelled into the ITA standard setting - or, in priority order, a reference MTZ's, a fitting model's, the -C axis order, a non-standard -S symbol's. Free-R flags are drawn again on the written axes. Reported as SETTING_OPERATOR / SETTING_SOURCE. - Reference MTZ: the group is kept in its setting; after the merge every cell mapping onto the reference cell (times the twin laws) is scored by the reference CC, the best is re-seated and re-merged, and the free flags are inherited only where CC >= 0.5 over >= 50% of the reference range (REFERENCE_MISMATCH otherwise; --mode scale gates the same way). REFERENCE_OPERATOR / _CC / _MATCHED_FRACTION / _FREE_FLAGS_INHERITED. - -S: a fixed group is put on the axes its absences name before merging (SeatGroupByAbsences), fixing -S 18 on a cell whose pure axis is not c. - --model: a model in another setting is now a claim the null tests; where it fits, the data are written in its setting and the validation is remade on those axes (KeepModelVerdict carries the decisions over). Tests: [setting] (synthetic #18/#17/I222/C222/c-unique P21/C2 beta/I2->C2/P1, -C and -S order, absence seating, permuted reference with flags). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C	2026-09-25 20:16:42 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00