162 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	699bff1cb1	Merge branch 'sym-evidence' into rc175: POINTLESS/ZANUDA symmetry evidence for 18 open-arm sets, kept with the battery ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 16:02:51 +02:00
leonarski_fandClaude Opus 5.5	0bbe086b24	battery: symmetry evidence from POINTLESS and ZANUDA for 18 open-arm sets ... For every open-arm set where rugnux and the deposition differ in symmetry (sym_over, sym_screw, screw_undetermined, accepted alternatives) and the sets the twin review flagged, an independent assessment by established programs is now kept beside the manifest in tools/battery/symmetry/<id>/: a summary.md and the trimmed POINTLESS and ZANUDA logs. Each row names its directory in "symmetry_evidence". Nothing reads this for scoring, and no accepted or rejected answer changes. symmetry_evidence.py makes the evidence for one set. It re-runs the battery run's binary with the unmerged export (p.hkl must equal the run's) and runs POINTLESS on the observations. It merges them with AIMLESS in the deposited group and the model's setting, then runs ZANUDA with the deposited model on three free sets, each chosen per lattice-holohedry orbit. It runs POINTLESS and ZANUDA again on the depositor's structure factors, and tabulates the axial reflections of both data sets for screw questions. ZANUDA's preference can move with the free set (6toc), which is why there are three. What the programs found, reported neutrally: an exact pseudo-symmetry, a twin or another effect can produce any of these results, and the aim is only which answers the battery accepts. 6oww: ZANUDA prefers P 43 21 2 to P 21 on all four data sets, with lower R-free. 6toc and 8xtg: ZANUDA prefers the higher group, but R-free is mixed (6toc) or favours the deposited group (8xtg). 6pxb and 8xte: ZANUDA keeps the deposited P 32, and the deposited model in the higher group refines far worse. 3r6o, 4bwl, 2wnq, 8c3e: the deposited model gives the higher group no support. 5ebi: ZANUDA cannot reproduce the deposited fit. Screw sets: see each summary. Run on rc175 fcbe920 battery run 20261012-0139 with CCP4 9.0.014 (POINTLESS 1.14.2, AIMLESS 0.8.2, ZANUDA 1.097 / REFMAC 5.8.0431). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 16:02:01 +02:00
leonarski_fandClaude Opus 5.5	d172f22074	tools: remove jfjoch_mr_replay ... A developer tool, not built or installed upstream. It also failed to link in both portable builds (static cuFFT without the device-link step on Linux with CUDA, ws2_32 on Windows). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 12:59:43 +02:00
leonarski_fandClaude Opus 5.5	230c5e3917	battery: 2ygz leaves the open arm; 6oww accepts P 41 21 2 as an open question ... 2ygz is a withdrawn PDB entry of the same protein as 2wnq, so it scores nothing that 2wnq does not. 6oww's deposited 8-chain P 1 21 1 model obeys the 4-fold and 2-fold the higher group adds, and its |Fc|^2 correlate across the added 422 operators at 0.97-1.00, so P 41 21 2 is accepted beside the deposited group - marked as an open question, as no refinement has compared the two. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 09:47:39 +02:00
leonarski_fandClaude Opus 5.5	fcbe92042b	Merge branch 'af-mr3' into rc175: --af-model searches several copies per asymmetric unit (Matthews probability, sequential translation function, fixed-copy rigid body, tNCS) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 01:36:26 +02:00
leonarski_fandClaude Opus 5.5	63e0a85467	rugnux --af-model: several copies in the asymmetric unit, tNCS, a search-matched null, faster ... Copy number. The hard "Vm >= 4: MULTI_COPY, no decision" rule is gone. Copies are searched one after another up to the largest copy number with a Matthews probability of 1% (a Gumbel distribution with Kantardjieff & Rupp's 2003 mode and mean, no resolution dependence); COPIES_LIKELY reports the most probable. Sequential search. Copy n's translation function carries copies 1..n-1 as a known partial structure: the cross term 2 Re(conj(F_known) F(t)) goes on the same FFT grid (CPU FFTW and GPU cuFFT; KnownStructureFactors, AssemblyCorrelation). Its peaks, past those that clash, are ranked by the correlation of the whole assembly with E^2 (the TF itself, linear in |F|^2, rewards copying a placed copy onto an origin shift), the first that stands out (TF Z >= 8), packs and lowers R-free by 0.01 is kept; each copy is refined beside the others held fixed (RefineRigidBody's fixed copies), and the first copy again beside the second, the earlier ones once more at the end. Packing: ClashFraction counts C-alpha clashes with other copies and all their mates, binned (7mzt: 51 s of clash checks -> 1.9 s). Search quality: rotation-function peaks hill-climbed off the 6 deg sampling (6ze4: RF Z 10.4 -> 16.0, 3.3 deg off the deposited orientation -> 0.2); the first copy goes down the TF list until three placements that pack are refined, so a group is CLASH only where no offered orientation packs; groups are searched side by side, and every group the best one leads is searched again with the best one's copy orientations tried first, the better search kept. tNCS: the native Patterson of E^2 (5-10 A) off the lattice; placed copies' orientations join every further copy's search; and where the tNCS vector turns one group's screw axes into another's (TncsRelatesGroups), a group whose search stopped at fewer copies does not lose to the other - UNDECIDED. Measured on an in-house tNCS crystal (vector c/3 along the 3-fold): both hands fit with three copies to within 0.01 in R-free, and where one hand's search failed the other was taken as ENANTIOMORPH; now UNDECIDED. Null: the replicates are placed by the translation function in their own orientation, as the search placed the real one (a null at the real centroid reads high), and the gate is 4.51 = t(8) sqrt(1 + 1/9), 3 sd of a nine-draw null, the gate --model's MODEL_FIT is now read at. Speed: placements scored on the GPU structure-factor engines; further copies' trials score with the first copy's bulk solvent held, the assembly refitted at the end. Survey of 193 sets (jfjoch_mr_replay on stored merges, RTX 5080 shared with other jobs): MR median 1.15 s, max 24 s; demos 0.8-1.1 s. Report: COPIES and COPIES_LIKELY columns, MODEL_PRIOR_COPIES. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 00:27:17 +02:00
leonarski_fandClaude Opus 5.5	d906868bb2	battery: kept runs by machine role (keep --as workstation|server); the server run complete ... The kept-results directories are named by the machine's role, not its host name: tools/battery/results/workstation/<run>/ (1 socket, RTX 5080) and results/server/<run>/ (4 sockets, 4x T4). battery.py keep takes the role as --as; the kept files, untouched, still record the real host and hardware. The server's rc175 full battery now has all three files (open + inhouse, private false); checked for the private arm's set ids and input directories. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 22:33:35 +02:00
leonarski_fandClaude Opus 5.5	695bc907b4	battery: keep runs in the repository (battery.py keep, tools/battery/results/<host>/<run>/) ... Run directories are scratch, and the paper tooling needs the runs it draws from to outlive them. battery.py keep RUN copies a run's report.md, results.json and manifest.json as written into tools/battery/results/<host>/<run>/ (outside docs/, which Sphinx publishes), the host and run name read from the report. Open and in-house runs only: a run its manifest marks private, whose report is report_PRIVATE.md or says PRIVATE ARM, or with an arm or row outside those two, is refused. A run of which only report.md exists is kept with what there is. The CI job now keeps results.json and manifest.json beside report.md, in /data/battery/ci_reports/<run>/ and in the artifact, so a run on either host leaves what keep needs. paper_plots.py resolves a RUN by name among the kept runs after runs_root. Kept: the rc175 full battery on mpc2898 (20261011-1440_57fef1_rc175-57fef1c-full, all three files) and on mx-aare-test (20261011-1236_549939_1.0.0-rc.175, report.md only; the other two are to follow from that host). Both checked for the private arm's set ids and input directories. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 22:22:29 +02:00
leonarski_fandClaude Opus 5.5	eec8dcee9f	battery: the wrong-model null as a CI step of its own (battery.py nullmodel) ... The battery job ran the ci tier and the wrong-model null in one step with one exit status, so a red job did not say which of the two failed. battery.py nullmodel runs the null on its own, in a run directory whose report is just that section, and exits 1 when a pair failed; run --no-nullmodel keeps it out of a --tier ci run. The workflow runs `run $SUBSET --no-nullmodel`, then (for a ci-tier run) `nullmodel` as a separate step that runs whatever the first did, and keeps both reports. The ci tier's own rows are unchanged; a plain `run --tier ci` still includes the null as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 21:30:34 +02:00
leonarski_fandClaude Opus 5.5	f800669839	jfjoch_mr_replay: --af-model's molecular replacement re-run on a run's merged MTZ files ... A developer utility, not installed: the search, the decision and the hand check run on the answer's p.mtz (and the twin hypothesis' p_twin_<group>.mtz) that a rugnux run wrote, so a survey of the search over every dataset with a predicted model takes minutes instead of a reprocessing per set. The hand check of an ENANTIOMORPH decision moves from the rugnux driver into CheckEnantiomorphHand, which both call; unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 20:50:47 +02:00
leonarski_fandClaude Opus 5.5	4d17aae4e3	battery: wrong-model null takes about 6 minutes of rugnux (measured) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 14:36:12 +02:00
leonarski_fandClaude Opus 5.5	0a3fadb9ef	battery: wrong-model null step, run with every ci-tier run ... A ci-tier run (or any run given --nullmodel) now also runs five ci-tier sets of the open arm (8u0i, 6iu5, 9khr, 9b22, 6pxc) once without --model and once with each of five WRONG models, the deposited coordinates of other open-arm entries (already in the pdb_cache, so no new data): similar size, sequence identity under 25%, gesamt Q-score at most 0.20, one hard-ish model of the same secondary-structure class per set, one in the same space group or a similar cell, and for the two enantiomorphic sets one in the other enantiomorph. nullmodel.json records every model's size, identity, Q-score and reason. Each wrong-model run is scored against the no-model run (nullmodel.py): (a) p.hkl, the p.mtz reflections and p.cif identical, (b) no KEY= line outside MODEL VALIDATION moved, (c) MODEL_FIT not ACCEPTED, no decision taken, CC_MODEL_CONFIRMED_TO_D_MIN NONE and R_free >= 0.50 (Wilson's random-structure R is 0.586 acentric). The pairs go to nullmodel_results.json and a report section of their own; the sets' own rows are untouched. A failed pair fails the run, and the Gitea battery workflow prints the section into the job log. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 11:50:21 +02:00
leonarski_fandClaude Opus 5.5	467ac90226	Merge branch 'nocrystal' into rc175: the no-crystal test asks its own significance (1e-9); exact tail logged; nothing_1/2 join the ci tier ... Exact: p.hkl md5 and MTZ data identical on every real set (34 open/inhouse incl. the ci tier, all 27 private); both no-crystal controls now ~7 decades inside the bar instead of 0.3-0.9. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:03:35 +02:00
leonarski_fandClaude Opus 5.5	7d5551fd86	battery: the no-crystal controls join the ci tier ... nothing_1 and nothing_2 now pass the no-crystal test by about seven decades (tails 10^-2.06 and 10^-2.70 against 1e-9) instead of 0.3-0.9, so a change that moves one stray reflection no longer flips them, and a regression that lets a chance lattice through fails CI. The ci tier is 28 sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:02:58 +02:00
leonarski_fandClaude Opus 5.5	49384c4aa8	Merge branch 'mv-ncs' into rc175: models expanded by their strict NCS operators; the written cell follows the model's indexing ... Exact: p.hkl md5 and MTZ reflection data identical on 18 sets. 4nwv R-free 0.545 -> 0.297 and cell in the depositor's frame; written cells corrected on 9rcs, 9b22, 9qw8, myob_x06da_powder_1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 13:36:31 +02:00
leonarski_fandClaude Opus 5.5	1f27fba1f3	Model validation expands strict NCS; the written cell follows the model's indexing ... - ModelValidation reads the model and, where it carries non-given NCS operators (_struct_ncs_oper 'generate' / MTRIX without the given flag), expands them (gemmi expand_ncs, vendored assembly.cpp from GEMMI 0.7.5, MPL-2.0 as the rest of gemmi_gph). Same in the battery's model_check.py and depdata_check.py. An icosahedral-virus entry deposited as 3 of the 45 chains of its ASU was scored against a fifteenth of its atoms: R-free 0.545 -> 0.297. - The indexing the model chooses (AdoptModelFrame) now moves the cell with the indices (reseat_integrated in the main path; CellInBasis in the forecast, twin-file and stills/scale paths). A near-symmetry law (b within 2% of c, beta a degree off 90, a~b in P1) relabelled the reflections only, so the files were written with two axes swapped or beta flipped relative to the indices. Where the law moved the cell, the validation is remade on the moved cell. Measured on 18 open/in-house sets with models: p.hkl and MTZ reflection data bit-identical on all; cell and model R unchanged where the law is exact; the cell now matches the deposited one on the 6 pseudo-merohedral cases (orthorhombic a/c swap, three monoclinic beta flips, a P1 alpha/beta swap), R-free within +-0.001 there. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 13:35:49 +02:00
leonarski_fandClaude Opus 5.5	8c8fb24779	battery: a timeout or interrupt kills rugnux's whole process group ... subprocess.run's timeout killed only the direct child. Under --gpulock that is flock, under `stress --sanitizer` compute-sanitizer, and in both cases rugnux went on running on the card, orphaned, beside whatever took the GPU lock next - found when a cut-short memcheck stress run left its rugnux behind. Each run now gets a process group of its own, killed whole on a timeout, Ctrl-C or SIGTERM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:57:38 +02:00
leonarski_fandClaude Opus 5.5	7f4c9e2f8a	battery README: cut the sweep for compute-sanitizer stress runs; read the Xid after a device fault ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:46:06 +02:00
leonarski_fandClaude Opus 5.5	d495a01de4	battery: a handled GPU error is a crash only when it is a device fault ... A planned fallback that rugnux logs with a CUDA error text - the rigid body's zone over its engine's reserve, revalidated on the CPU, or an indexer worker out of device memory while other jobs held the card - would have turned sets into crashes in every run. A handled error now counts only when it names a device fault (unspecified launch failure, illegal memory access/address, misaligned address, launch timeout, device-side assert, "CUDA device unusable"), which takes the context down for every thread and is never a property of the data. "Processing failed" still counts whatever its cause. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:24:48 +02:00
leonarski_fandClaude Opus 5.5	1fc06f6e6c	battery: a crash is its own verdict and fails the run; stress subcommand ... A run that was killed by a signal, or whose log has "Processing failed", a GPU error (also one rugnux handled and went on from), "terminate called" or a non-zero compute-sanitizer ERROR SUMMARY is now verdict `crash` (score.crash_note), decided before anything else, so it fails even a no-crystal control whose normal refusal exits 1. A crash counts as a failure in every pass rate, gets a CRASHED section at the top of the report and a CRASHED line under the console table, and makes `run` exit 1 whatever tier the set is in. Reports and compares re-read crashes from the log, so an older run's crash shows as one: the GPU crash of 9ig7 in the last full run showed only as "fail, exit 1: Step 8 FAILED" before - failure_note now prefers the [error] line it follows. `battery.py stress` runs named sets --repeat times, --concurrent at once, optionally under compute-sanitizer (memcheck/racecheck/synccheck/initcheck) or with CUDA_LAUNCH_BLOCKING=1, and exits 1 on any crash; it reports distinct p.hkl per set and keeps out of the run list. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:00:50 +02:00
leonarski_fandClaude Opus 5.5	d8a3c6ea15	battery: add five large-cell sets to the open arm ... 4MTK (P61, c 653 A, PILATUS 6M miniCBF, SBGrid 235), 4NWV (I222 virus-like particle, 370-411 A axes, Rayonix MX-300 marCCD, SBGrid 9), 7D98 (P6122, c 602 A, EIGER 4M miniCBF at 1.9 A, XRDa 13), 9V16 (P41212, c 451 A, EIGER1 16M HDF5, XRDa 338) and 4P8R (P6522, c 701 A, Rayonix MX-225 marCCD, IRRMC), tagged large-cell (plus long-axis / virus). References are the deposited space group, cell and resolution. 7D98 is pinned to its 250-frame miniCBF sweep: the HDF5 masters beside it repeat frames 61-260 in 60-frame pieces. 4P8R's archive ships per-frame .bz2 files, decompressed after extraction. EXTERNAL_TEST_DATA.md gains the five rows (DOIs resolved), the archive notes and the updated collection counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 20:31:28 +02:00
leonarski_fandClaude Opus 5.5	7624bbe12e	battery: paper_plots.py draws the other figures when no run has REFMAC R-free ... The REFMAC check is opt-in, so a battery run without --model-check has no R-free pairs, and the axis range min()/max() over them crashed the script before any figure was drawn. The R-free figure is now its own gnuplot block, left out (with a line in summary.txt saying why) when there is nothing to draw; d_min and the time figures are drawn as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	d28c8b8bb3	battery: same-absence groups of one Laue class pass as indistinguishable_alternative ... sgequiv keyed space groups on their proper+improper rotation matrices, so groups with identical merged intensities but different point groups (I41md vs I-42d, Pc vs P2/c, Pna21 vs Pnma, P2 vs P2/m) were called a "REAL difference", contradicting its own docstring, and score.py compared point groups literally on the XDS arms. A run that wrote one member of such a pair and listed the other in SPACE_GROUP_ALTERNATIVES failed as sym_other / sym_over. sgequiv now keys each group, in every setting gemmi tabulates, on its Laue rotations (operations plus their negatives), centring and absences; describe() separates the new different-point-group pairs from the old same-point-group ones, and enantiomorph handling is unchanged. score.py passes a row, on both arms, when some setting of the reference has the same signature as our group in our own setting: verdict pass, cause indistinguishable_alternative, the reason saying whether rugnux listed the reference. Pass, not unscored: unlike an unmeasured screw axis, everything was measured and the answer is as right as the data allow; only a model or a strong anomalous signal decides. The separate cause keeps such rows countable. Being listed in SPACE_GROUP_ALTERNATIVES alone is deliberately not enough: that list also carries the screw alternatives of an undetermined axis, whose absences differ, and those stay under the screw rule. The open arm's same-point-group UNDECIDABLE pass now also requires the setting check. results.json gains sg_alternatives (rugnux's SPACE_GROUP_ALTERNATIVES). KDP joins the ci tier. Re-scoring the open and in-house arms of the two latest full runs changes one verdict in each: kdp_x10sa_20keV fail/sym_other -> pass/indistinguishable_alternative. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 16:27:23 +02:00
leonarski_fandClaude Opus 5.5	366935fe26	battery: right-align the reference values of the run table too, so decimal points line up ... Each numeric pair is now rugnux and reference both right-aligned to a fixed width around the "/", with fixed decimals per column (d_min 2, ISa / CC1/2 / R_meas / R-free 1, time 0), so the slashes and the decimal points of a column sit on one line; "-" takes the same width. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:39:20 +02:00
leonarski_fandClaude Opus 5.5	c5970a76f4	ci: the battery workflow keeps the battery machine's own paths and mounts ... The workflow runs on the battery machine, not on the development workstation, so the data roots (/data/open, /data/inhouse), the whole /data/battery mount, the scratch directory (/data/battery/scratch/<run id>), the kept reports (/data/battery/ci_reports) and the runner's valid_volumes are back as that machine has them; the remark tying the image to one host's driver is gone. The private arm stays out because the site config names only open and inhouse. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:38:43 +02:00
leonarski_fandClaude Opus 5.5	9beb8a7813	battery: align the run table's pairs on their slashes, the ticks in a column of their own ... Each rugnux / reference pair is now rugnux right-aligned to a fixed width, "/", the reference left-aligned to the same width ("-" for a missing value), so every slash of a column sits on one line; the tick or cross follows the space-group pair in its own column. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:37:50 +02:00
leonarski_fandClaude Opus 5.5	8c6670a3f4	battery: a ci tier, a closing table per set and a failing exit on it ... `battery.py run` now ends with one line per set, each column rugnux / reference: the space group with a tick or cross by the scorer's own verdict, d_min, ISa, CC1/2 overall and in the highest shell, R_meas of the lowest shell, the model's R-free and rugnux's wall time. `battery.py table RUN` prints it for any finished run (rows are rescored, so older runs get the new columns). The highest-shell CC1/2 is new in results.json (cc_half_high, read off rugnux's own shell table) with XDS's beside it (cc_half_high_ref, from CORRECT.LP's finest shell; the in-house manifest's refs regenerated with `refs --write`, which added only that key). A new "ci" tier: 25 sets that passed with the same space group and steady d_min / ISa in all four rc174/rc175 full runs (and in the rc175-next verdict runs where they ran), with no warning that questions the lattice or the group. About 10 minutes of rugnux (616 s measured) on the RTX 5080. A run that includes ci-tier sets exits 1 when one of them fails (lattice, symmetry, crash, timeout, missing input); nothing else fails a run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:31:32 +02:00
leonarski_fandClaude Opus 5.5	8e134c405d	tools: run the broker on a stored dataset, without a detector ... For demonstrating and testing a collection followed while it is written: - jfjoch_stream2_replay replays a stored dataset as a DECTRIS detector's stream2 (ZeroMQ PUSH on the stream2 port): start message, every image at a given rate, end message. - fake_simplon.py stands in for the detector's SIMPLON control interface: answers the configuration and status requests the broker makes, accepts every command, reads back idle. It needs port 80. Not installed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 15:52:22 +02:00
leonarski_fandClaude Opus 5.5	f97700e084	battery: read 8owm and 8rud as the deposited .cbf.gz; 6ZR0's zip is intact ... The MXRDR zips for both sets hold gzip-compressed frames. Our staging had expanded them to .cbf. rugnux reads .cbf.gz natively, so the manifest now points at the files exactly as deposited. 6ZR0's zip, as the Keele repository serves it, has a readable central directory with all 1060 members. Our earlier download was short, which lost the last frame. Remove the statement that the repository's zip is damaged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 19:59:40 +02:00
leonarski_fandClaude Opus 5.5	f3c3ea5d3c	battery: add 2ygz to the open arm ... A P21 crystal in the same form as 2xfw, but pseudo-merohedrally twinned with a twin fraction of 0.33 at an obliquity of 0.29 deg. That is the largest twin fraction at the highest obliquity in the arm. The PDB entry was withdrawn, so the reference cell and resolution come from Table 1 of Campeotto et al. (2018) Sci. Rep. 8, 14876, and the set runs without a model. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 17:09:15 +02:00
leonarski_fandClaude Opus 5.5	5421c569ed	rugnux: log the time of each section of the run's tail at debug level ... A TimingMark (common/Logger.h) logs "TIMING <section> <seconds>" at debug level when a section ends, so the lines go to <prefix>_log.txt and stay off the terminal. Marked: each pass of RunPipeline (pre-pass, probe, output) and its first-pass indexing, image pass, scale/merge/symmetry phase, output files and census wait; inside the scale/merge phase every merge (labelled by its role: P1, P1 space-group search, P1 all observations, the adopted group, P1 cross-check, quality guard), the post-refinement, the space-group search and its two SearchSpaceGroup arms, adoption, analyses, the P1 cross-check wait, model validation and writing; inside RotationScaleMerge the ingest and, per arm, partial scaling, combine and fulls scaling, correction surfaces and the merge; the console's steps; and the results report. tools/rugnux_timing.py tabulates the tail of a log. Only the log changes: the outputs are byte-identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 16:13:17 +02:00
leonarski_fandClaude Opus 5.5	37b7fdd38d	rugnux: the terminal shows numbered steps with timing; the full log goes to <prefix>_log.txt ... Standard mode prints only steps, one-line results with the time since the line above, a rule closing each step with its total, warnings (!!) and errors (** ERROR:), and a summary at the end. Every log line of every module goes to <prefix>_log.txt instead, at debug level and timestamped, opened with version, build, GPU, threads, command line and the header geometry. -v puts the whole log back on the terminal. - RugnuxObserver gains OnStep / OnNote. RunPipeline posts them only on a pass that makes the result (the observer is held in steps_, nullptr on probe passes and on copies run beside the main pass); RunAllPasses opens steps for the checks between passes (rotation scale, metric symmetry, geometry walk) and notes its re-runs. OnPhase is written to the log as a marker. - RugnuxConsole (CLI only): on a terminal a progress bar and a spinner status line drawn in place with \r, durations only; not a terminal, plain ASCII with the time of day on every line and quarter milestones for long passes. isatty/_isatty, no environment variables. A warning repeated by later merges is shown once more on one line and then counted per step. - The CLI's former stdout blocks (space-group search, merging statistics, per-image cost, processing time, tNCS/twinning/anisotropy in --mode scale) go to the log; the calibration result block stays on the terminal as that mode's summary. The MX summary takes its rows from the report's own SUMMARY (ReportDocument::summary_rows); WriteResultReport returns the document it rendered. HDF5's automatic error-stack printing is switched off. - The banner keeps the "Version X (git abcdef" form battery.py reads off a bare invocation; battery.py's failure note now reads p_log.txt, whose lines are not wrapped. Output only: p.mtz is bit-identical on the three in-house reference sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:11:21 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62	2026-06-17 20:36:24 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	caef26873e	v1.0.0-rc.145 (#55) ... This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144. * **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future. * XDS plugin: Improve performance of VDS reading. * jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD. * jfjoch_writer: Clean-up of HDF5 routines to better handle issues. Reviewed-on: #55	2026-05-06 21:50:02 +02:00
leonarski_f	87fde1b32e	v1.0.0-rc.142 (#52) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8 * Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent Reviewed-on: #52 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 16:47:53 +02:00