174 Commits

Include renames

Author	SHA1	Message	Date
Filip LeonarskiandClaude Fable 5.1	82e5de1261	macOS: sign jfjoch_viewer and rugnux with a Developer ID and notarize them ... JFJOCH_MACOS_SIGNING_IDENTITY (the certificate's SHA-1) makes the install step sign the shipped program with hardened runtime and secure timestamp: macdeployqt's -sign-for-notarization for the viewer bundle, a codesign INSTALL(CODE) for rugnux. tools/notarize_macos.sh then submits the cpack artefact to Apple, waits for the verdict and staples the ticket to the .dmg; the rugnux binary inside the .tgz is submitted as a zip, as a bare executable cannot carry a ticket. Both macOS CI jobs resolve the identity from the runner's keychain, check for the hardened runtime and notarize on every push, so a rejection shows up with the change that caused it. The identity is passed as the hash, not the name: Qt's generated deploy script writes DEPLOY_TOOL_OPTIONS unquoted, so a name with spaces splits into several macdeployqt arguments. The notarytool profile has to live in the login keychain file (store-credentials --keychain): the default data-protection keychain becomes unreadable once the screen locks, which an unattended runner's does. Verified end to end on the runner: both artefacts accepted by Apple, the stapled image and a quarantined copy of the app and of rugnux pass Gatekeeper as Notarized Developer ID, signing and the credential lookup work from a launchd-started process with the console locked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-14 15:01:30 +02:00
jungfrauandClaude Opus 5.5	8430d69731	battery: 4nwv joins the ci tier (29 sets) ... The virus capsid (I222, ~400 A cell, marCCD) passed with the deposited group in every rc175 full run: 71 s on the workstation, 159 s and again in 20261014-0141 on the server. It adds a second large set to the regression check, about a minute of rugnux on the RTX 5080. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 03:44:57 -04:00
jungfrauandClaude Opus 5.5	e7d5964e75	battery: server run of the rc175 full battery at 6e9b891eb ... Open 196/203 (97%; the 20261011 server run had 195/205), inhouse 47/47, no crash. Kept as results/server/20261014-0141_6e9b89_1.0.0-rc.175 (report.md, results.json, manifest.json as written); no private arm on this site. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 03:44:41 -04:00
leonarski_fandClaude Opus 5.5	ea597507df	battery: workstation reference is the rc175 full battery at a45135926 ... Replaces the kept f853596 run with 20261013-1944_a45135_rc175-a451359-full (open + inhouse; the private arm is never kept). Against f853596: no verdict changes beyond 5ky6 fail -> unscored (screw undeterminable), sum 4145 -> 4088 s; the other moves are the last bits of device code without FMA contraction (8xtf ISa 7.7 -> 7.9) and anomalous sigma. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-13 21:17:39 +02:00
leonarski_fandClaude Opus 5.5	62e64235df	CI: the validation battery is a job of Build Packages, after the unit tests ... The unit-test job had joined the battery workflow's concurrency group to keep the two off the GPU at the same time. A group keeps one pending run or job and cancels it for a newer one, cancel-in-progress or not, so the battery waiting for its runner was cancelled by the same push's unit-test job, on every push since 140739a69. The battery is now a job in build_and_test.yml that needs the unit tests and runs whatever their outcome, so within one run the two take turns; no concurrency group, so a merge to main always gets both, and a manual release (create_release) runs neither. A manual subset or the full battery is a Run workflow with battery_subset (empty = full). Two pushes close together can still overlap on the GPU; a runner with capacity 1 for the GPU jobs is the later fix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-13 16:54:44 +02:00
leonarski_fandClaude Opus 5.5	e80735a261	Merge branch 'review-fixes-rc175' into rc175: review fixes (quality guard serial on one card, DeviceBytes as measured, CBF plain stores, GPU test guards), docs brought up to the code, GPU_DATA_ANALYSIS, acknowledgements, CHANGELOG ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-13 15:31:56 +02:00
leonarski_fandClaude Opus 5.5	563ad27af6	battery: workstation reference is the rc175 full battery at f853596 ... Replaces the kept 57fef1c run with 20261013-0557_f85359_rc175-f853596-full (open + inhouse; the private arm is never kept). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-13 15:31:44 +02:00
leonarski_f	7b4dcce480	Merge rc175 into review-fixes-rc175	2026-10-13 13:30:00 +02:00
leonarski_fandClaude Opus 5.5	30cf5856d4	docs: bring the rugnux and analysis pages up to the code (MR fallback, GPU decode, live runs) ... - CPU_DATA_ANALYSIS_IMAGE §0: CBF byte-offset and marCCD widening are decoded on the card; the host decoders are the CPU build's path. - CPU_DATA_ANALYSIS_DECISIONS §14: the null runs for every model the rigid body is tried on; the device choice is made once on the card's total (structure factors and rigid body a quarter, scale fits an eighth) and a CUDA failure ends the validation, deciding nothing; a subgroup the model fits is written as <prefix>_mr; new §14.10 on molecular replacement (model kind, rotation and translation functions, likelihood, Matthews, tNCS and P1 rules, null, what it may change). - CPU_DATA_ANALYSIS: scope and references gain molecular replacement. - CPU_DATA_ANALYSIS_INDEXING: the near tie decided early and re-checked once every frame is in. - RUGNUX: merging is the default and _process.h5 optional; live following of a collection with a broker URL; --model's MR fallback. - RUGNUX_INSTALL, RUGNUX_OVERVIEW, RUGNUX_TUTORIAL, RUGNUX_REPORT: GPU work list, <prefix>_mr outputs, MR in --mode mx for rotation and stills, rigid body only in --mode scale. - RUGNUX_ADVANCED: -N default as the usage states it. - battery README: the ci tier has 28 sets. - ModelValidation.cpp: a comment still said a GPU failure reran the validation on the CPU (comment only). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 09:03:31 +02:00
leonarski_fandClaude Opus 5.5	5636208278	battery: cytochrome c scored against the hand of its reference model; ZANUDA/POINTLESS evidence in the accepted-alternative texts ... The three cytochrome c rows take P 32 2 1 (154) from their reference models, whose 2-fold holds to measurement noise; XDS's label carried an arbitrary hand. The why-texts of 6oww, 6toc, 6pxb, 8xte and 8xtg now quote the tools/battery/symmetry evidence; no accepted answer changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-13 07:40:16 +02:00
leonarski_fandClaude Opus 5.5	2a32c28cca	battery nullmodel: a section title is '# N. TITLE', not any comment line starting with a digit ... The CC(model, data) table's rows ('# 4.13 ...') were read as section titles, so every key after the table in 5. MODEL VALIDATION counted as a decision outside it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 00:29:06 +02:00
leonarski_fandClaude Opus 5.5	d0d225cb99	rugnux --model: molecular replacement where the rigid body does not place the model ... One --model now does the right thing by itself: - A prediction (AlphaFold/ModelCIF, read from the file) goes straight to molecular replacement. - An experimental structure is placed by the rigid body as before. The null now runs on every --model run (no NOT_TESTED): "placed" = the fit beats its null (4.51 sd) in the answer's merge or in the twin hypothesis' merge. Placed: behaviour unchanged (hand and reindex may go into p.mtz); a subgroup the model fits and prefers is written as <prefix>_mr. - Not placed (or no crystal cell): molecular replacement with the experimental-model path. It may relabel the hand into p.mtz past the anomalous veto, never reindexes; another group or the twin subgroup goes only to <prefix>_mr. Maps, R-free and CC come from validating the MR placement with every decision switched off; MODEL_FIT stays the rigid body's verdict. - The twin hypothesis' merge is made in memory with a model where a metric re-ask promoted the group or the model's group has fewer rotations than the adopted one; written and reported only where the model chooses it. - --mr is kept as "--model, searched for by MR straight away". - Report: MODEL_PLACEMENT= RIGID_BODY|MR|NOT_PLACED, MODEL_DECISIONS_TAKEN gains MR_ENANTIOMORPH, MODEL_CHOICE_FILE_PREFIX/SPACE_GROUP; the MR keys move into 5. MODEL VALIDATION; MODEL_FIT absent where no rigid body ran. MR NOT_PLACED is never a warning. - Battery wrong-model step: check (c) also fails a wrong model placed by MR. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 22:20:36 +02:00
leonarski_fandClaude Opus 5.5	699bff1cb1	Merge branch 'sym-evidence' into rc175: POINTLESS/ZANUDA symmetry evidence for 18 open-arm sets, kept with the battery ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 16:02:51 +02:00
leonarski_fandClaude Opus 5.5	0bbe086b24	battery: symmetry evidence from POINTLESS and ZANUDA for 18 open-arm sets ... For every open-arm set where rugnux and the deposition differ in symmetry (sym_over, sym_screw, screw_undetermined, accepted alternatives) and the sets the twin review flagged, an independent assessment by established programs is now kept beside the manifest in tools/battery/symmetry/<id>/: a summary.md and the trimmed POINTLESS and ZANUDA logs. Each row names its directory in "symmetry_evidence". Nothing reads this for scoring, and no accepted or rejected answer changes. symmetry_evidence.py makes the evidence for one set. It re-runs the battery run's binary with the unmerged export (p.hkl must equal the run's) and runs POINTLESS on the observations. It merges them with AIMLESS in the deposited group and the model's setting, then runs ZANUDA with the deposited model on three free sets, each chosen per lattice-holohedry orbit. It runs POINTLESS and ZANUDA again on the depositor's structure factors, and tabulates the axial reflections of both data sets for screw questions. ZANUDA's preference can move with the free set (6toc), which is why there are three. What the programs found, reported neutrally: an exact pseudo-symmetry, a twin or another effect can produce any of these results, and the aim is only which answers the battery accepts. 6oww: ZANUDA prefers P 43 21 2 to P 21 on all four data sets, with lower R-free. 6toc and 8xtg: ZANUDA prefers the higher group, but R-free is mixed (6toc) or favours the deposited group (8xtg). 6pxb and 8xte: ZANUDA keeps the deposited P 32, and the deposited model in the higher group refines far worse. 3r6o, 4bwl, 2wnq, 8c3e: the deposited model gives the higher group no support. 5ebi: ZANUDA cannot reproduce the deposited fit. Screw sets: see each summary. Run on rc175 fcbe920 battery run 20261012-0139 with CCP4 9.0.014 (POINTLESS 1.14.2, AIMLESS 0.8.2, ZANUDA 1.097 / REFMAC 5.8.0431). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 16:02:01 +02:00
leonarski_fandClaude Opus 5.5	d172f22074	tools: remove jfjoch_mr_replay ... A developer tool, not built or installed upstream. It also failed to link in both portable builds (static cuFFT without the device-link step on Linux with CUDA, ws2_32 on Windows). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 12:59:43 +02:00
leonarski_fandClaude Opus 5.5	230c5e3917	battery: 2ygz leaves the open arm; 6oww accepts P 41 21 2 as an open question ... 2ygz is a withdrawn PDB entry of the same protein as 2wnq, so it scores nothing that 2wnq does not. 6oww's deposited 8-chain P 1 21 1 model obeys the 4-fold and 2-fold the higher group adds, and its |Fc|^2 correlate across the added 422 operators at 0.97-1.00, so P 41 21 2 is accepted beside the deposited group - marked as an open question, as no refinement has compared the two. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 09:47:39 +02:00
leonarski_fandClaude Opus 5.5	fcbe92042b	Merge branch 'af-mr3' into rc175: --af-model searches several copies per asymmetric unit (Matthews probability, sequential translation function, fixed-copy rigid body, tNCS) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 01:36:26 +02:00
leonarski_fandClaude Opus 5.5	63e0a85467	rugnux --af-model: several copies in the asymmetric unit, tNCS, a search-matched null, faster ... Copy number. The hard "Vm >= 4: MULTI_COPY, no decision" rule is gone. Copies are searched one after another up to the largest copy number with a Matthews probability of 1% (a Gumbel distribution with Kantardjieff & Rupp's 2003 mode and mean, no resolution dependence); COPIES_LIKELY reports the most probable. Sequential search. Copy n's translation function carries copies 1..n-1 as a known partial structure: the cross term 2 Re(conj(F_known) F(t)) goes on the same FFT grid (CPU FFTW and GPU cuFFT; KnownStructureFactors, AssemblyCorrelation). Its peaks, past those that clash, are ranked by the correlation of the whole assembly with E^2 (the TF itself, linear in |F|^2, rewards copying a placed copy onto an origin shift), the first that stands out (TF Z >= 8), packs and lowers R-free by 0.01 is kept; each copy is refined beside the others held fixed (RefineRigidBody's fixed copies), and the first copy again beside the second, the earlier ones once more at the end. Packing: ClashFraction counts C-alpha clashes with other copies and all their mates, binned (7mzt: 51 s of clash checks -> 1.9 s). Search quality: rotation-function peaks hill-climbed off the 6 deg sampling (6ze4: RF Z 10.4 -> 16.0, 3.3 deg off the deposited orientation -> 0.2); the first copy goes down the TF list until three placements that pack are refined, so a group is CLASH only where no offered orientation packs; groups are searched side by side, and every group the best one leads is searched again with the best one's copy orientations tried first, the better search kept. tNCS: the native Patterson of E^2 (5-10 A) off the lattice; placed copies' orientations join every further copy's search; and where the tNCS vector turns one group's screw axes into another's (TncsRelatesGroups), a group whose search stopped at fewer copies does not lose to the other - UNDECIDED. Measured on an in-house tNCS crystal (vector c/3 along the 3-fold): both hands fit with three copies to within 0.01 in R-free, and where one hand's search failed the other was taken as ENANTIOMORPH; now UNDECIDED. Null: the replicates are placed by the translation function in their own orientation, as the search placed the real one (a null at the real centroid reads high), and the gate is 4.51 = t(8) sqrt(1 + 1/9), 3 sd of a nine-draw null, the gate --model's MODEL_FIT is now read at. Speed: placements scored on the GPU structure-factor engines; further copies' trials score with the first copy's bulk solvent held, the assembly refitted at the end. Survey of 193 sets (jfjoch_mr_replay on stored merges, RTX 5080 shared with other jobs): MR median 1.15 s, max 24 s; demos 0.8-1.1 s. Report: COPIES and COPIES_LIKELY columns, MODEL_PRIOR_COPIES. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 00:27:17 +02:00
leonarski_fandClaude Opus 5.5	d906868bb2	battery: kept runs by machine role (keep --as workstation|server); the server run complete ... The kept-results directories are named by the machine's role, not its host name: tools/battery/results/workstation/<run>/ (1 socket, RTX 5080) and results/server/<run>/ (4 sockets, 4x T4). battery.py keep takes the role as --as; the kept files, untouched, still record the real host and hardware. The server's rc175 full battery now has all three files (open + inhouse, private false); checked for the private arm's set ids and input directories. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 22:33:35 +02:00
leonarski_fandClaude Opus 5.5	695bc907b4	battery: keep runs in the repository (battery.py keep, tools/battery/results/<host>/<run>/) ... Run directories are scratch, and the paper tooling needs the runs it draws from to outlive them. battery.py keep RUN copies a run's report.md, results.json and manifest.json as written into tools/battery/results/<host>/<run>/ (outside docs/, which Sphinx publishes), the host and run name read from the report. Open and in-house runs only: a run its manifest marks private, whose report is report_PRIVATE.md or says PRIVATE ARM, or with an arm or row outside those two, is refused. A run of which only report.md exists is kept with what there is. The CI job now keeps results.json and manifest.json beside report.md, in /data/battery/ci_reports/<run>/ and in the artifact, so a run on either host leaves what keep needs. paper_plots.py resolves a RUN by name among the kept runs after runs_root. Kept: the rc175 full battery on mpc2898 (20261011-1440_57fef1_rc175-57fef1c-full, all three files) and on mx-aare-test (20261011-1236_549939_1.0.0-rc.175, report.md only; the other two are to follow from that host). Both checked for the private arm's set ids and input directories. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 22:22:29 +02:00
leonarski_fandClaude Opus 5.5	eec8dcee9f	battery: the wrong-model null as a CI step of its own (battery.py nullmodel) ... The battery job ran the ci tier and the wrong-model null in one step with one exit status, so a red job did not say which of the two failed. battery.py nullmodel runs the null on its own, in a run directory whose report is just that section, and exits 1 when a pair failed; run --no-nullmodel keeps it out of a --tier ci run. The workflow runs `run $SUBSET --no-nullmodel`, then (for a ci-tier run) `nullmodel` as a separate step that runs whatever the first did, and keeps both reports. The ci tier's own rows are unchanged; a plain `run --tier ci` still includes the null as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 21:30:34 +02:00
leonarski_fandClaude Opus 5.5	f800669839	jfjoch_mr_replay: --af-model's molecular replacement re-run on a run's merged MTZ files ... A developer utility, not installed: the search, the decision and the hand check run on the answer's p.mtz (and the twin hypothesis' p_twin_<group>.mtz) that a rugnux run wrote, so a survey of the search over every dataset with a predicted model takes minutes instead of a reprocessing per set. The hand check of an ENANTIOMORPH decision moves from the rugnux driver into CheckEnantiomorphHand, which both call; unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 20:50:47 +02:00
leonarski_fandClaude Opus 5.5	4d17aae4e3	battery: wrong-model null takes about 6 minutes of rugnux (measured) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 14:36:12 +02:00
leonarski_fandClaude Opus 5.5	0a3fadb9ef	battery: wrong-model null step, run with every ci-tier run ... A ci-tier run (or any run given --nullmodel) now also runs five ci-tier sets of the open arm (8u0i, 6iu5, 9khr, 9b22, 6pxc) once without --model and once with each of five WRONG models, the deposited coordinates of other open-arm entries (already in the pdb_cache, so no new data): similar size, sequence identity under 25%, gesamt Q-score at most 0.20, one hard-ish model of the same secondary-structure class per set, one in the same space group or a similar cell, and for the two enantiomorphic sets one in the other enantiomorph. nullmodel.json records every model's size, identity, Q-score and reason. Each wrong-model run is scored against the no-model run (nullmodel.py): (a) p.hkl, the p.mtz reflections and p.cif identical, (b) no KEY= line outside MODEL VALIDATION moved, (c) MODEL_FIT not ACCEPTED, no decision taken, CC_MODEL_CONFIRMED_TO_D_MIN NONE and R_free >= 0.50 (Wilson's random-structure R is 0.586 acentric). The pairs go to nullmodel_results.json and a report section of their own; the sets' own rows are untouched. A failed pair fails the run, and the Gitea battery workflow prints the section into the job log. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 11:50:21 +02:00
leonarski_fandClaude Opus 5.5	467ac90226	Merge branch 'nocrystal' into rc175: the no-crystal test asks its own significance (1e-9); exact tail logged; nothing_1/2 join the ci tier ... Exact: p.hkl md5 and MTZ data identical on every real set (34 open/inhouse incl. the ci tier, all 27 private); both no-crystal controls now ~7 decades inside the bar instead of 0.3-0.9. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:03:35 +02:00
leonarski_fandClaude Opus 5.5	7d5551fd86	battery: the no-crystal controls join the ci tier ... nothing_1 and nothing_2 now pass the no-crystal test by about seven decades (tails 10^-2.06 and 10^-2.70 against 1e-9) instead of 0.3-0.9, so a change that moves one stray reflection no longer flips them, and a regression that lets a chance lattice through fails CI. The ci tier is 28 sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:02:58 +02:00
leonarski_fandClaude Opus 5.5	49384c4aa8	Merge branch 'mv-ncs' into rc175: models expanded by their strict NCS operators; the written cell follows the model's indexing ... Exact: p.hkl md5 and MTZ reflection data identical on 18 sets. 4nwv R-free 0.545 -> 0.297 and cell in the depositor's frame; written cells corrected on 9rcs, 9b22, 9qw8, myob_x06da_powder_1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 13:36:31 +02:00
leonarski_fandClaude Opus 5.5	1f27fba1f3	Model validation expands strict NCS; the written cell follows the model's indexing ... - ModelValidation reads the model and, where it carries non-given NCS operators (_struct_ncs_oper 'generate' / MTRIX without the given flag), expands them (gemmi expand_ncs, vendored assembly.cpp from GEMMI 0.7.5, MPL-2.0 as the rest of gemmi_gph). Same in the battery's model_check.py and depdata_check.py. An icosahedral-virus entry deposited as 3 of the 45 chains of its ASU was scored against a fifteenth of its atoms: R-free 0.545 -> 0.297. - The indexing the model chooses (AdoptModelFrame) now moves the cell with the indices (reseat_integrated in the main path; CellInBasis in the forecast, twin-file and stills/scale paths). A near-symmetry law (b within 2% of c, beta a degree off 90, a~b in P1) relabelled the reflections only, so the files were written with two axes swapped or beta flipped relative to the indices. Where the law moved the cell, the validation is remade on the moved cell. Measured on 18 open/in-house sets with models: p.hkl and MTZ reflection data bit-identical on all; cell and model R unchanged where the law is exact; the cell now matches the deposited one on the 6 pseudo-merohedral cases (orthorhombic a/c swap, three monoclinic beta flips, a P1 alpha/beta swap), R-free within +-0.001 there. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 13:35:49 +02:00
leonarski_fandClaude Opus 5.5	8c8fb24779	battery: a timeout or interrupt kills rugnux's whole process group ... subprocess.run's timeout killed only the direct child. Under --gpulock that is flock, under `stress --sanitizer` compute-sanitizer, and in both cases rugnux went on running on the card, orphaned, beside whatever took the GPU lock next - found when a cut-short memcheck stress run left its rugnux behind. Each run now gets a process group of its own, killed whole on a timeout, Ctrl-C or SIGTERM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:57:38 +02:00
leonarski_fandClaude Opus 5.5	7f4c9e2f8a	battery README: cut the sweep for compute-sanitizer stress runs; read the Xid after a device fault ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:46:06 +02:00
leonarski_fandClaude Opus 5.5	d495a01de4	battery: a handled GPU error is a crash only when it is a device fault ... A planned fallback that rugnux logs with a CUDA error text - the rigid body's zone over its engine's reserve, revalidated on the CPU, or an indexer worker out of device memory while other jobs held the card - would have turned sets into crashes in every run. A handled error now counts only when it names a device fault (unspecified launch failure, illegal memory access/address, misaligned address, launch timeout, device-side assert, "CUDA device unusable"), which takes the context down for every thread and is never a property of the data. "Processing failed" still counts whatever its cause. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:24:48 +02:00
leonarski_fandClaude Opus 5.5	1fc06f6e6c	battery: a crash is its own verdict and fails the run; stress subcommand ... A run that was killed by a signal, or whose log has "Processing failed", a GPU error (also one rugnux handled and went on from), "terminate called" or a non-zero compute-sanitizer ERROR SUMMARY is now verdict `crash` (score.crash_note), decided before anything else, so it fails even a no-crystal control whose normal refusal exits 1. A crash counts as a failure in every pass rate, gets a CRASHED section at the top of the report and a CRASHED line under the console table, and makes `run` exit 1 whatever tier the set is in. Reports and compares re-read crashes from the log, so an older run's crash shows as one: the GPU crash of 9ig7 in the last full run showed only as "fail, exit 1: Step 8 FAILED" before - failure_note now prefers the [error] line it follows. `battery.py stress` runs named sets --repeat times, --concurrent at once, optionally under compute-sanitizer (memcheck/racecheck/synccheck/initcheck) or with CUDA_LAUNCH_BLOCKING=1, and exits 1 on any crash; it reports distinct p.hkl per set and keeps out of the run list. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:00:50 +02:00
leonarski_fandClaude Opus 5.5	d8a3c6ea15	battery: add five large-cell sets to the open arm ... 4MTK (P61, c 653 A, PILATUS 6M miniCBF, SBGrid 235), 4NWV (I222 virus-like particle, 370-411 A axes, Rayonix MX-300 marCCD, SBGrid 9), 7D98 (P6122, c 602 A, EIGER 4M miniCBF at 1.9 A, XRDa 13), 9V16 (P41212, c 451 A, EIGER1 16M HDF5, XRDa 338) and 4P8R (P6522, c 701 A, Rayonix MX-225 marCCD, IRRMC), tagged large-cell (plus long-axis / virus). References are the deposited space group, cell and resolution. 7D98 is pinned to its 250-frame miniCBF sweep: the HDF5 masters beside it repeat frames 61-260 in 60-frame pieces. 4P8R's archive ships per-frame .bz2 files, decompressed after extraction. EXTERNAL_TEST_DATA.md gains the five rows (DOIs resolved), the archive notes and the updated collection counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 20:31:28 +02:00
leonarski_fandClaude Opus 5.5	7624bbe12e	battery: paper_plots.py draws the other figures when no run has REFMAC R-free ... The REFMAC check is opt-in, so a battery run without --model-check has no R-free pairs, and the axis range min()/max() over them crashed the script before any figure was drawn. The R-free figure is now its own gnuplot block, left out (with a line in summary.txt saying why) when there is nothing to draw; d_min and the time figures are drawn as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	d28c8b8bb3	battery: same-absence groups of one Laue class pass as indistinguishable_alternative ... sgequiv keyed space groups on their proper+improper rotation matrices, so groups with identical merged intensities but different point groups (I41md vs I-42d, Pc vs P2/c, Pna21 vs Pnma, P2 vs P2/m) were called a "REAL difference", contradicting its own docstring, and score.py compared point groups literally on the XDS arms. A run that wrote one member of such a pair and listed the other in SPACE_GROUP_ALTERNATIVES failed as sym_other / sym_over. sgequiv now keys each group, in every setting gemmi tabulates, on its Laue rotations (operations plus their negatives), centring and absences; describe() separates the new different-point-group pairs from the old same-point-group ones, and enantiomorph handling is unchanged. score.py passes a row, on both arms, when some setting of the reference has the same signature as our group in our own setting: verdict pass, cause indistinguishable_alternative, the reason saying whether rugnux listed the reference. Pass, not unscored: unlike an unmeasured screw axis, everything was measured and the answer is as right as the data allow; only a model or a strong anomalous signal decides. The separate cause keeps such rows countable. Being listed in SPACE_GROUP_ALTERNATIVES alone is deliberately not enough: that list also carries the screw alternatives of an undetermined axis, whose absences differ, and those stay under the screw rule. The open arm's same-point-group UNDECIDABLE pass now also requires the setting check. results.json gains sg_alternatives (rugnux's SPACE_GROUP_ALTERNATIVES). KDP joins the ci tier. Re-scoring the open and in-house arms of the two latest full runs changes one verdict in each: kdp_x10sa_20keV fail/sym_other -> pass/indistinguishable_alternative. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 16:27:23 +02:00
leonarski_fandClaude Opus 5.5	366935fe26	battery: right-align the reference values of the run table too, so decimal points line up ... Each numeric pair is now rugnux and reference both right-aligned to a fixed width around the "/", with fixed decimals per column (d_min 2, ISa / CC1/2 / R_meas / R-free 1, time 0), so the slashes and the decimal points of a column sit on one line; "-" takes the same width. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:39:20 +02:00
leonarski_fandClaude Opus 5.5	c5970a76f4	ci: the battery workflow keeps the battery machine's own paths and mounts ... The workflow runs on the battery machine, not on the development workstation, so the data roots (/data/open, /data/inhouse), the whole /data/battery mount, the scratch directory (/data/battery/scratch/<run id>), the kept reports (/data/battery/ci_reports) and the runner's valid_volumes are back as that machine has them; the remark tying the image to one host's driver is gone. The private arm stays out because the site config names only open and inhouse. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:38:43 +02:00
leonarski_fandClaude Opus 5.5	9beb8a7813	battery: align the run table's pairs on their slashes, the ticks in a column of their own ... Each rugnux / reference pair is now rugnux right-aligned to a fixed width, "/", the reference left-aligned to the same width ("-" for a missing value), so every slash of a column sits on one line; the tick or cross follows the space-group pair in its own column. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:37:50 +02:00
leonarski_fandClaude Opus 5.5	8c6670a3f4	battery: a ci tier, a closing table per set and a failing exit on it ... `battery.py run` now ends with one line per set, each column rugnux / reference: the space group with a tick or cross by the scorer's own verdict, d_min, ISa, CC1/2 overall and in the highest shell, R_meas of the lowest shell, the model's R-free and rugnux's wall time. `battery.py table RUN` prints it for any finished run (rows are rescored, so older runs get the new columns). The highest-shell CC1/2 is new in results.json (cc_half_high, read off rugnux's own shell table) with XDS's beside it (cc_half_high_ref, from CORRECT.LP's finest shell; the in-house manifest's refs regenerated with `refs --write`, which added only that key). A new "ci" tier: 25 sets that passed with the same space group and steady d_min / ISa in all four rc174/rc175 full runs (and in the rc175-next verdict runs where they ran), with no warning that questions the lattice or the group. About 10 minutes of rugnux (616 s measured) on the RTX 5080. A run that includes ci-tier sets exits 1 when one of them fails (lattice, symmetry, crash, timeout, missing input); nothing else fails a run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:31:32 +02:00
leonarski_fandClaude Opus 5.5	8e134c405d	tools: run the broker on a stored dataset, without a detector ... For demonstrating and testing a collection followed while it is written: - jfjoch_stream2_replay replays a stored dataset as a DECTRIS detector's stream2 (ZeroMQ PUSH on the stream2 port): start message, every image at a given rate, end message. - fake_simplon.py stands in for the detector's SIMPLON control interface: answers the configuration and status requests the broker makes, accepts every command, reads back idle. It needs port 80. Not installed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 15:52:22 +02:00
leonarski_fandClaude Opus 5.5	f97700e084	battery: read 8owm and 8rud as the deposited .cbf.gz; 6ZR0's zip is intact ... The MXRDR zips for both sets hold gzip-compressed frames. Our staging had expanded them to .cbf. rugnux reads .cbf.gz natively, so the manifest now points at the files exactly as deposited. 6ZR0's zip, as the Keele repository serves it, has a readable central directory with all 1060 members. Our earlier download was short, which lost the last frame. Remove the statement that the repository's zip is damaged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 19:59:40 +02:00
leonarski_fandClaude Opus 5.5	f3c3ea5d3c	battery: add 2ygz to the open arm ... A P21 crystal in the same form as 2xfw, but pseudo-merohedrally twinned with a twin fraction of 0.33 at an obliquity of 0.29 deg. That is the largest twin fraction at the highest obliquity in the arm. The PDB entry was withdrawn, so the reference cell and resolution come from Table 1 of Campeotto et al. (2018) Sci. Rep. 8, 14876, and the set runs without a model. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 17:09:15 +02:00
leonarski_fandClaude Opus 5.5	5421c569ed	rugnux: log the time of each section of the run's tail at debug level ... A TimingMark (common/Logger.h) logs "TIMING <section> <seconds>" at debug level when a section ends, so the lines go to <prefix>_log.txt and stay off the terminal. Marked: each pass of RunPipeline (pre-pass, probe, output) and its first-pass indexing, image pass, scale/merge/symmetry phase, output files and census wait; inside the scale/merge phase every merge (labelled by its role: P1, P1 space-group search, P1 all observations, the adopted group, P1 cross-check, quality guard), the post-refinement, the space-group search and its two SearchSpaceGroup arms, adoption, analyses, the P1 cross-check wait, model validation and writing; inside RotationScaleMerge the ingest and, per arm, partial scaling, combine and fulls scaling, correction surfaces and the merge; the console's steps; and the results report. tools/rugnux_timing.py tabulates the tail of a log. Only the log changes: the outputs are byte-identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 16:13:17 +02:00
leonarski_fandClaude Opus 5.5	37b7fdd38d	rugnux: the terminal shows numbered steps with timing; the full log goes to <prefix>_log.txt ... Standard mode prints only steps, one-line results with the time since the line above, a rule closing each step with its total, warnings (!!) and errors (** ERROR:), and a summary at the end. Every log line of every module goes to <prefix>_log.txt instead, at debug level and timestamped, opened with version, build, GPU, threads, command line and the header geometry. -v puts the whole log back on the terminal. - RugnuxObserver gains OnStep / OnNote. RunPipeline posts them only on a pass that makes the result (the observer is held in steps_, nullptr on probe passes and on copies run beside the main pass); RunAllPasses opens steps for the checks between passes (rotation scale, metric symmetry, geometry walk) and notes its re-runs. OnPhase is written to the log as a marker. - RugnuxConsole (CLI only): on a terminal a progress bar and a spinner status line drawn in place with \r, durations only; not a terminal, plain ASCII with the time of day on every line and quarter milestones for long passes. isatty/_isatty, no environment variables. A warning repeated by later merges is shown once more on one line and then counted per step. - The CLI's former stdout blocks (space-group search, merging statistics, per-image cost, processing time, tNCS/twinning/anisotropy in --mode scale) go to the log; the calibration result block stays on the terminal as that mode's summary. The MX summary takes its rows from the report's own SUMMARY (ReportDocument::summary_rows); WriteResultReport returns the document it rendered. HDF5's automatic error-stack printing is switched off. - The banner keeps the "Version X (git abcdef" form battery.py reads off a bare invocation; battery.py's failure note now reads p_log.txt, whose lines are not wrapped. Output only: p.mtz is bit-identical on the three in-house reference sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:11:21 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00