249 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	51706641fa	RESULT-CHANGING: molecular replacement's transform FFT on the GPU wherever it fits the search's budget ... The molecular transform's FFT went to the GPU only where its buffers took at most a quarter of the card (MapToFPhiGPUFits), the last quarter rule left in molecular replacement. On a large cell that is never: 4nwv's 768^3 box needs 5.45 GB, so on a 16 GB card it ran on the CPU in FFTW, single-threaded - 12.5 s of the search's 31 s. Now it is judged, as every other part of the search, against the search's budget (80% of the card less what is resident; the search holds nothing else yet when it runs), and the device writes only the cropped transform the search keeps (CropHalfL, moved to ModelFFT), so it needs 4.46 GB instead of 5.45. cuFFT and FFTW agree to float rounding, not bit for bit, so wherever the transform moves to the GPU the search's functions can differ in their last bits: on 4nwv on 16 GB and 8.55 GB cards it moves (it now does on any card above about 5.6 GB, before above 21.8 GB); the small boxes were on the GPU before and give the same values (the same kernel, cropped). Measured on the RTX 5080, timing benchmark: 4nwv's transform 14.1 -> 2.2 s, MR 31.6 -> 20.1 s, run 93.0 -> 89.2 s; MR's peak device memory unchanged at 8.7 GB (the FFT's own 5.2 GB comes before anything else is held). On this card the reflection files, maps, placed models, reports and the MR candidate lines of myob, cytc and 4nwv still came out byte-identical to rc175-metal-merge. [MolecularReplacement] passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 13:41:48 +02:00
leonarski_fandClaude Opus 5.5	011b98838e	Molecular replacement: one GPU plan per merge on model validation's budget, the transform cropped, output unchanged ... What molecular replacement put on its GPU was decided part by part on a quarter of the card each: every group's rigid-body pool (up to three engines), every null's (up to nine), the score engines' cache, the --mr hand check's structure factors. The groups and the nulls run side by side, so the quarters added up: on 4nwv on a 16 GB card the search held 4 rigid-body engines of 1.88 GB, two score engines of 1.64 GB, the 1.82 GB molecular transform, two translation functions' 1.2 GB and the 2.4 GB the allocation pool kept from the validation before - 13.8 GB at its peak, 83% of the card. On an 8.55 GB card that pages. Now each merge's search is planned once, before it starts, on the budget model validation uses (ModelGPUBudget: 80% of the card's total, less what a merge engine still at work holds): the transform, one rigid-body engine and the largest score engine take their place in that order where they still fit, more rigid-body engines only within half of the budget, and what is left is a room the score engines (least recently used given up first, as before) and the rotation and translation functions' working buffers share, each waiting for the other to finish rather than spilling past it (PlanMolecularReplacementGPU). A part that does not fit runs on the CPU, decided on its size alone. - One rigid-body pool per merge serves every fit of the merge - the groups' first and further copies, the nulls, the tNCS shifts, the carried copies - sized for the largest (the most copies Matthews allows, in any group) and reserved once, instead of a pool made and freed for each. RigidBodyGPUPool::Cover widens a planned pool to another model or group; its zones are kept by group, atom count and resolution. RigidBodyGPUPool::Create (the quarter rule) is gone. - The sizes are worked out on the CPU while the transform is computed. - The molecular transform keeps only the indices it is read at (|k|, l <= box / d_min + 1): 0.84 GB instead of 1.82 on 4nwv, host and device. Its rows keep their full length: cut short too, the rotation function read them 15% slower. - The translation function's prepared setup is kept on the host and uploaded by each call, so the device holds it only while a function runs; the cross-term offsets are uploaded only where there are cross terms. - The allocation pool is trimmed before the search, after the transform's GPU FFT, before a score engine is made and before the hand check, whose structure factors are now judged against the budget. - rugnux passes the merge engine's bytes, where the finalist ledger keeps it, to the search, the hand check and the validation of the placed model. Measured on the RTX 5080 (16.6 GB), timing benchmark: peak device memory during MR 13.8 -> 8.7 GB on 4nwv (start 3.0 -> 0.66 GB after the trim), 1.67 -> 1.14 GB on cytc, 6.3 -> 0.76 GB on myob. On 4nwv the plan holds the transform 0.84 GB, one rigid-body engine of 2.10 GB and leaves 10.35 GB of a 13.29 GB budget to the score engines and the functions; a second engine would pass half of the budget (one engine against two: +1.2 s on the rescoring and the null). MR 31.7 -> 31.6 s on 4nwv. Reflection files, maps, placed models, reports and the MR candidate lines byte-identical to rc175-metal-merge on myob, cytc and 4nwv; [MolecularReplacement], [ModelValidation], RigidBody*, ModelScale*, [CUDAMemHelpers] pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 13:33:39 +02:00
leonarski_fandClaude Opus 5.5	1ebb28e3ab	Merge cytc-mr into rc175: molecular replacement decides an R-free tie between groups with as many copies on every reflection (paired, 3 standard errors), else UNDECIDED as before ... Cytochrome C from its sequence-exact AlphaFold model now places in P 32 2 1; MR replay survey of 193 sets: UNDECIDED 7 -> 4, all three newly decided correctly, no wrong decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 11:44:13 +02:00
leonarski_fandClaude Opus 5.5	6cdc063b4c	Molecular replacement: an R-free tie between groups with as many copies decided on every reflection ... Two space groups of a merge whose placements are within R_FREE_MARGIN (0.03) of each other in R-free left the choice UNDECIDED. Where both placed the same number of copies they have the same parameters, so the working reflections compare them as fairly as the free ones, and there are about twenty times as many: the other group is now ruled out where the chosen placement's |F_obs - F_calc| is lower by 3 standard errors of the paired per-reflection difference over all reflections to the scoring limit. Groups with different copy counts, and the tNCS carry rule, are unchanged. Found on the timing benchmark's cytochrome C set started from the bovine AlphaFold model (the sequence-exact one): three copies placed in both P 32 2 1 (R-free 0.427) and P 31 2 1 (0.437), so MR_DECISION was UNDECIDED. Not a regression: the 2026-10-11 tNCS work ran this set with the horse model, whose first copy did not place in P 31 2 1 at all, so P 32 2 1 was chosen with no tie (still so at rc175 HEAD). With the 380 free reflections at 3 A the two hands differ by 0.010 +- 0.012 in R-free; over all 7285 reflections by 0.029 at 11 standard errors. Now ENANTIOMORPH P 32 2 1, 3 copies, placement CC 0.744. MR replay survey of 193 sets (AlphaFold models): multi-copy UNDECIDED 7 -> 4, all three now correct (7q6j P 21 21 21; both cytochrome C x06da sets P 32 2 1); one PAE-domain case (8tha) breaks its whole- model hand tie towards the deposited group but stays UNDECIDED by the domain rule; no other set changes, no wrong decision. Every tie the rule decided did so at 8.5-12.4 standard errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 11:43:33 +02:00
leonarski_fandClaude Opus 5.5	36062a55f1	Merge mv-gpu-budget into rc175: model validation plans its GPU parts against one budget, 80% of the card's total, after trimming the allocation pool and releasing the FFT indexers; extra engines only within half of it ... On the RTX 5080 nothing moves between GPU and CPU and the output is byte-identical; on cards below about 9-10 GB parts that used to fall back to the CPU now stay on the GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 10:15:42 +02:00
leonarski_fandClaude Opus 5.5	3373106ae1	Model validation: budget counts only what is really held; fewer extra engines ... On an 8.55 GB card (WDDM) the validation planned 6.81 GB of a 6.84 GB budget, but the card already held more than the 20% headroom allowed, and the driver paged GPU memory to system RAM: every rigid-body fit ran ~5x slower. Measured on 4nwv on a 16 GB card, at the start of model validation the device held 1.1 GB beyond a fresh context, and on 6oel 2.7 GB: - the stream-ordered allocation pool's free-but-reserved memory, left by the image passes and the scaling's pooled buffers (1.07 GB on 4nwv, 2.1 GB on 6oel); the validation's engines are allocated outside the pool and cannot reuse it; - the image pass's FFT indexers (IndexerThreadPool), alive for the whole run (0.8 GB on 4nwv, 0.5 GB on 6oel), although nothing indexes after the merge. Both are now released: the validation trims the pool (cuda_trim_allocation_pool) before it plans, and rugnux drops the indexer pool once the leftover-lattice census, its last user, has ended, before the model is validated or searched for. What is left beside the validation is 0.06 GB over a fresh context on both sets (streams, idle cuFFT plans). Extra engines are now added only within half of the budget. They change no number and bought little where measured (4nwv, 16 GB card: 13.6 s with one rigid-body engine, 13.1 s with two, 12.6 s with five), so the card keeps headroom instead. On 16 GB 4nwv now plans 2 rigid-body and 2 scale-fit engines, 6.56 GB (measured peak 7.3 GB with the context); on an 8.55 GB card one of each, 4.42 GB. Reflection files, maps and models byte-identical to rc175 HEAD on 4nwv, 8tyy, 6oel and 3inp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 09:56:15 +02:00
leonarski_fandClaude Opus 5.5	b9fd1c8d75	Merge timing-bench into rc175: battery.py timing, a three-set benchmark (myoglobin and cytochrome C from AlphaFold DB models, 4nwv from a moved deposited model) placed by molecular replacement, checked and timed per step ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 09:25:34 +02:00
leonarski_fandClaude Opus 5.5	f99ed0b5ac	battery.py timing: a three-set benchmark placed by molecular replacement, checked and timed per step ... `battery.py timing --rugnux BIN` runs myob_x10sa and cytc_x10sa from their AlphaFold DB predictions (AF-P68082-F1 and AF-P62894-F1, v6, committed verbatim with their predicted aligned error under tools/battery/models/, CC BY 4.0) and 4nwv from its deposited model with strict NCS expanded and moved by a fixed rigid-body transform (30 deg about (0.36, 0.48, 0.80) through the centroid, shift (12, -18, 25) A), made at run time from the site's pdb_cache. Each set runs the battery's own command for its arm, in an ordinary run directory, so the site config, results.json, report, table and compare all apply. A set passes when the battery's own verdict passes, MODEL_PLACEMENT is MR, MR_SPACE_GROUP is the expected group with its hand, R_FREE is under its limit, CC_MODEL_CONFIRMED_TO_D_MIN is not NONE, and the placed model's intensities correlate with the reference model's (|F|^2, shell-normalised, 10-4 A, best over the lattices' changes of basis) at 0.30 or more. That last check is independent of rugnux's scores: correct placements give 0.55 / 0.90 / 0.96; the myoglobin placement turned by 30 deg with rugnux's report unchanged gives 0.035, the unplaced 4nwv start model 0.011, a placement relabelled to the other hand 0.18. Rows carry the checks and the per-step times from the TIMING step lines; the report gets a Timing benchmark section; the run exits 1 on any failure. On the workstation (RTX 5080, rc175 8eacce): myoglobin passes in 10 s (R-free 0.44), 4nwv in 106 s (MR 34 s, R-free 0.315 against 0.297 for the deposited placement). Cytochrome c fails: MR places three copies of the monomer in both P 32 2 1 and P 31 2 1 within 0.03 in R-free and so places neither (MR_DECISION UNDECIDED), where the three-copy reference model decides the hand at once. battery.py now imports fcntl only for the full-run lock, falls back to Popen.kill where there are no process groups, and keeps the binary's extension in the run directory, so a timing run can work on Windows (untried there). Licence and credit: licenses/alphafold-db.txt (from models/LICENSE.txt via COLLECT.sh), a new "Data files in the repository" table in THIRD_PARTY_NOTICES.md, and the AlphaFold DB with Jumper et al. 2021 and Varadi et al. 2022 and 2024 in docs/ACKNOWLEDGEMENT.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 09:01:46 +02:00
Filip LeonarskiandClaude Opus 5.5	229a432a66	RESULT-CHANGING (owner approved, ulp level): Metal profile fit, CPU Gaussian from a shared exp ... Pass B of BraggIntegrationEngineCPU is split where a device can take over: per reflection, what the fit needs is worked out in double (BraggFitSetup), the profile, its readable mass and the reweighted fit run in single precision (FitProfile -> BraggFitOut), and the error, the summation guard and the result are finished in double. BraggIntegrationEngineMetal runs the middle as a Metal kernel, one thread per reflection, summing in the CPU's order (FitOnDevice); the CUDA engine is untouched. The Gaussian's exp is now BraggExp (BraggExp.h), on the CPU and on the GPU: no GPU exp reproduces the host's expf, and BraggExp uses only +, -, * and exact power-of-two scaling, so both compile it to the same bits. Checked on the Apple M6 over every float in [-110, 89] (2.2e9 arguments), with subnormal products and sums matching too. It is closer to correctly rounded than macOS expf (max 0.53 ulp; 0.2% of arguments below -1 not correctly rounded vs 0.62%). This is the one result change: per observation |dI|/sigma p99.9 <= 1.7e-6, max 1e-4, 10-30x below what ba3f75475 moved. Merged statistics (resolution, completeness, CC1/2, R_meas, I/sigma, ISa, space group) are identical on lysozyme, cytochrome C, myoglobin, a small-molecule set and 4nwv; only tNCS fields that flip on any last-digit change move (cytochrome C's pseudo-translation flag, whose modulation ratio sits at the 2.5 threshold, turns off). The Metal engine stays bit-identical to the CPU engine: BraggIntegrationEngineMetalTest passes unchanged, and catches planted kernel bugs (another exp, 3 iterations, a dropped lost-peak test, <= for the disk, inverted exclude logic). Measured wall time, Apple M6 (CPU pass B -> GPU pass B): lysozyme 24.7 -> 18.6 s, myoglobin 39.8 -> 35.7 s, cytochrome C 59.3 -> 49.0 s, 4nwv 150.2 -> 131.3 s; the small-molecule set, sparse frames, 92.5 -> 95.6 s (noise of +-3 s). Metal's fast::exp in the kernel instead measured the same on every set (18.6, 35.8, 49.1, 131.2 s), so it would buy nothing for the bitwise check it would cost. On the CPU alone BraggExp costs nothing measurable (cytochrome C --no-gpu 72.8 -> 73.1 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 08:18:43 +02:00
leonarski_fandClaude Opus 5.5	0d25d765cf	Model validation: one GPU budget for all its parts, 80% of the card's total ... What a validation puts on its GPU was decided part by part on fixed fractions of the card: the structure factors to d_min on a quarter, the null's structure factors on whatever of that quarter the first left, the rigid-body engines on another quarter, the scale fits on an eighth. On an 8 GB card that sent the null's structure factors of a 4.4 GB validation to the CPU (and held the rigid body to one engine) although everything the validation holds fits the card with room to spare. Now PlanModelValidationGPU decides all of it at once, before anything is reserved, from the card's total memory: every engine is reserved up front and kept to the end, so their sum is held against one budget, 80% of the card less what a merge engine still at work holds there (the finalist ledger's P1 merge, RotationScaleMerge::DeviceBytes). Each part takes one engine where it still fits, in the order structure factors to d_min, rigid body, the null's structure factors, scale fits; a part that does not fit runs on the CPU. What is left goes to more rigid-body engines (up to one per null replicate plus the real fit) and then more scale-fit engines, which only change how many fits run at once. A second validation may run beside where twice this one's whole GPU plan fits the cards' budgets; it gets this one's bytes as gpu_beside_bytes, which limit its engine counts and never its placement, so its numbers are those of the validation it stands in for. One log line states the placement, the bytes held and the budget. - RigidBodyGPUPool::Plan sizes the pool without engines (EngineBytes), Reserve(n) makes them; Create (Plan + engines on a quarter of the card) stays for molecular replacement, whose own rules, and the --mr hand check's structure factors (ReadModelAnomalousMap, a quarter), are unchanged. - The null's resolution is worked out before the plan, so its engine is decided and reserved with the others instead of after the indexing probe. Placement changes only on smaller cards: with the logged footprints of the open-arm sets, 4nwv's null structure factors move to the GPU below ~9 GB (and its rigid body at 6 GB), 8tyy's structure factors to d_min below ~8.7 GB; on 16 GB and larger nothing moves. On the RTX 5080 (16 GB), 4nwv, 8tyy, 6oel and 3inp give byte-identical reflection files, maps and models to rc175 HEAD, with more rigid-body engines where they fit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 08:05:50 +02:00
Filip LeonarskiandClaude Opus 5.5	f05e67b2b4	Metal Bragg integration engine on macOS, output unchanged (bit-identical to the CPU engine) ... BraggIntegrationEngineMetal is BraggIntegrationEngineCPU with its integer pixel work run as Metal kernels, one thread per reflection: the neighbour mask, the owner map, pass A's walk over every box and the background clip. Everything in double, the profile learning and pass B stay on the CPU, in the CPU engine's own code - Apple GPUs have no double, and no GPU exp matches macOS expf bit for bit. The sums are integers, so the GPU's order does not matter; the stencil arithmetic is the same BraggStencil.h, compiled at run time (MSL 3.0, no offline Metal toolchain needed) with fast math off (sqrt and division correctly rounded, measured) and fp contraction off, as the host already builds (-ffp-contract=off, now extended to Objective-C++). The image is read in place (unified memory). The CPU engine is split so both engines share the finishing code: pass A counts into BraggBoxSums (integer; the double ring sums were exact integers already), the clip threshold is its integer floor, FinishPassA and ProfilePasses are shared. Verified byte-identical p.hkl and MTZ data before/after the split and with the Metal engine on lysozyme, cytochrome C, myoglobin, 4nwv and a small-molecule set. Measured on an Apple M6 (wall, CPU engine -> Metal): 4nwv 180.5 -> 151.2 s, cytochrome C 72.7 -> 59.8 s, lysozyme 29.2 -> 22.5 s, myoglobin 43.3 -> 41.2 s; the sparse small-molecule set +1.5 s (few reflections per frame leave the GPU idle). CI: a macOS step runs tests/BraggIntegrationEngineMetalTest.cpp, which requires every float of every reflection and every count to match the CPU engine over 19 setting combinations on random crowded frames (negative pixels, exact-pixel ties, hot, masked and overloaded pixels), three frames per engine; it fails without a Metal device and if the GPU path fell back to the CPU. Ten planted kernel bugs were each caught. rugnux --no-gpu hides the Metal device too (gpus_are_hidden). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 23:18:42 +02:00
leonarski_fandClaude Opus 5.5	76adf3f0fa	Merge origin/rc175 (macOS signing and notarization) into rc175 ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 18:47:07 +02:00
Filip LeonarskiandClaude Fable 5.1	82e5de1261	macOS: sign jfjoch_viewer and rugnux with a Developer ID and notarize them ... JFJOCH_MACOS_SIGNING_IDENTITY (the certificate's SHA-1) makes the install step sign the shipped program with hardened runtime and secure timestamp: macdeployqt's -sign-for-notarization for the viewer bundle, a codesign INSTALL(CODE) for rugnux. tools/notarize_macos.sh then submits the cpack artefact to Apple, waits for the verdict and staples the ticket to the .dmg; the rugnux binary inside the .tgz is submitted as a zip, as a bare executable cannot carry a ticket. Both macOS CI jobs resolve the identity from the runner's keychain, check for the hardened runtime and notarize on every push, so a rejection shows up with the change that caused it. The identity is passed as the hash, not the name: Qt's generated deploy script writes DEPLOY_TOOL_OPTIONS unquoted, so a name with spaces splits into several macdeployqt arguments. The notarytool profile has to live in the login keychain file (store-credentials --keychain): the default data-protection keychain becomes unreadable once the screen locks, which an unattended runner's does. Verified end to end on the runner: both artefacts accepted by Apple, the stapled image and a quarantined copy of the app and of rugnux pass Gatekeeper as Notarized Developer ID, signing and the credential lookup work from a launchd-started process with the console locked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-14 15:01:30 +02:00
leonarski_fandClaude Opus 5.5	b574a0e600	docs: the min-pix 1 search gate is de novo only ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 14:59:04 +02:00
leonarski_f	cb044459ba	Merge commit '86f17740f' into stillsidx	2026-10-14 13:41:39 +02:00
leonarski_fandClaude Opus 5.5	86f17740f6	ACKNOWLEDGEMENT: Markstein reference without the working note ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 13:15:22 +02:00
leonarski_fandClaude Opus 5.5	51397c15f4	Merge FFT indexer GPU peak search rewrite into rc175: ~45% faster per call, output unchanged ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 13:15:17 +02:00
leonarski_fandClaude Opus 5.5	804206155b	PROTOTYPE (owner decision): stills min-pix choice searches the lattice once per frame, ~2x faster ... Changes results - not for merging without the owner's approval. The per-image min-pix scheme indexed every frame at min-pix 3, 2 and 1, each with the 30/80/all seed escalation, then indexed the winner again: 3.9 indexer calls per frame. Measured per frame on a serial lysozyme run (2000 frames, de novo and against the known cell): - where several lists index, the searches return the same lattice (97-98% of pairs); - the gain over a fixed min-pix 2 has two sources: (A) frames min-pix 2 indexes but whose min-pix 1/3 list indexes ~26 more spots (a better per-frame refinement), and (B) frames with almost no spot of two pixels (min-pix 3/2 lists of 1-3 spots, ~50 single pixels at min-pix 1) on which the min-pix 1 search fits 8-9 spots. De novo, 189 of 194 such B lattices had cells unrelated to the crystal's; against the known cell they index, but leaving them out raises I/sigma and lowers R_pim. Now: the lattice is searched for on the min-pix 2 list (min-pix 3 where that does not index); the first list that indexes hands its indexer answer to the others, which are only refined and scored (FrameIndexerCalls::lattice_found). The min-pix 1 list is searched on its own only where the min-pix 2 list holds at least ViableCellMinSpots spots. Ties keep the larger min-pix, as before. Measured on two full 24960-frame serial runs (RTX 5080, box loaded by other agents, load 9-20; loop rate is frames/s of the image loop; I/sig, R_pim, observations of the P41212 merge): known cell (ffbidx, -C): base 785 / 750 -> 1130 / 1017 frames/s (warm 6000-frame ABAB 860 -> 1453) run A: obs 5.37M -> 5.30M, I/sig 13.09 -> 13.02, R_pim 6.48 -> 6.46%, CC1/2 .970 -> .973 run B: obs 4.91M -> 4.89M, I/sig 12.00 -> 11.95, R_pim 6.50 -> 6.47%, CC1/2 .971 -> .970 de novo (FFT): base 89 / 91 -> 174 / 195 frames/s run A: obs 3.96M -> 3.91M, I/sig 10.18 -> 10.12, R_pim 6.94 -> 6.98%, CC1/2 .960 -> .967 run B: obs 3.91M -> 3.89M, I/sig 10.53 -> 10.52, R_pim 7.01 -> 7.03%, CC1/2 .965 -> .961 fixed min-pix 2, for scale: ffbidx 1496 / 1316 frames/s, obs 4.94M / 4.54M, I/sig 12.62 / 11.77. The indexing rate drops (ffbidx 59.5 -> 56.9%, 55.2 -> 52.4%; FFT 60.5 -> 52.1%, 60.2 -> 47.9%): the frames that go are the spurious B lattices; observations stay within 1.3%. Indexer calls per frame 3.89 -> 1.67 (de novo, 2000 frames); the min-pix 1 search is skipped on 58% of frames. De novo, 23 of 691 frames whose lattice was the crystal's now take a superlattice (13, same spots indexed) or a cell fitted to 6-12 spots of a weak frame (10) from the min-pix 2 search. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 10:34:04 +02:00
leonarski_fandClaude Opus 5.5	ab645847ed	FFT indexer GPU peak search ~45% faster per call, output unchanged ... calculate_fft_result (the per-direction prominence peak search) was 61-65% of the GPU kernel time of a de-novo stills run, and the card is what limits that loop. It is one dependent chain per direction, latency-bound. Two exact changes, taken where the spectrum holds a whole background window (any FFT max cell above ~30 A; the general loop stays for the rest): - the sliding window written out as its three phases (clamped at the start, centred, clamped at the end): the same additions and subtractions in the same order, so the running double sum is bit for bit the same, but the loads no longer sit behind data-dependent while loops and are issued ahead; - the window mean divides by a constant width, so the double division (a long FP64 iteration on a consumer card) is replaced by the reciprocal multiply plus one correction by the exact remainder, which returns the correctly rounded quotient (Markstein). Checked on the CPU against the division for 4e8 random dividends over all odd widths 3-401: 0 differences. Measured (RTX 5080, box loaded by other builds, load 12-25): kernel minimum 1.50 -> 0.83 ms, mean under the 8-stream pool 4.7 -> 2.8 ms; de-novo stills 2000 frames ABAB 113/120 -> 127/138 frames/s. p.hkl byte-identical (2000 and 3000 frames, base and new min-pix schemes). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 10:32:03 +02:00
leonarski_fandClaude Opus 5.5	593590b357	rugnux -X FFT is the algorithm: cuFFT on a GPU, FFTW without one; -X cuFFT names the GPU implementation ... In rugnux people choose the algorithm, not where it runs. -X fft (and the old -F) now resolves at parse time to the GPU FFT indexer where a GPU is visible and --no-gpu is not given, and to FFTW otherwise. -X cufft is the GPU implementation alone, as -X fft was. -X cufft and -X ffbidx without a GPU now stop at once with a clear message instead of failing later in the indexer pool. Auto is unchanged and not redundant: on stills with a known cell and a GPU it takes FFBIDX. Only rugnux's CLI spelling changes. The internal IndexingAlgorithmEnum, the broker and its OpenAPI enum keep FFT = the GPU implementation; the CLI maps onto it, and the command line the viewer writes for a run (RugnuxCommandLine) gives the enum's FFT back as -X cufft, which re-parses to the same thing. A run without -X, and -X fft on a machine with a GPU, process exactly as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 10:13:23 +02:00
leonarski_fandClaude Opus 5.5	bfd715b64d	rugnux --no-gpu: run on the CPU only, without touching CUDA, on a machine with a GPU ... hide_gpus() in CUDAWrapper makes get_gpu_count() answer 0 without calling cudaGetDeviceCount, so every GPU decision in the process (indexer Auto -> FFTW, preprocessing/spot finding/decode, pre-scan, scaling engine, model validation, MR) takes the path it takes on a machine without a GPU, and no CUDA context is created. rugnux reads --no-gpu in main, ahead of the parser, because set_gpu_blocking_sync() already asks CUDA for the devices. The banner says 'CPU only (--no-gpu)'. The 'run it on the CPU' advice in the out-of-GPU-memory messages and in the install docs now names --no-gpu instead of CUDA_VISIBLE_DEVICES=. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-14 10:13:23 +02:00
leonarski_fandClaude Opus 5.5	6e9b891eb4	MR score-engine cache and merge buffers: no hang, no doubled wait; near-tie doc follows offline decide-late ... - MolecularReplacement: an engine whose making throws anything (std::bad_alloc, a gemmi error), not only a JFJochException, is refused and its reservation given back, so a thread waiting for the same engine is woken instead of waiting forever. - RotationScaleMergeGPU::Alloc waits for the GPU work beside itself (GPU_BUSY_TIMEOUT), so its buffers are allocated plainly Synchronous; SynchronousAfterWorkBeside added a second 10-minute wait in front of it. - CPU_DATA_ANALYSIS_INDEXING: an offline run decides its near ties on the whole sweep from the start; only a live run (broker URL or --simulate-live) decides early and re-checks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-13 21:25:15 +02:00
leonarski_fandClaude Opus 5.5	7d64a62872	Merge branch 'plots-disposition' into rc175: <prefix>_plot.txt gains a per-frame disposition column (1 merged, 0.5 downgraded, 0 rejected) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-13 15:44:34 +02:00
leonarski_fandClaude Opus 5.5	5901c75547	rugnux: _plot.txt gets a graded per-frame disposition column ... A new last column, "disposition", gives each frame's merge disposition as a number to plot directly: 1 merged, 0.5 downgraded, 0 rejected; nan for a frame that never got a scale (never reached the merge) and on a run that merged nothing, as the file writes every value nothing measured. Existing columns keep their indices. 0.5 is a marker, not a weight. "Downgraded" is a label applied after the merge (RotationScaleMerge::CountSweepDisposition marks the merged frames inside a flagged sweep-quality range); the merge applies no factor of its own to them - they are carried at whatever weight their own scale and sigmas give - so there is no fixed or per-frame reduction factor that could honestly be written instead. Writer-only change: on three open-arm sets with downgraded and rejected frames p.hkl is identical to the same commit without it, the first ten columns are unchanged, and the column's 1/0.5/0 counts equal the report's merged/downgraded/rejected counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 15:44:04 +02:00
leonarski_fandClaude Opus 5.5	53c81dc5df	CHANGELOG: rc.175 gains a jfjoch_writer line for the writer-end-catch work ... Bounded memory and efficient closing of old data files on long runs, a failure ending the run in an error state the broker hears about, and the pusher's log line when a stalled writer makes it drop images; the master-file ordering moves here from the Fixes line. Owner-requested edit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 13:30:21 +02:00
leonarski_f	7b4dcce480	Merge rc175 into review-fixes-rc175	2026-10-13 13:30:00 +02:00
leonarski_fandClaude Fable 5.1	1be6093429	Writer: received images live in a byte-bounded ring buffer until written and republished ... Every stage between the socket and the HDF5 write was bounded by a message count and never by bytes: ZeroMQ receive watermark 100, cbor_fifo 200, outside_fifo 1000, repub_fifo 200 - up to some 1500 images in flight per writer process, each a whole compressed image, so 6-12 GB for a JUNGFRAU 4M and 30-60 GB for a 9M, and the writer's memory was a quantity nobody had chosen. ReceiveRingBuffer is a fixed ring of bytes owned by the ImagePuller base class, so the ZeroMQ and TCP pullers share it. A message is received straight into it (TCP reads the payload into the ring; ZeroMQ copies the message in) and travels as a shared_ptr region in ImagePullerOutput; the deserialised image points into that region, republish sends it as is, and the bytes come back when the last holder lets go. Space is reclaimed in arrival order: a region released out of turn - republish finishes with messages in its own order - is marked and waits for the ones in front of it. Because of that, what republish may hold is capped at a quarter of the ring, and an image over the cap is not republished, exactly as one that finds the republish queue full; otherwise a stalled republish consumer would pin the ring and slow the writer to its rate. A message larger than the whole ring (a START or END of a very long run) goes on the heap, so it cannot wedge the receiver. A full ring stalls the receive thread, and what that does depends on the stream. On TCP the pusher waits behind the writer's BUSY heartbeats and gives up only after a minute in which no image at all was written. On ZeroMQ the pusher's send blocks for one second, then drops the image and sends without waiting for the rest of the run - so on ZeroMQ a ring that is smaller than the writer's stalls costs images, where the old 1500-message queues absorbed them. That is the trade: the budget is now a number, and it has to be chosen. jfjoch_writer sets it with -m/--rcv_ring_mb, default 2048 MB per writer process; the ZeroMQ receive watermark default goes from 100 to 10 messages, since it now only counts what sits beyond the ring. The ring's peak use is in the end-of-run log line, and docs/JFJOCH_WRITER.md has a Memory section with the sizing rule. The puller's own default is no ring at all, which keeps every other user as it was: the broker's DECTRIS stream2 puller (JFJochServices, JFJochReceiverService) stays zero-copy with the ZeroMQ message as the buffer, and the broker's direct HDF5 writer (HDF5FilePusher -> FileWriter) never goes through a puller. The puller threads now also catch std::exception, as the writer does since the previous commit: deserialising the END message of a million-image run allocates a vector per per-image array, and a bad_alloc leaving the thread ended the process. Tests: ReceiveRingBufferTest (reuse, wrap-around, full ring, out-of-order release, heap fallback, zero size, stop, a region outliving its ring); ZMQImageCommTest_SmallRingBuffer (64 one-megabyte images through a three-image ring, nothing lost, peak within the bound); one of the two writers in TCPImageCommTest_2Writers_WithAck runs on a three-image ring. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-13 12:51:34 +02:00
leonarski_fandClaude Opus 5.5	8b43cce9eb	docs: Models and reference data - what a model or reference can and cannot change ... A new page, RUGNUX_MODEL_AND_REFERENCE.md, beside the other rugnux pages in the toctree and linked from RUGNUX.md and the --model section of RUGNUX_ADVANCED.md. It lists every option that brings in a model or reference data (--model, --mr, the PAE file beside a prediction, -z and --reference-column, -S, -C, --force-rotation-lattice, --report-resolution), sorts every decision into data-only, model-informed label in <prefix>.mtz (hand past the anomalous veto, indexing ambiguity, setting), model-driven separate files (<prefix>_mr) and report-only statistics, walks a --model run step by step, says exactly where a model can touch the space group and what guards it (the null, the anomalous veto, the indexing margin, the MR rules, the wrong-model battery step), and describes the free set and model bias (a deposited model's R-free on rugnux's own test set is not cross-validated; -z with the PDB's structure factors inherits the depositor's set). Includes the one pre-merge use of a model: serial stills with -C/-S, where it breaks the per-image indexing ambiguity. Documentation only; behaviour as merged in rc175 (7ed970503). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 09:05:18 +02:00
leonarski_fandClaude Opus 5.5	402b3d1293	ACKNOWLEDGEMENT: autoPROC and Clemens Vonrhein; DOIs verified; DISTL; tNCS credit in MR ... - All 84 DOIs checked (82 on Crossref, 2 DataCite): title, journal, volume, pages and first author match; Matthews 1968 and Yeates 1997 links percent-encode their parentheses; the MOSFLM link points at a page that answers. - autoPROC: an explicit acknowledgement of Clemens Vonrhein (Global Phasing) for the principle that a pipeline should verify the metadata it is given - beam centre, geometry, goniometer - and correct it where it is wrong; Vonrhein et al. (2011) Acta Cryst. D67, 293-302 (DOI checked). - Zhang, Sauter et al. (2006) DISTL, already credited in SpotFindingSettings.h, gets its entry. - Molecular replacement: tNCS looked for in the native Patterson as Read, Adams & McCoy do; the same credit at DetectTranslationalNcs (comment only). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 09:03:32 +02:00
leonarski_fandClaude Opus 5.5	1bd251cdd1	docs: GPU_DATA_ANALYSIS, every GPU engine of the crystallographic pipeline ... One page listing the CUDA engines by pipeline stage - decode and preprocessing, per-image analysis (azimuthal and ROI integration, spot finders, extractor), pre-scan accumulators, indexing, prediction and integration, the scaling engine, model validation and molecular replacement - each with what it computes, how closely it agrees with the CPU, and a link to the section describing the method. Added to the toctree after CPU_DATA_ANALYSIS_DECISIONS. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 09:03:31 +02:00
leonarski_fandClaude Opus 5.5	30cf5856d4	docs: bring the rugnux and analysis pages up to the code (MR fallback, GPU decode, live runs) ... - CPU_DATA_ANALYSIS_IMAGE §0: CBF byte-offset and marCCD widening are decoded on the card; the host decoders are the CPU build's path. - CPU_DATA_ANALYSIS_DECISIONS §14: the null runs for every model the rigid body is tried on; the device choice is made once on the card's total (structure factors and rigid body a quarter, scale fits an eighth) and a CUDA failure ends the validation, deciding nothing; a subgroup the model fits is written as <prefix>_mr; new §14.10 on molecular replacement (model kind, rotation and translation functions, likelihood, Matthews, tNCS and P1 rules, null, what it may change). - CPU_DATA_ANALYSIS: scope and references gain molecular replacement. - CPU_DATA_ANALYSIS_INDEXING: the near tie decided early and re-checked once every frame is in. - RUGNUX: merging is the default and _process.h5 optional; live following of a collection with a broker URL; --model's MR fallback. - RUGNUX_INSTALL, RUGNUX_OVERVIEW, RUGNUX_TUTORIAL, RUGNUX_REPORT: GPU work list, <prefix>_mr outputs, MR in --mode mx for rotation and stills, rigid body only in --mode scale. - RUGNUX_ADVANCED: -N default as the usage states it. - battery README: the ci tier has 28 sets. - ModelValidation.cpp: a comment still said a GPU failure reran the validation on the CPU (comment only). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 09:03:31 +02:00
leonarski_fandClaude Opus 5.5	5e916fe120	docs: --model places by rigid body, then molecular replacement; --mr; <prefix>_mr files ... RUGNUX_ADVANCED's --model page described the rigid body alone. It now says how the unified flow places a model: an experimental model by the rigid body, checked against its null in the answer's or the twin hypothesis' merge, and by molecular replacement where it is not placed; a prediction straight to molecular replacement; --mr skipping the rigid body; MODEL_PLACEMENT. It says what a placed model may change in <prefix>.mtz (the hand, and from the rigid body an indexing-ambiguity reindex) and that another group or the twin subgroup the model chooses goes to <prefix>_mr.mtz/.cif with <prefix>_mr_model.pdb/.cif. --mode scale places by the rigid body only (no molecular replacement, --mr refused); a full run does molecular replacement on serial stills too (checked: a stills run with a predicted model reports MODEL_PLACEMENT= MR). The options table gains --mr and the --model row the fallback. RUGNUX_INTEGRATION no longer says rugnux does no molecular replacement, and ACKNOWLEDGEMENT names --model beside --mr for predicted models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 08:07:44 +02:00
leonarski_fandClaude Opus 5.5	645d1cc1bb	CHANGELOG: rc.175 says --model falls back to molecular replacement and takes predicted models; viewer refresh ... The --mr line now says what the unified --model flow does for a user: a predicted model is accepted, and a model the rigid body does not place goes to molecular replacement, with --mr forcing it. One viewer line for the Files panel refresh (F5 and the poll) and the combo boxes that no longer reset during live collection. Owner-requested edit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 07:41:37 +02:00
leonarski_fandClaude Opus 5.5	d0d225cb99	rugnux --model: molecular replacement where the rigid body does not place the model ... One --model now does the right thing by itself: - A prediction (AlphaFold/ModelCIF, read from the file) goes straight to molecular replacement. - An experimental structure is placed by the rigid body as before. The null now runs on every --model run (no NOT_TESTED): "placed" = the fit beats its null (4.51 sd) in the answer's merge or in the twin hypothesis' merge. Placed: behaviour unchanged (hand and reindex may go into p.mtz); a subgroup the model fits and prefers is written as <prefix>_mr. - Not placed (or no crystal cell): molecular replacement with the experimental-model path. It may relabel the hand into p.mtz past the anomalous veto, never reindexes; another group or the twin subgroup goes only to <prefix>_mr. Maps, R-free and CC come from validating the MR placement with every decision switched off; MODEL_FIT stays the rigid body's verdict. - The twin hypothesis' merge is made in memory with a model where a metric re-ask promoted the group or the model's group has fewer rotations than the adopted one; written and reported only where the model chooses it. - --mr is kept as "--model, searched for by MR straight away". - Report: MODEL_PLACEMENT= RIGID_BODY|MR|NOT_PLACED, MODEL_DECISIONS_TAKEN gains MR_ENANTIOMORPH, MODEL_CHOICE_FILE_PREFIX/SPACE_GROUP; the MR keys move into 5. MODEL VALIDATION; MODEL_FIT absent where no rigid body ran. MR NOT_PLACED is never a warning. - Battery wrong-model step: check (c) also fails a wrong model placed by MR. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 22:20:36 +02:00
leonarski_fandClaude Opus 5.5	f567771d56	CHANGELOG: rc.175 names --mr, which takes predicted and experimental models ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 20:08:50 +02:00
leonarski_fandClaude Opus 5.5	30c7bee15f	rugnux: --af-model becomes --mr, for predictions and experimental structures alike ... - The option is --mr <file>. What kind of model it is is read from the file: a prediction where it is ModelCIF (_ma_qa_metric) or has no cell and one B value per residue in 0-100 (AlphaFold PDB/mmCIF); an experimental structure otherwise. A prediction is handled as before (pLDDT >= 70 trim, B from pLDDT, PAE domains where the AlphaFold DB's PAE file sits beside it). An experimental structure is taken as its polymer - no waters, ligands, hydrogens or alternative conformers, occupancy 1, isotropic B as refined - with a coordinate error of 0.67 A (a prediction's at pLDDT 90) for sigma_A; a B column above 100 no longer stops the search. - The report section is MOLECULAR REPLACEMENT (EXPERIMENTAL), its keys MR_* (MR, MR_MODEL_FILE, MR_MODEL_KIND, MR_DECISION, MR_SPACE_GROUP, MR_COPIES, MR_HAND_CHECK, MR_SECONDS, ...); the <prefix>_mr outputs are unchanged. - The twin hypothesis' merge that is made only for the model to test (a metric re-ask promoted the indexed lattice's group) is written and reported only where the model chooses the subgroup. Raw deposited coordinates of nine control sets, unconverted: every one placed and confirmed against the deposition (all chains as one body), the groups the same as with the prediction or the prepared model. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 19:30:22 +02:00
leonarski_fandClaude Opus 5.5	6ee167df9a	Merge rc175 (the null turns each copy about its own centroid) into mr-7qij ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 16:49:20 +02:00
leonarski_fandClaude Opus 5.5	c6912e2a06	rugnux --af-model: tNCS half-vector moves checked, the merge's own group not ruled out beside tNCS, P1's first copy judged by its orientation (WIP, validation pending) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 16:49:02 +02:00
leonarski_fandClaude Opus 5.5	aabee7423f	CHANGELOG: rc.175 release notes ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 14:10:51 +02:00
leonarski_fandClaude Opus 5.5	0d33d7a40a	rugnux --af-model: likelihood judging, PAE domains, twin comparison, tNCS carry test, faster multi-copy search ... - Likelihood (Read 2001; Z-score over random positions as McCoy et al. 2005): log-likelihood gain of Rice/Woolfson E distributions, sigma_A from the pLDDT coordinate error with its scale refined, to 2.5 A (or the 20000 lowest-resolution reflections). Further-copy TF peaks ranked by the gain on the search's reflections (peaks that lower it dropped); a copy whose TF Z is below 8 also stands out where its refined likelihood Z-score reaches 8 - only where the copies placed are as many as Matthews makes most likely. The likelihood is computed only where a translation function does not stand out. Copy structure factors exact from one FFT of the cell's grid, or read off the molecular transform where it reaches the likelihood's resolution. - PAE domains: the AlphaFold DB's predicted_aligned_error JSON beside the model (same name pattern) is read automatically; average linkage on the symmetrised PAE (10 A, domains >= 40 residues). Where the whole model is placed nowhere, each domain is searched by itself side by side; placed domains must agree, and a domain alone may only confirm the answer's group. - Twin comparison: where the answer's merge chose and the twin hypothesis' merge exists, the chosen placement read as a crystal in the subgroup (its symmetry images as copies, the subgroup's setting found from the two cells) against the model searched for afresh in the subgroup, both against the data twinned by the law (fraction refined on the working set, Yeates 1997); the subgroup is taken by a 0.03 R-free lead plus its own null/margin -> TWIN_SUBGROUP, written to p_mr with law and fraction. With --af-model rugnux makes the subgroup merge wherever a metric re-ask promoted the indexed group. - tNCS rule: a group with fewer copies beside tNCS (screws bridged by the vector, or the enantiomorph) no longer leaves the choice undecided where the chosen copies, carried into it and refined, fit worse by 0.03 in R-free. - Speed: the molecular transform's FFT on the GPU (cuFFT, serialised), no rigid body for trials that lower the likelihood, re-search only with tNCS, PAE domains searched in parallel. Survey (193 sets, MR replay on stored merges): single GREEN 58 -> 68, multi 48 -> 58, partial 8 -> 9; 4bwl (and, in full runs, 2wnq and a withdrawn entry of the same protein) TWIN_SUBGROUP P21 instead of a wrong AGREES; a P21212 set with tNCS now SPACE_GROUP P21212 instead of UNDECIDED; no wrong call. CPU and GPU builds make the same decisions on 18 demo/multi-copy/twin sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 13:21:51 +02:00
leonarski_fandClaude Opus 5.5	fcbe92042b	Merge branch 'af-mr3' into rc175: --af-model searches several copies per asymmetric unit (Matthews probability, sequential translation function, fixed-copy rigid body, tNCS) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 01:36:26 +02:00
leonarski_fandClaude Opus 5.5	63e0a85467	rugnux --af-model: several copies in the asymmetric unit, tNCS, a search-matched null, faster ... Copy number. The hard "Vm >= 4: MULTI_COPY, no decision" rule is gone. Copies are searched one after another up to the largest copy number with a Matthews probability of 1% (a Gumbel distribution with Kantardjieff & Rupp's 2003 mode and mean, no resolution dependence); COPIES_LIKELY reports the most probable. Sequential search. Copy n's translation function carries copies 1..n-1 as a known partial structure: the cross term 2 Re(conj(F_known) F(t)) goes on the same FFT grid (CPU FFTW and GPU cuFFT; KnownStructureFactors, AssemblyCorrelation). Its peaks, past those that clash, are ranked by the correlation of the whole assembly with E^2 (the TF itself, linear in |F|^2, rewards copying a placed copy onto an origin shift), the first that stands out (TF Z >= 8), packs and lowers R-free by 0.01 is kept; each copy is refined beside the others held fixed (RefineRigidBody's fixed copies), and the first copy again beside the second, the earlier ones once more at the end. Packing: ClashFraction counts C-alpha clashes with other copies and all their mates, binned (7mzt: 51 s of clash checks -> 1.9 s). Search quality: rotation-function peaks hill-climbed off the 6 deg sampling (6ze4: RF Z 10.4 -> 16.0, 3.3 deg off the deposited orientation -> 0.2); the first copy goes down the TF list until three placements that pack are refined, so a group is CLASH only where no offered orientation packs; groups are searched side by side, and every group the best one leads is searched again with the best one's copy orientations tried first, the better search kept. tNCS: the native Patterson of E^2 (5-10 A) off the lattice; placed copies' orientations join every further copy's search; and where the tNCS vector turns one group's screw axes into another's (TncsRelatesGroups), a group whose search stopped at fewer copies does not lose to the other - UNDECIDED. Measured on an in-house tNCS crystal (vector c/3 along the 3-fold): both hands fit with three copies to within 0.01 in R-free, and where one hand's search failed the other was taken as ENANTIOMORPH; now UNDECIDED. Null: the replicates are placed by the translation function in their own orientation, as the search placed the real one (a null at the real centroid reads high), and the gate is 4.51 = t(8) sqrt(1 + 1/9), 3 sd of a nine-draw null, the gate --model's MODEL_FIT is now read at. Speed: placements scored on the GPU structure-factor engines; further copies' trials score with the first copy's bulk solvent held, the assembly refitted at the end. Survey of 193 sets (jfjoch_mr_replay on stored merges, RTX 5080 shared with other jobs): MR median 1.15 s, max 24 s; demos 0.8-1.1 s. Report: COPIES and COPIES_LIKELY columns, MODEL_PRIOR_COPIES. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 00:27:17 +02:00
leonarski_fandClaude Opus 5.5	ee350f1f74	rugnux --model: a wrong model confirms no shell, fits no hand, and fits the GPU reserve ... Three defects the battery's wrong-model null found (5 sets x 5 deposited models of unrelated proteins): CC(model, data) significance. The per-shell sigma was Fisher's transform on independent reflections, but the data and a scaled model share the falloff across a shell, the anisotropic B fitted to these very data, the bulk solvent and the symmetry enhancement epsilon - whatever the model is. 21 of 24 wrong models read CC_MODEL_CONFIRMED_TO_D_MIN to the data's d_min (10-15 sigma in most shells). The correlation is now formed on I/epsilon and |F_model|^2/epsilon, and its null is empirical: the data at h against the model at h + d for the 26 nearest lattice offsets, which keeps everything smooth in reciprocal space and loses what the model knows of h. SIGMA = (z - mean) / max(sd, 1/sqrt(N-3)) in Fisher z over those 26, a new CC_NULL column shows the null, and a shell confirms only beside a coarser shell that also clears 3 sigma (one shell alone did so once in 216 wrong-model shells). Prototyped on the p_maps.mtz of a full open-arm run and the null runs: wrong models confirm 0/24 (was 21/24), the deposited models confirm 197/200 (was 198/200 in the prototype's reading; the one lost has CC_MODEL 0.05-0.13 in every shell). The report no longer says a model cannot agree by accident. MODEL_FIT gate. Accepted at 3.0 sample sds of a nine-draw null, as if the sd were known; an unrelated P1 helical model on a P3(1) helical crystal was ACCEPTED at +3.05. Against 48 draws it reads +2.09 and the best random placement fits as well as it does, and drawing the null about the cell axes instead changes nothing (+2.15): the cause is the nine-draw sd, not the orientation. The gate is now 3 sigma allowing for that: t(8) * sqrt(1 + 1/9) = 4.51. The deposited models of the open arm read +7.0 and up. The indexing lead stays at 3 sigma (INDEXING_MARGIN_SIGMA): it is read only behind the fit gate, and a deposited model settles one at +4.15. Rigid-body GPU reserve. The finest zone's observations were counted by the data's d, but the zones count them by the d of the index in the model's ASU, and a model's group need not keep the data's metric (an I2(1)3 model on I222 data: 2320 in the zone against 1712 reserved), so the validation died with "zone over the engine's reserve". Counted the zones' way, under every indexing probed. Tests: ModelValidation_CCModelUnrelatedModelConfirmsNothing (Fisher alone reads +4.7 on an unrelated model with an anisotropic falloff; the new sigma stays under 1), ModelValidation_WrongModelInAGroupOffTheDataMetric (fails with the old count), ResultReport_ModelValidationSection (CC_NULL column, lone-shell rule). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 21:30:34 +02:00
leonarski_fandClaude Opus 5.5	50e3b59fb7	rugnux --af-model: hand check reads only the anomalous map, on MR's placement and solvent ... The ENANTIOMORPH hand check ran a whole ValidateAgainstModel on the placed model: model read and frame probe, a rigid body, a 9-replicate null (~0.5 s of the ~0.6-0.9 s), sigma_A, three maps written. The decision needs only --model's rule: the model fits its null (the MR choice has already required its own null of the same construction) and the anomalous map at the model's atoms is not inverted. ReadModelAnomalousMap (ModelValidation) computes F_calc and F_mask to the data's resolution (GPU engine where there is one), carries them through the bulk solvent MR scored the placement with (the overall scale and anisotropic B do not move phases), and reads the anomalous map with the same code as the validation: its coefficient construction and its reading at the atoms are factored out of Validate (anomalous_coefficients, read_anomalous_map) and used by both, so plain --model is unchanged. Hand check 0.6/0.6/0.9 s -> 0.027/0.028/0.040 s on 11if/7ris/9zmu; same ADOPTED/VETOED on all seven cases tried (the three demos, two anomalous lysozyme sets and their mirrored-detector runs), with the scatterer means and deepest troughs within 0.1 sigma of the full validation's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 18:11:01 +02:00
leonarski_fandClaude Opus 5.5	2a3d60fb82	rugnux --af-model: anomalous veto on the enantiomorph, search on the GPU, cheaper rescoring ... - An MR ENANTIOMORPH decision is taken into the answer's label only as --model takes a model's enantiomorph: ValidateAgainstModel runs on the placed model (in a scratch directory, removed after) and its adopted_model_enantiomorph - fit over its null, anomalous map at the atoms not inverted - decides; otherwise ENANTIOMORPH_VETOED. The hand-check numbers go to the MODEL PRIOR section. - Rotation and translation functions on the GPU (MolecularReplacementGPU.cu): one block per orientation for the rotation function; per orientation the copies, the bin weights, a gather of the Crowther-Blow coefficients (CSR lists, fixed order, no atomics) and a cuFFT C2R, block-reduced statistics finished on the host in a fixed order. The CPU path (FFTW) is unchanged in what it computes; the orientation-independent part of the translation function (TranslationSetup) is now built once per group on either path. - Rescoring: a placement clashing with its mates over twice the limit is not refined, one rigid-body engine per group, and the null is built only for the candidate the choice falls on. Measured MR step on the RTX 5080 (before -> after, seconds; hand check extra where it runs): 3r6o 1.77 -> 0.67, neg 2.14 -> 0.37, 11if 1.20 -> 0.51 (+0.6 hand check), 7ris 1.46 -> 0.56 (+0.6), 9zmu 4.71 -> 1.07 (+0.9). Rotation 0.50 -> 0.02 s and translation 1.17 -> 0.13 s on 9zmu. Mirrored-detector runs of two anomalous lysozyme sets: the hand check vetoes (map inverted at the model's atoms); the unmirrored runs adopt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 17:10:48 +02:00
leonarski_fandClaude Opus 5.5	60a8bf3981	rugnux --af-model (experimental): single-copy molecular replacement of a predicted model ... A predicted model (AlphaFold mmCIF, pLDDT in the B column) is trimmed at pLDDT 70, given B-factors from the pLDDT (Oeffner et al. 2022) and placed by molecular replacement, one copy in the ASU: - rotation function on a uniform 6 deg sampling of the point group's rotation ASU, each orientation scored by the correlation of E^2 with the symmetry-summed |F|^2 read off the model's molecular transform, computed once by FFT and interpolated (Navaza 1994); - full-symmetry Crowther & Blow translation function (one FFT per orientation and group, through MapFromFPhi/FFTW) for the 50 best orientations in every group with the merge's rotations; - the best three placements per group through RefineRigidBody (GPU where there is one), bulk solvent and R-free to 3 A, a C-alpha packing check against symmetry mates, Matthews, and the random-orientation null for a placed candidate. Decision: PLACED = TF Z >= 8, clash <= 5%, Vm in [1.5, 4), null >= 3 sd. Within the answer's merge the placed group with the lowest R-free is chosen where it leads every other packing group by 0.03; the twin hypothesis' merge is searched, and chosen, only where no group of the answer's merge can hold the model at all (Matthews, packing). An enantiomorph relabels the answer as --model does; another group goes to <prefix>_mr.mtz/.cif plus <prefix>_mr_model.pdb; the MODEL PRIOR report section carries the table. The search runs on the CPU in both builds; without --af-model nothing changes. Measured (RTX 5080, MR step): 3r6o twin I41 chosen over I4122 (Matthews + clash) in 1.8 s, solution 0.09 A CA rmsd from Phaser's; 11if P41 -> P43 1.2 s; 7ris P3121 -> P3221 1.7 s; 9zmu P6122 -> P6522 4.7 s; 11if's model against 3r6o's data NOT_PLACED. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 13:35:05 +02:00
leonarski_fandClaude Opus 5.5	d7521bf1ea	TCP pusher: never block the receiver thread; queue bounded by image-buffer slots ... TCPStreamPusher::SendImage is called from the receiver's frame-transformation threads. It used a 128-deep per-connection queue and waited when it was full. With images routed per file ((n / images_per_file) % writers), every writer receives a whole file as a burst at the full frame rate (1000 images/file at 2 kHz fills 128 in ~64 ms), so the queue filled on every burst and all transformation threads ended up waiting on one writer. They then stopped returning FPGA frame buffers, and the FPGA dropped packets: partial images, images without a background estimate, data loss on every writer. The ZeroMQ pusher has an unbounded queue and never blocks, which is why it coped. - The per-connection queue is unbounded: every element owns a distinct image-buffer slot, so the bound is the image buffer's slot count. When slots run out the receiver drops only the write (GetImageSlot() == nullptr), after analysis. Absorbing one file burst therefore needs an image buffer of about images_per_file slots (~20 GB for a 9M at 1000 images/file); below that, images are still dropped, now as writer-queue-full instead of packet loss. - SendImage never blocks. A dead or wedged writer is still caught in SendAll (15 s silence / 60 s without progress), after which WriterThread returns the queued slots unsent. - The 200 ms fallback in StopDataCollectionThreads is gone: at END it closed the connection of any writer whose queue stayed full for 200 ms - a normal state for a slow but healthy writer - discarding up to 128 queued images and never sending END. END now follows the whole backlog. - A per-connection enqueue mutex makes "check active, enqueue" atomic against "clear active, enqueue the end sentinel": an image could land behind the sentinel when a connection was torn down mid-run, leaking its slot (CheckIfBufferReturned then threw) or being sent as stale data in the next run. New stress test SendImageRacingEnd_NoSlotLeak (verified to fail with the race widened and the lock removed). - TearDownConnection marks the connection broken first, so the destructor returns a backlog instead of sending it. Writer side: - jfjoch_writer -b/--rcv_buffer sets the TCP receive socket buffer (was hardcoded to the kernel default). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-10 21:11:09 +02:00
leonarski_fandClaude Opus 5.5	a897b08a7d	Merge branch 'doubled-cell' into rc175: a doubled-cell hypothesis for a half-translation pseudo-symmetry that breaks down with resolution ... Narrow, guarded: the doubled pass runs only where the added class grows with resolution and is resolved; the doubled cell is adopted only where one pseudo-translation explains the added class in every shell and the point group does not rise. Fixes the one remaining lattice failure (half cell -> deposited cell, R-free 0.574 -> 0.275); p.hkl identical on the ci tier, the supercell gate set, 78 further supercell-candidate sets and all private sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 15:18:14 +02:00
leonarski_fandClaude Opus 5.5	b99cd3b5c6	rugnux doubled cell: model validation back in the doubled pass; docs for the final rules ... The doubled pass now runs only on resolved, rising classes (two sets of ~134 with probe logs), so the twenty-minute model placement in a doubled cubic cell can no longer arise, and the adopted answer is validated again (9min: R-free 0.275 against the deposited model, 0.574 in the half cell). Final rules, measured on gate + ci tier + all sets with a de-modulated axial row + 49 sets with a baseline supercell occupancy >= 8% + the private arm: 9min fail -> pass (P212121, doubled cell); every other set run keeps its verdict and a bit-identical p.hkl. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 15:16:58 +02:00
leonarski_fandClaude Opus 5.5	7580256ac7	WIP: doubled-cell rise must end at a quarter of the lattice's intensity ... On strong data a class at one or two percent of the lattice rises significantly (0.009 to 0.059 at 4 standard errors on a ci-tier triclinic crystal, which was then adopted doubled and failed), and a half cell missing a few percent of its intensity still describes the crystal. The doubled deposition ends at 0.35-0.40, those classes at 0.06. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:14:10 +02:00