7 Commits

Include renames

Author	SHA1	Message	Date
jungfrauandClaude Opus 5	65eb86ab66	Size the post-refine gather by its reflections, not by its frames ... Both passes of the gather are chunked over OUTCOMES, one per frame, but their cost is the tens of thousands of reflections inside each. ThreadsForWork was being asked about the frames: a rotation dataset has on the order of a thousand of them against a floor of 32768 items per thread, so it answered 1, and the pass that walks every reflection in the run ran on a single thread on every dataset in the test set. RotationScaleMerge::Ingest does the identical thing and asks with the observation count, which is what makes this an inconsistency rather than a tuning choice. The reflection total is already to hand a few lines up. The split is unchanged - only the worker count moves - so the output is identical. Measured on the heaviest crystal: the phase drops 33 %, from 8.6 s to 5.5 s, and its mean occupancy goes from 4.1 threads to 12.3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 18:05:19 -04:00
jungfrauandClaude Opus 5	2308bbad8c	Stop scaling and merging what the resolution range excludes ... A crystal integrated to the detector corner but merged well short of it carries observations through the whole merge that the merge then discards. On the heaviest dataset in the rotation test set that is 63.3 M partials of which 6.4 M are ever used: the other nine tenths are sorted, uploaded, scaled, combined, error-modelled and post-refined before anything looks at their resolution. Ingest copied every one of them unconditionally, and the d_min limit was first applied far downstream, in the ASU grouping. They are now dropped at ingest, immediately after the one big sort: - WHOLE raw-hkl runs are dropped, on the same rawrun_d the ASU grouping already tests. A per-observation test is not equivalent - a run is in or out today by one member's d - and using a different rule here would put the two out of step. - The drop happens AFTER the flux meter, which takes each frame's mean background over every reflection on it, and after the sort, so neither changes. - The compaction runs in index order, so a frame's observations stay contiguous and keep their order, and every per-frame sum keeps its sequence of roundings. Post-refinement reads the integration outcomes rather than the merge arrays, so it still sees every reflection. That is the point: --integration-high-resolution buys the same time by never integrating the reflections, and pays for it in the per-frame geometry, which wants them. Two more passes over the observation array go with it. The incident-flux divide was 15.8 % of all user cycles to read one int and divide one float across 5 GB; the per-frame mean it needs is now accumulated by the ingest fill loop - one frame, one thread, same order, so bit-exact - and the divide rides on the finiteness pass that already touches that field. And the geometry post-refinement is fitted on a bounded sample of partials, selected by a hash of the raw hkl so whole rocking events are kept or dropped together and the sweep and the detector are thinned uniformly. The sample size is 8 M and the reason it is not smaller is measured. Over a 126x thinning the fitted rotation scale is flat to 2e-5 and the beam centre moves 0.03 px, but the CELL scale breaks between 8 M and 4 M: the axis step keeps the 20 000 strongest events, so once the pool approaches that size it starts fitting weaker ones and the second pass's cell shifts by ~0.1 %. Measured on the heaviest crystal, three A/B pairs with the order alternated: 68.4 s -> 37.3 s wall, 530 s -> 221 s of CPU. Whole battery 8m07s -> 7m15s, space group 21/24 with the same three disagreements as before, no failures. Bit-identical is not available on the GPU path - the resident reductions and the fulls emit order depend on array length - so what is shown is that every difference sits inside the spread the unmodified binary has against itself between two runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 18:04:55 -04:00
jungfrauandClaude Opus 5	088ba1cff8	Stop paying for workers and threads that do no work ... Four changes, all from measuring why this 48-core machine was SLOWER than a quarter of itself on a single job. The image loop now takes a worker count of its own. It is GPU-bound, every worker builds a private analysis engine of tens of megabytes of device and pinned memory - and every one of those allocations implicitly synchronises the device - so past a handful per card another worker adds setup and contention and no throughput. Measured at about 23 ms of pure setup per extra worker, which is why the penalty is WORSE on short runs: 200 images cost 0.86 s of loop at 12 workers and 2.25 s at 48. Capped at four per GPU, floor of eight. Every other phase still gets the full thread count, because each one starts its own workers. Ingest built its array with a serial push_back over every observation of every frame - 63 million of them on the largest crystal here. Each frame's block offset is known before anything is written, so the frames convert together, each still written by one thread in its own order. The pass that buckets observations by h was the single most expensive thing in a large run - 24% of all cycles, in five instructions. It strided an array of 80-byte observations to read one 4-byte field, and its store address depended on the loaded value, so the store buffer could not retire and the misses stopped overlapping. The sweep that already reads every observation now copies h out as it goes, and the bucketing walks that instead. And the post-refine passes took the raw thread count. One of them runs 134 times inside the rotation-scale fit, starting 48 threads each time to divide 390k terms among them; it is gated on the work now, like everything else. Measured on one crystal, N=48: 12.80 s -> 11.45 s, which is what 12 threads used to cost, and on the best-matched pair the two are now level. On the heaviest crystal ingest goes 13.5 s -> 9.9 s and the run 77.8 s -> 69.4 s. Battery 9m01s -> 8m24s, space group 21/24, no failures. Also restores get_gpu_numa_node() - the sysfs lookup deleted with NUMAHWPolicy - and an opt-in CPU pin to that node behind JFJOCH_PIN_CPU_TO_GPU_NODE. It is off because it measured neutral here: all four GPUs hang off two of the four nodes, so pinning to them costs a worker the other half of the machine. It is kept for boxes whose GPUs are spread over every socket. The old lookup had a latent bug - CUDA reports the PCI id with upper-case hex and sysfs paths are lower case, so on three of this machine's four GPUs it would have silently returned "unknown". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 10:53:47 -04:00
jungfrauandClaude Opus 5	f36349060a	Sort the partials and the post-refine events in buckets, in parallel ... Both were one std::sort on one thread over tens of millions of elements, and together they were a third of a crowded crystal's run. Bucketing by h first makes them parallel. h is the comparator's leading key, so the sorted array is exactly the buckets laid end to end, and each bucket sorts on its own thread. In Ingest the keys are built straight into their bucket slot, so this replaces the build pass rather than adding one and the packed-key array is never duplicated; the extra memory is a few hundred kilobytes of histograms. Buckets are taken largest first, because the tail of the phase is whichever bucket finishes last. The run split falls out of the same structure for free: a run of equal (h,k,l) never crosses an h boundary, so each bucket counts its own runs, a scan over the buckets gives the offsets, and the arrays are sized exactly - which also removes the repeated growth the push_backs were paying for. The h range comes from the finiteness pass, which already reads every observation. The partials order became total in an earlier commit, when the observation index was added as the last key. That is what makes this safe rather than merely fast: the permutation is uniquely determined, so a bucket sort produces the same one a single sort would. Measured on a crystal with 66 million partial observations: Ingest 15.2 s and 14.3 s -> 8.3 s and 7.4 s, the post-refine event sort out of the top ten gaps entirely, the whole crystal 2m22s -> 1m24s. Battery 15m32s -> 10m05s. Same space group on all 24 crystals, none failed, and no crystal's R_meas moved by more than 0.3 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 22:31:05 -04:00
jungfrauandClaude Opus 5	e11c2a2b20	Fit the goniometer rotation scale in closed form ... The fit has ONE parameter, and it was handed to Ceres as one residual block per rocking event - 8 million of them on a large crystal. Each block is a functor, an auto-diff cost function and a loss object on the heap, and the solver then factorises an 8-million-by-one Jacobian on every iteration. It cost 13.7 s. The residual is closed-form in k. A rotation preserves length, so |p_lab| is |e_mid| whatever k is and only the z component moves; Rodrigues gives it exactly: r(k) = C + A cos(a k) - B sin(a k) = C + R cos(a k + psi) C = lambda |e|^2 / 2 + u_z (u.e), A = e_z - u_z (u.e), B = (u x e)_z with a the event's angle from the sweep centre. That is the same function the functor computes - Ceres uses the exact Rodrigues form here, so there is no small-angle branch to disagree with - and it reduces the fit to minimising a smooth function of one variable over the interval the solver was bounded to. It is scanned on a grid and then closed in by golden section; the objective's curvature jumps wherever an event crosses the Huber knee, which is why this is not a Newton iteration. The coefficients are computed in double and stored narrowed. Their rounding moves the minimiser by ~1e-10, and k is carried downstream as a float, so the committed value is the same to far more digits than anything reads. One pass over the events yields the five per-fifth partial sums, so the all-data fit and the five leave-a-fifth-out folds share it. That matters because the jackknife only runs when the fit is big enough to act on, and on a crystal that trips it the old code paid for six full solves. The partials gather ahead of it counted first and then filled instead of growing one vector by push_back tens of millions of times, which copied the whole thing on every doubling. Measured: unchanged verdict and k to five decimals on the regression crystals. Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 13m35s together with the scale/merge changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 18:56:09 -04:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00