f09fe4e2c50d3199ccfc900d68253358bb06558d
29
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5ee0f22a61 |
Build the detector's lookup tables once, not once per worker
The image loop gives every worker its own analysis engine, so a run builds ninety-six of them. Each one derived, from scratch, tables that are the same in all of them: the byte-per-pixel mask, the resolution mask, the radial kernel, and the checksum that names the shared device tables. The checksum was the worst of it, because it is part of the cache KEY and so is computed before the lookup - a hit still hashed the whole table. On a 16 Mpx detector that is the bin table, the corrections and the mask, 126 MB an engine, about twelve gigabytes over a run, to answer a question whose answer had not changed. The header said it cost nothing measurable; a profile says otherwise, and says it is worst exactly during the ramp when the machine has nothing else to do. It cannot simply be remembered against the address, which is what it exists to catch: a buffer can be freed and another allocated where it was, and the cache would then hand back a device copy of something else. So the owner of the bytes computes it instead. The azimuthal mapping writes its two tables in its constructor and never again. The pixel mask re-derives its binary form and its checksum on every path that changes the mask, and all of those paths are now private to the class. The key therefore still describes the bytes as they are at the moment of the lookup. The resolution mask was two passes over every pixel - a float comparison into a vector<bool>, then a bit-by-bit repack - in each of the ninety-six. It is one pass now, writing the packed form directly, built once for the limits asked for and handed out as a shared pointer so a worker keeps the mask it was given. The radial kernel is cached on the six numbers it is derived from. Nothing computes a different value; only who computes it changes. Byte-identical merged output on a 16 Mpx set and on a small one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n8riB6X59oRjkrSHzNPAU |
||
|
|
16639e9de2 |
Add up the profile accumulators in an order the schedule cannot change
Every accumulator in this file that could be an integer already is one, and the spot finder's reduce_rings_shared says why: a preprocessed pixel is an exact int32, integer addition is associative, and a threshold that moves in its last bits between runs flips every pixel sitting on it. Four accumulators here were still floats, and they reach the intensity rather than a diagnostic. * The radial background curve. s_radv and rad_sum sum int32 pixel values, so int64 is not an approximation of the old sum, it IS the old sum - and the curve is subtracted from every reflection's background. * The learned profile grid and its second moments, which are sums of (px - bkg) / I over every strong reflection of the frame. There is no exact integer form, so these are fixed point at 2^20: a quantum of 1e-6 of one I-normalised pixel, far below the Poisson noise of the pixel it came from, and some five orders of headroom inside a signed 64-bit accumulator. * The normalisation total in build_profiles, which divides every cell of the profile - 128 lanes on one address, in arrival order. Now summed as integers, exactly, from the grid it normalises. * The fit's own reductions, s_num and s_den among them, which ARE the fitted intensity. These stay float, so fixed point would be a real precision trade over an unbounded range; instead each warp leaves its total in a slot of its own and every thread adds the slots up by warp index. WARP_ATOMIC_ADD is order-independent for the integer accumulators it was written for and not for these, which is what block_sum is for. With the prediction ordering of the previous commit, a run is now reproducible: the same command on the same images writes byte-identical .hkl and .mtz, at -N 1 and at -N 48, on a 16 Mpx rotation set and on a large-cell one. Before, all four differed. The battery is unchanged where it was ever stable: space group identical on all 24 crystals, reflection count on 20, R_meas on 22. The two that move are the two the battery has always seen move between runs of an unchanged binary - which is the point, since they stop moving now. Total 8m02s against 7m55s, inside the noise of per-crystal times quantised to a second. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n8riB6X59oRjkrSHzNPAU |
||
|
|
01b619cfd0 |
Take the double precision out of the box integrator's inner loops
boxsum summed its ring background in double and compared each ring pixel against a double threshold. The pixels are integers: a sum of at most a thousand int32 values is exact in a 64-bit integer AND exact in a double, so the two agree bit for bit, and comparing an integer against the floor of the threshold accepts exactly the same pixels as comparing it against the threshold itself. Both loops now do integer arithmetic. That was 39% of the card's double-precision pipe on the development machine and about three quarters of it on the production one, where the double rate is unchanged from Turing while the single rate has doubled - so this is worth more there than here. Alongside it, three things in the combine kernel. rr_nusable was computed by a whole extra walk over every observation and then never downloaded or read by anything. sum_wb and sum_cwb have no F in them, so they are the same in all three reweights and only the last round's values are ever used - two thirds of them were two divisions each, discarded. And CombineParams was the one parameter struct in the file without __restrict__, so the compiler could not assume the observation arrays and the freshly allocated fulls arrays were distinct. Measured on a crystal with 66 million partial observations: boxsum 12.2 s -> 8.3 s, the combine kernel 8.0 s -> 7.6 s, whole crystal 1m17s -> 1m12s. Battery 15m32s -> 9m59s. Same space group on all 24 crystals, none failed. Two things measured and NOT kept, recorded so they are not tried again: sorting the raw-hkl runs by length so a warp holds runs of similar length - it trades away the locality of neighbouring runs in the permutation and came out slower (7.6 s -> 8.8 s); and page-locking the integrator's host staging arrays individually - eleven separate registrations of small heap allocations overlap on shared pages and the driver refuses them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c1b85c7e88 |
Reduce within the warp before the Bragg integration atomics
fit and boxsum were 80% of GPU time on a crowded crystal - 78 s of it. Neither was bandwidth- or occupancy-bound: both sat at about an eighth of the issue rate the card can sustain, stalled. What stalls them is the block-wide accumulations. Every one has all 128 lanes of the block adding into one shared address, and a shared-memory atomicAdd on a float or a 64-bit integer has no instruction on either Turing or Ada - it compiles to a compare-and-swap retry loop. So those 128 lanes serialise into 128 retries, eighteen times per thread in fit. Summing across the warp first and letting one lane do the atomic leaves four per block instead of 128. That is the whole story: the arithmetic below was worth 2%, the atomics 5.6x. The arithmetic is still worth having, and is what was expected to matter: - compute_shell ran on all 128 threads of a block for a value that belongs to the reflection. It is two software double-precision divisions, on a card whose double throughput is a thirty-second (a sixty-fourth on the production one) of its single. One thread does it now. - The Kabsch inner loop divided by the same weight three times; the compiler emits the whole correctly-rounded sequence each time. One reciprocal now. Likewise the two Gaussian widths and the profile normalisation, which are constant over a reflection's cells and were divided per cell. - boxsum read the pixel before deciding whether it wanted it. The window is the bounding box of an ellipse, so nearly half of it is neither the signal disk nor the background ring, and those slots were fetching a cache line for nothing. Measured: fit 50.8 s -> 9.1 s, boxsum 27.5 s -> 12.1 s. A crowded crystal 2m22s -> 1m58s, a 16M-pixel one 39.5 s -> 37.2 s, the whole battery 12m30s -> 11m35s. Same space group on all 24 crystals, none failed. The integer sums are unchanged - addition is associative. The float ones move in their last bits and become more reproducible, since a fixed shuffle tree replaces whatever order the atomics arrived in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
538f3504d3 |
v1.0.0.rc-161 (#71)
Build Packages / build:windows:nocuda (push) Successful in 20m4s
Build Packages / Unit tests (push) Skipped
Build Packages / build:viewer-tgz:cpu (push) Successful in 16m5s
Build Packages / build:viewer-tgz:cuda (push) Successful in 17m26s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 27m46s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m17s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 26m13s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 23m17s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 28m11s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m30s
Build Packages / build:rpm (rocky8) (push) Successful in 24m34s
Build Packages / build:rpm (rocky9) (push) Successful in 21m30s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 23m33s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 20m18s
Build Packages / DIALS test (push) Successful in 18m23s
Build Packages / XDS test (durin plugin) (push) Successful in 11m30s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m2s
Build Packages / Generate python client (push) Successful in 49s
Build Packages / Build documentation (push) Successful in 1m21s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 29m45s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
67dca388bd |
v1.0.0-rc.160 (#70)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 18m44s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m11s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m54s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m40s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m10s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m23s
Build Packages / build:rpm (rocky8) (push) Successful in 11m30s
Build Packages / build:rpm (rocky9) (push) Successful in 12m51s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m8s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m21s
Build Packages / DIALS test (push) Successful in 13m22s
Build Packages / XDS test (durin plugin) (push) Successful in 9m2s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m55s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m57s
Build Packages / Generate python client (push) Successful in 23s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 10m24s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
451310f43d |
v1.0.0-rc.158 (#68)
Build Packages / Unit tests (push) Successful in 1h32m35s
Build Packages / build:windows:cuda (push) Successful in 18m0s
Build Packages / build:viewer-tgz:cpu (push) Successful in 7m37s
Build Packages / build:viewer-tgz:cuda (push) Successful in 8m55s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 14m13s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 14m11s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m35s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m57s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 14m23s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m45s
Build Packages / build:rpm (rocky8) (push) Successful in 11m39s
Build Packages / build:rpm (rocky9) (push) Successful in 14m0s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 13m42s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m38s
Build Packages / DIALS test (push) Successful in 14m55s
Build Packages / XDS test (durin plugin) (push) Successful in 7m11s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m7s
Build Packages / XDS test (neggia plugin) (push) Successful in 8m34s
Build Packages / Generate python client (push) Successful in 28s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 9m55s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
54c0100e8e |
v1.0.0-rc.157 (#67)
Build Packages / Unit tests (push) Successful in 1h28m28s
Build Packages / build:windows:nocuda (push) Successful in 14m45s
Build Packages / build:windows:cuda (push) Successful in 13m13s
Build Packages / build:viewer-tgz:cpu (push) Successful in 6m47s
Build Packages / build:viewer-tgz:cuda (push) Successful in 7m22s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m52s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 14m16s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 13m19s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m50s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m40s
Build Packages / build:rpm (rocky8) (push) Successful in 11m18s
Build Packages / build:rpm (rocky9) (push) Successful in 12m4s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m55s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m22s
Build Packages / DIALS test (push) Successful in 13m37s
Build Packages / XDS test (durin plugin) (push) Successful in 8m47s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m4s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m45s
Build Packages / Generate python client (push) Successful in 34s
Build Packages / Build documentation (push) Successful in 1m4s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 7m16s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
d6389e12da |
v1.0.0-rc.156 (#66)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 15m31s
Build Packages / build:viewer-tgz:cpu (push) Successful in 5m46s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m9s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m25s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m21s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m41s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m18s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m26s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m33s
Build Packages / build:rpm (rocky8) (push) Successful in 10m32s
Build Packages / build:rpm (rocky9) (push) Successful in 12m23s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m12s
Build Packages / DIALS test (push) Successful in 12m6s
Build Packages / XDS test (durin plugin) (push) Successful in 8m15s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m12s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m35s
Build Packages / Generate python client (push) Successful in 27s
Build Packages / Build documentation (push) Successful in 54s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 12m37s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
54c667190f |
v1.0.0-rc.155 (#65)
Build Packages / Unit tests (push) Successful in 1h26m8s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m38s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m45s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m39s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m55s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 14m35s
Build Packages / build:rpm (rocky8) (push) Successful in 12m28s
Build Packages / build:rpm (rocky9) (push) Successful in 13m20s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m15s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 11m43s
Build Packages / DIALS test (push) Successful in 14m21s
Build Packages / XDS test (durin plugin) (push) Successful in 7m48s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m52s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m31s
Build Packages / Generate python client (push) Successful in 15s
Build Packages / Build documentation (push) Successful in 53s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65 |
||
|
|
75e401f0e5 |
v1.0.0-rc.153 (#63)
Build Packages / Unit tests (push) Successful in 1h31m59s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 8m43s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m5s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m27s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m24s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 10m27s
Build Packages / build:rpm (rocky8) (push) Successful in 9m20s
Build Packages / build:rpm (rocky9) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m54s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m38s
Build Packages / DIALS test (push) Successful in 12m13s
Build Packages / XDS test (durin plugin) (push) Successful in 7m8s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m8s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m50s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / Build documentation (push) Successful in 50s
Build Packages / Create release (push) Skipped
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63 |
||
|
|
cc3eb8352c |
v1.0.0-rc.148 (#58)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m28s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m9s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m47s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m58s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 11m39s
Build Packages / build:rpm (rocky8) (push) Successful in 11m43s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 12m59s
Build Packages / Generate python client (push) Successful in 35s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2204) (push) Successful in 11m48s
Build Packages / build:rpm (rocky9) (push) Successful in 12m32s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m24s
Build Packages / XDS test (durin plugin) (push) Successful in 7m35s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m50s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m40s
Build Packages / DIALS test (push) Successful in 11m19s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58 |
||
|
|
fc68a9baed |
v1.0.0-rc.146 (#56)
Build Packages / Unit tests (push) Skipped
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m34s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m0s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m23s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m16s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m49s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m32s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 9m15s
Build Packages / XDS test (durin plugin) (push) Successful in 7m16s
Build Packages / Generate python client (push) Successful in 16s
Build Packages / build:rpm (rocky9) (push) Successful in 10m12s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 47s
Build Packages / DIALS test (push) Successful in 10m18s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 5m46s
Build Packages / build:rpm (rocky8) (push) Successful in 1h41m2s
Build Packages / XDS test (neggia plugin) (push) Successful in 1h59m18s
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56 |
||
|
|
bb9f5c715f |
v1.0.0-rc.135 (#44)
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m55s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m28s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 8m56s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m47s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m7s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 12m31s
Build Packages / build:rpm (rocky8) (push) Successful in 12m59s
Build Packages / build:rpm (rocky9) (push) Successful in 14m5s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 15m30s
Build Packages / Generate python client (push) Successful in 1m18s
Build Packages / Build documentation (push) Successful in 1m3s
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m8s
Build Packages / XDS test (durin plugin) (push) Successful in 9m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m59s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 9m12s
Build Packages / DIALS test (push) Successful in 11m44s
Build Packages / Unit tests (push) Successful in 1h23m8s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4) * jfjoch_viewer: Improve image render performance Reviewed-on: #44 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
928789a67c |
v1.0.0-rc.130 (#37)
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 10m14s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m4s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 12m0s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 10m31s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m0s
Build Packages / Generate python client (push) Successful in 45s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m17s
Build Packages / Create release (push) Has been skipped
Build Packages / Build documentation (push) Successful in 42s
Build Packages / build:rpm (rocky8) (push) Successful in 10m56s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m8s
Build Packages / build:rpm (rocky9) (push) Successful in 11m27s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m10s
Build Packages / Unit tests (push) Successful in 1h14m45s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Rotation indexer has two retries if failes * jfjoch_broker: Rotation indexer handles small number of rotation images (like test shot) * jfjoch_broker: Integration calculates background mask based on R2 radius * jfjoch_process: HDF5 files are not saved by default Reviewed-on: #37 |
||
|
|
1ab257af6c |
v1.0.0-rc.125 (#32)
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 12m8s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 12m57s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 12m55s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 12m0s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 13m30s
Build Packages / Generate python client (push) Successful in 20s
Build Packages / Unit tests (push) Has been skipped
Build Packages / Create release (push) Has been skipped
Build Packages / Build documentation (push) Successful in 39s
Build Packages / build:rpm (rocky8) (push) Successful in 9m23s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 10m33s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m2s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 8m42s
Build Packages / build:rpm (rocky9) (push) Successful in 9m38s
This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
07fe4dd3bb |
v1.0.0-rc.124 (#31)
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 11m23s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 10m32s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m15s
Build Packages / Generate python client (push) Successful in 19s
Build Packages / Build documentation (push) Successful in 49s
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m13s
Build Packages / build:rpm (rocky8) (push) Successful in 9m10s
Build Packages / build:rpm (rocky9) (push) Successful in 9m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 8m52s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 8m42s
Build Packages / Unit tests (push) Successful in 1h12m44s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 11m30s
This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
27496b8207 |
v1.0.0-rc.122 (#29)
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 7m57s
Build Packages / Generate python client (push) Successful in 18s
Build Packages / Build documentation (push) Successful in 35s
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (rocky9) (push) Successful in 8m28s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 7m6s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 8m9s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 8m44s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 7m56s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 7m25s
Build Packages / Unit tests (push) Successful in 1h11m19s
Build Packages / build:rpm (rocky8) (push) Successful in 6m31s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 6m40s
This is an UNSTABLE release. * jfjoch_broker: Add thresholding to prefer shorter vectors after FFT * jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress) * jfjoch_viewer: Display file opening errors * jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available Reviewed-on: #29 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
f44c6520a8 |
v1.0.0-rc.107 (#12)
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 7m33s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 8m46s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 7m19s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 7m50s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 9m8s
Build Packages / Generate python client (push) Successful in 17s
Build Packages / Build documentation (push) Successful in 43s
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (rocky8) (push) Successful in 8m25s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 7m39s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 8m13s
Build Packages / build:rpm (rocky9) (push) Successful in 9m33s
Build Packages / Unit tests (push) Successful in 1h14m4s
This is an UNSTABLE release. * jfjoch_viewer: Minor polishing of new functionality * jfjoch_broker: User NaN for empty azimuthal bins Reviewed-on: #12 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> |
||
|
|
fb9462f67b | v1.0.0-rc.92 | ||
|
|
061152279c | v1.0.0-rc.91 | ||
|
|
14f5051692 | 1.0.0-rc.89 | ||
|
|
99b7dc07f7 | 1.0.0-rc.87 | ||
|
|
5d9d2de4a4 | v1.0.0-rc.81 | ||
|
|
20cd8dc4ac | v1.0.0-rc.76 | ||
|
|
c67337cfe1 | v1.0.0-rc.72 | ||
|
|
fa95858008 | v1.0.0-rc.69 | ||
|
|
a32cec3404 | v1.0.0-rc.67 | ||
|
|
0d1b388f4d | v1.0.0-rc.50 |