1122 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_f	d34c19f6d3	VERSION: 1.0.0-rc.159	2026-07-13 11:58:24 +02:00
leonarski_fandClaude Opus 4.8	85fcac09b5	scale_merge: floor the R-free test-set size on small data ... AssignRfreeFlags gains a min_free_reflections floor (default 500): on small data, where the 5% fraction would give too few test reflections for a stable R-free (Brunger's ~500-2000 rule), the fraction is lifted toward ~500 free reflections, capped at 10% so a large test set never steals working data. An explicit rfree_fraction is still honoured; pass 0 to disable. For ordinary data (5% already clears the floor) this is inactive and the fraction stays flat, so the cross-dataset-identical property is preserved; the floor only lifts the fraction on genuinely small datasets, where per-dataset R-free stability outweighs cross-dataset identity (a shared reference set keeps exact identity there). Tests updated to isolate the pure-hash guarantees with the floor off, plus a new small-data floor + cap test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 11:50:28 +02:00
leonarski_fandClaude Opus 4.8	069b00349f	frontend: show calibration progress and relabel Pedestal as Calibration ... The status-bar pill only filled a progress bar during the Measuring state, even though the receiver already reports a real progress value during pedestal/mask calibration. Render the same bar for the Pedestal state (filled coral to match its pill colour), and relabel the state and its button as "Calibration"/"CALIBRATE". The internal state value stays "Pedestal" and the /pedestal endpoint is unchanged for back-compatibility. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 11:40:30 +02:00
leonarski_fandClaude Opus 4.8	d891551127	rugnux: make model validation batch-consistent for fragment screening ... Reworks the --model / French-Wilson / R-free path so a batch of datasets of one crystal form yields comparable maps and a shared test set, and fixes a French-Wilson prior bug. - French-Wilson: fix inverted centric/acentric prior - the call passed !centric into the selector, swapping the two Wilson forms for every weak reflection. Add a test pinning a centric weak |F| below an acentric one. - R-free flags: assign by a pure per-hkl hash instead of a per-dataset resolution-shell stratification, so every dataset of one crystal form gets the same free set (needed for ensemble refinement / PanDDA). Still a pure function of the Laue-ASU key, so Bijvoet/symmetry mates never split. - R-free flags: import a reference MTZ's FreeR_flag column when present (CCP4 test=0, auto-complement for a phenix-style test=1), so a whole campaign inherits one shared test set. New ApplyReferenceFreeFlags, wired post-merge in both the full-analysis and --scale paths. - Model maps: scale the model with an overall scale, anisotropic B and flat bulk solvent only - drop the free-form per-shell K(1/d^2) rescale, which reshaped each dataset's radial amplitude profile differently and made a batch of maps non-comparable. - Merohedral indexing: model validation resolves the ambiguity by lowest R-free only when there is no reference; with a reference MTZ the indexing is already resolved against it (merge stage / stills scaling) and that authoritative choice is kept. - Remove a stale <algorithm> include; update docs and changelog. Tests: french_wilson, rfree, reference_mtz (new), reindex, hkl_key all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 11:31:33 +02:00
leonarski_fandClaude Opus 4.8	f81830df4f	scale_merge: rescue a genuine high-symmetry merge from the chi2 gate by its systematic error ... A weakly-scaled genuine high-symmetry crystal (cubic insulin Ins_I_3) was demoted I23 -> I222: its cubic 3-fold passes the operator-CC gate, but merging across it inflates the within-orbit chi2 to ~2x the best subgroup - just past the self-consistency bound. A merohedral twin (Ins_H_2's R3->R32) lands at the same chi2 ratio, so the ratio alone cannot separate a genuine-but-poorly-scaled promotion from a twin. Add a systematic-error test that does separate them. merge_systematic_b solves for the extra intensity-proportional error b (sigma^2 + (b I)^2) that a point group's merge must invoke to reach chi2 = 1. A genuine symmetry step gains multiplicity without intensity-proportional disagreement, so b barely grows (Ins_I_3 b x1.6 vs its confirmed subgroup); a twin forces non-equivalent reflections together, so b balloons (Ins_H_2 x2.2). A candidate whose chi2 is only just past the ratio bound (max_merge_chi2_rescue) is rescued when its b, relative to the largest confirmed subgroup (by rotation-set inclusion), stayed within max_systematic_b_ratio. Battery: 20/24 space-group match (was 19/24) - Ins_I_3 -> I23, every other crystal bit-identical including the twins that must stay in lower symmetry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 09:36:29 +02:00
leonarski_fandClaude Opus 4.8	b350c46777	docs: reference-based space-group/ambiguity resolution and model alignment ... Document, in CPU_DATA_ANALYSIS.md, how a reference MTZ fixes the space group and cell and resolves the indexing (merohedral) ambiguity by CCref (new §10.9), and how model validation reindexes the data into the model's enantiomorph and probes merohedral reindexings by R-free (new §14.5); add the matching rc.159 changelog entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 08:58:11 +02:00
leonarski_fandClaude Opus 4.8	386f345758	rugnux: probe indexing ambiguities against the model in validation ... ModelValidation now fits the scaled model to each reindexing of the observed reflections (identity plus the twin laws from ReindexAmbiguityOperators) and keeps the one with the lowest R-free. The scale + per-shell + R computation is factored into a per-candidate lambda so every candidate is scored identically. No-op for a holohedral crystal (no twin laws), e.g. lysozyme, where the only candidate is the identity. The enantiomorph/screw ambiguity is still resolved from the model hand, not probed - |Fcalc| is the same for both hands so R-free cannot distinguish them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 08:20:27 +02:00
leonarski_fandClaude Opus 4.8	465cf1aa5e	rugnux: resolve the indexing ambiguity against a reference MTZ (rotation) ... A reference MTZ is now allowed for rotation data: it fixes the space group and cell (on the CLI) and resolves the indexing (merohedral) ambiguity, but is NOT used to scale - the rotation merge stays self-consistent. Previously rotation + reference was a hard error. After the space group is determined, ChooseReindex/ReferenceIntensityCC pick the reindexing whose merged intensities best correlate with the reference and, if it is not the identity, the twin-law reindex is baked into the integration outcome (hkl labels only; the cell is unchanged, the reindex is metric- preserving), the rotation scale-merge is re-ingested and the data re-merged in that indexing. No-op for a holohedral crystal (no twin laws), e.g. lysozyme. Stills resolve the per-image ambiguity in ScaleOnTheFly, not here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 08:20:19 +02:00
leonarski_fandClaude Opus 4.8	e91b520863	rugnux: fix SG/cell from a reference MTZ, and reindex data into the model's enantiomorph ... Two ways the processing now adopts symmetry from an external reference: - Reference MTZ (-z) fixes the space group and unit cell, unless -S / -C override them (the explicit flag always wins - so -S with the "wrong" enantiomorph is allowed). The cell is a soft reference: indexing may still drift within tolerance, as with -C. - rugnux --model: when the data was merged in the enantiomorph of the model's space group (e.g. data P4(1)2(1)2, model P4(3)2(1)2 - the merged intensities cannot tell them apart), ValidateAgainstModel reindexes the observed reflections into the model's hand (via gemmi change_of_hand_op). This keeps the data setting consistent with the model; it does not change the R-factors, which use |F|. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 22:30:33 +02:00
leonarski_fandClaude Opus 4.8	d00f4303c8	scale_merge: add a reindexing-ambiguity resolver ... Shared helper to resolve the indexing (alternative-indexing / reindexing) ambiguity that arises when the lattice symmetry is higher than the crystal Laue symmetry (merohedral / pseudo-merohedral): a crystal can be validly indexed in several ways related by reindexing operators, and the right one is chosen by agreement with a reference (reference-MTZ intensities, or later a model) - so no reference-free ambigator (cosym) is needed. ReindexAmbiguity.{h,cpp}: - ReindexAmbiguityOperators() = gemmi::find_twin_laws() = the coset operators (empty for a holohedral crystal, i.e. no ambiguity); - ReindexReflections() applies an operator to a copy of the merged reflections; - ChooseReindex() picks identity or the twin law that maximizes a caller-supplied score; - ReferenceIntensityCC() is a ready scorer (Pearson CC vs a reference, matched by Laue-ASU key). This does NOT touch the screw-axis / enantiomorph ambiguity (P4_1 vs P4_3), which leaves the merged intensities unchanged and is adopted from the reference/model rather than scored. Tests: holohedral -> no operators, merohedral (P4) -> operators present, a misindexed dataset is recovered by reference CC (score >0.99), a correctly indexed one keeps identity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 22:30:18 +02:00
leonarski_fandClaude Opus 4.8	9f23976e02	tests: add Catch2 tests for R-free flag assignment and French-Wilson ... RfreeFlagsTest: determinism (pure function of the reflection), ~5% free fraction, Friedel/Bijvoet pairs never split, symmetry equivalents share a flag, resolution stratification (free set spans shells), and fraction=0 flags nothing. FrenchWilsonTest: strong reflections reduce to sqrt(I), weak/negative intensities get a positive amplitude, amplitudes are always finite and non-negative (incl. centric / epsilon>1 reflections in P4(3)2(1)2), and unusable sigma falls back to sqrt(max(I,0)). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 22:06:13 +02:00
leonarski_fandClaude Opus 4.8	e7fe203625	docs: add rc.159 changelog (rugnux --model, French-Wilson amplitudes, R-free flags) ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 21:40:40 +02:00
leonarski_fandClaude Opus 4.8	97e6d99d03	docs: document French-Wilson amplitudes, R-free flags and model-based maps ... Update CPU_DATA_ANALYSIS.md for the new merge/analysis capabilities: - §10.7 R-free test-set flags: Friedel-merged Laue-ASU key (Bijvoet-safe), deterministic splitmix64 selection, resolution stratification; - §10.8 French-Wilson amplitudes: posterior-mean |F| from I and sigma with the Wilson prior (centric/acentric, epsilon multiplicity, strong-reflection short-circuit), written as MTZ F/SIGF and mmCIF F_meas_au; - §14 model-based validation (rugnux --model): Fcalc, flat bulk solvent + overall and per-shell scaling, R-work/R-free vs a model, and 2Fo-Fc / Fo-Fc maps. Also refresh the scope list and references, and correct the old "intensities only" note (the output now carries amplitudes too). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 21:40:40 +02:00
leonarski_fandClaude Opus 4.8	d6924aa901	rugnux: add --model for R-free and 2Fo-Fc / Fo-Fc maps from an atomic model ... New offline capability: `rugnux --model model.pdb` validates the merged intensities against a PDB model. It computes structure factors from the model on a grid (IT92 x-ray form factors, FFT), adds an optimized flat bulk solvent (SolventMasker + gemmi Scaling: k_sol, b_sol, overall k and anisotropic B) and a per-resolution-shell scale, then reports R-work / R-free and writes 2Fo-Fc and Fo-Fc CCP4 maps plus a map-coefficient MTZ (FP, FC, PHIC, FWT/PHWT, DELFWT/PHDELWT, FREE). No refinement of the structure itself; the model is only re-fractionalized into the data cell (a rigid cell adjustment). The observed amplitudes are the French-Wilson |F| already on the merged reflections (MergedReflection.F, computed at end of merge), so the R-free/maps use exactly the same amplitudes as the written reflection file. ModelValidation.{h,cpp} holds the logic; wired via --model (ProcessConfig.model_path) into both the full pipeline (Rugnux::Run write site) and the --scale re-merge path. Links the unified `gemmi` static target for the model / structure-factor / map headers. Verified on lyso_ref: R-free 0.346, mean 2Fo-Fc at atom centres +2.16 sigma, maps written. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 21:27:44 +02:00
leonarski_fandClaude Opus 4.8	87af0f419b	scale_merge: compute French-Wilson amplitudes at end of merge, write them alongside intensities ... Merged reflections carried only intensities; a naive sqrt(max(I,0)) turns every weak or negative measurement into a biased (or zero) amplitude. Add a French-Wilson step so the merged reflections themselves carry proper Bayesian amplitudes. New FrenchWilson.{h,cpp}: ApplyFrenchWilson() fills F and sigmaF on each MergedReflection with the posterior mean |F| given I and its sigma under the Wilson prior: - correct centric vs acentric prior (gemmi is_reflection_centric); - epsilon (symmetry-enhancement) multiplicity: the shell mean is <I/eps> and the prior mean for a reflection is eps * <I/eps>; - numerically stable log-shift integration of the posterior; - strong reflections (I > 4 sigma) short-circuit to sqrt(I) where the FW bias is negligible; - unusable I/sigma falls back to sqrt(max(I,0)). It runs as the last step of the merge routine (both MergeOnTheFly and RotationScaleMerge), so F/sigmaF are part of the merged result and downstream consumers (file writing AND, later, map calculation) all use the same amplitudes rather than each recomputing FW and risking divergence. The writers just emit the fields; extended additively: - MTZ: F (F) + SIGF (Q) columns; - mmCIF: _refln.F_meas_au / _refln.F_meas_sigma_au; - HKL text: F sigmaF appended (rfree flag kept in place for back-compat). Existing intensity columns are unchanged, so current readers keep working while phenix.refine (etc.) can now refine against the amplitudes. Verified on lyso (1.3 A): all 173 negative-intensity reflections get F>0, strong reflections have F/sqrt(I)=1.000, no NaN, and the mmCIF _refln loop is well-formed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 21:27:44 +02:00
leonarski_fandClaude Opus 4.8	4c95a289b6	scale_merge: harden R-free flag assignment (Bijvoet-safe, deterministic, stratified) ... The R-free flags were drawn with a per-reflection std::bernoulli_distribution off a shared std::mt19937(12345). Two problems: in anomalous mode I(+) and I(-) are separate merged rows, so a Bijvoet pair could be split across the work and free sets (biasing R-free); and the draw order depended on merge/call order, so it was not reproducible or parallel-safe. The resolution-shell grouping was cosmetic — it still drew an independent Bernoulli per row, so it neither tied mates together nor guaranteed a per-shell quota. New AssignRfreeFlags() (own file RfreeFlags.{h,cpp}) replaces both call sites (stills MergeOnTheFly, rotation RotationScaleMerge). It: - keys each reflection by its Friedel-merged (Laue) ASU index, so all symmetry- and Friedel-equivalent reflections (including a Bijvoet pair) share one flag and are never split; - selects the free set by a deterministic splitmix64 hash of that key (same idiom as the CC1/2 half-set split), so it is reproducible run-to-run and order-independent; - stratifies by resolution: exactly round(fraction * n) of the distinct reflections in each of 20 shells is flagged free. Verified on an anomalous lyso merge (29536 refl, 5% free): 13548 Bijvoet pairs, 0 split; free fraction 4.6-5.4% in every resolution shell. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 21:27:44 +02:00
leonarski_fandClaude Opus 4.8	5addb4483c	gemmi_gph: vendor the model / structure-factor headers into the single gemmi target ... Extend the vendored GEMMI subset (v0.7.5) with the atomic-model, structure-factor, bulk-solvent and map machinery so the whole thing builds as one static `gemmi` library instead of a separate target: - add the model/SF/map sources compiled into `gemmi`: pdb, resinfo, polyheur, calculate, eig3, ccp4 - add the v0.7.5 headers these pull in (model.hpp, dencalc.hpp, sfcalc.hpp, solmask.hpp, scaling.hpp, fourier.hpp, grid.hpp, ccp4.hpp, it92.hpp, ...) plus third_party/pocketfft (FFT), half, tinydir Only the low-level string/math/symmetry headers were present before; this makes the vendored copy a complete, self-consistent gemmi that can read a PDB and do density / structure-factor / map calculations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 21:27:44 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.158	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.157	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.156	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65 1.0.0-rc.155	2026-06-25 22:01:48 +02:00
leonarski_f	6136f858af	v1.0.0-rc.154 (#64) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.154	2026-06-25 18:12:00 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63 1.0.0-rc.153	2026-06-23 20:29:49 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62 1.0.0-rc.152	2026-06-17 20:36:24 +02:00
leonarski_f	ef52dac2ee	v1.0.0-rc.151 (#61) ... * jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname Reviewed-on: #61 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.151	2026-06-16 14:13:29 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60 1.0.0-rc.150	2026-06-15 20:24:15 +02:00
leonarski_f	ea575f790a	v1.0.0-rc.149 (#59) ... * XDS plugin: Fix HDF5 mutex to run on multiple processors Reviewed-on: #59 1.0.0-rc.149	2026-06-13 21:27:41 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58 1.0.0-rc.148	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56 1.0.0-rc.146	2026-05-28 18:48:35 +02:00
leonarski_f	75f1c5f954	SHIM library improvements from the HDF Group	2026-05-08 11:39:51 +02:00
leonarski_f	caef26873e	v1.0.0-rc.145 (#55) ... This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144. * **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future. * XDS plugin: Improve performance of VDS reading. * jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD. * jfjoch_writer: Clean-up of HDF5 routines to better handle issues. Reviewed-on: #55 1.0.0-rc.145	2026-05-06 21:50:02 +02:00
leonarski_f	7d34e8a049	v1.0.0-rc.144 (#54) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Improve performance of preview JPEG image generator at receiver startup (saving about 150 ms on measurement start for 16M) Reviewed-on: #54 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.144	2026-05-01 17:06:36 +02:00
leonarski_f	6e0bb971ac	v1.0.0-rc.143 (#53) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Avoid copying gain calibration together with DiffractionExperiment Reviewed-on: #53 1.0.0-rc.143	2026-04-30 22:16:50 +02:00
leonarski_f	87fde1b32e	v1.0.0-rc.142 (#52) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8 * Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent Reviewed-on: #52 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 16:47:53 +02:00
leonarski_f	d760b12a18	v1.0.0-rc.141 (#51) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times * frontend: Fix selection of FFTW in indexing settings Reviewed-on: #51 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 13:04:54 +02:00
leonarski_f	239a441ee6	v1.0.0-rc.140 (#50) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection * jfjoch_broker: Minor bug fixes for rare conditions Reviewed-on: #50	2026-04-29 21:40:22 +02:00
leonarski_f	4878318c27	v1.0.0-rc.139 (#49) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start` * jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start' * jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time * jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received * jfjoch_broker: Use nanoseconds internally for frame/image/readout time * jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished) * jfojch_broker: Improve profiling of different data analysis steps * jfjoch_broker: Record integration reflection count * jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds) * jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked * jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets **Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time. Reviewed-on: #49 1.0.0-rc.139	2026-04-29 09:50:50 +02:00
leonarski_fandtakaba_k	230480e390	v1.0.0-rc.138 (#48) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Cleanup DECTRIS start-up code to enable a shorter start time jfjoch_broker: Allow for asynchronous start to allow overlapping detector configuration with other beamline preparations jfjoch_broker: Goniometer axis name is converted to lowercase jfjoch_broker: Fix bug, where wrong HTTP error codes were returned jfjoch_broker: Improve sigma estimation during merging (K. Takaba) --------- Co-authored-by: takaba_k <kiyofumi.takaba@psi.ch> Reviewed-on: #48 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.138	2026-04-27 19:56:14 +02:00
leonarski_f	c981e1b91c	v1.0.0-rc.137 (#46) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Better track time for each operation in the processing stack * jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress) * jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress) * jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0) Reviewed-on: #46 1.0.0-rc.137	2026-04-25 19:59:21 +02:00
leonarski_f	c1c170112c	v1.0.0-rc.136 (#45) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress). Reviewed-on: #45 1.0.0-rc.136	2026-04-20 11:54:33 +02:00
leonarski_f	bb9f5c715f	v1.0.0-rc.135 (#44) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4) * jfjoch_viewer: Improve image render performance Reviewed-on: #44 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.135	2026-04-16 11:59:59 +02:00
leonarski_f	4a852b4d6b	v1.0.0-rc.134 (#43) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long * jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata * XDS plugin: Add generation of Jungfraujoch plugin for XDS * CI: Add tests with XDS and DIALS (`xia2.ssx`) Reviewed-on: #43 1.0.0-rc.134	2026-04-09 13:30:47 +02:00
leonarski_f	81bd9a06a1	CI pipeline upgrade (#42) ... Updates to CI pipeline * New docker image for Ubuntu 22.04 with CMake 3.26 * New docker image for Rocky 9 with DIALS 3.27 * New automated test to check for DIALS processing with xia2.ssx Reviewed-on: #42	2026-03-28 20:06:23 +01:00
leonarski_f	6133da1377	v1.0.0-rc.133 ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Use httplib for HTTP server instead of Pistache * jfjoch_broker: Drop OpenSSL support * jfjoch_broker: Base work for multi-lattice support in the future * Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib) Reviewed-on: #41 1.0.0-rc.133	2026-03-26 20:50:33 +01:00
leonarski_f	2ab9deb20b	v1.0.0-rc.123 (#40) ... Documentation update Reviewed-on: #40 1.0.0-rc.132	2026-03-12 12:17:11 +01:00
leonarski_f	166fcdb68f	v1.0.0-rc.131 (#39) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Fix bug in saving JUNGFRAU calibration (pedestal/pedestalRMS) * jfjoch_viewer: Fix calibration (pedestal) images being open flipped * jfjoch_process: Add space group detection (EXPERIMENTAL) Reviewed-on: #39 1.0.0-rc.131	2026-03-07 11:34:04 +01:00
leonarski_f	928789a67c	v1.0.0-rc.130 (#37) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Rotation indexer has two retries if failes * jfjoch_broker: Rotation indexer handles small number of rotation images (like test shot) * jfjoch_broker: Integration calculates background mask based on R2 radius * jfjoch_process: HDF5 files are not saved by default Reviewed-on: #37 1.0.0-rc.130	2026-03-06 14:38:56 +01:00
leonarski_f	64002f1e29	v1.0.0-rc.129 (#36) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files) * jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction * jfjoch_broker: Image pusher statistics are accessible via the REST interface * jfjoch_writer: Supports TCP image socket and for these auto-forking option Reviewed-on: #36 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.129	2026-03-05 22:13:12 +01:00
leonarski_f	3036b1d37f	v1.0.0-rc.128 (#35) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Handle properly reuse of image buffer locations * jfjoch_broker: Fix bug in counting idle slots * jfjoch_broker: Force obtuse angle for monoclinic cells * jfjoch_process: Change scaling refinement tolerance Reviewed-on: #35 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.128	2026-03-03 22:24:44 +01:00