28 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	a7d3ada3ab	analysis: what runs over the images is one stated mode, shared by broker, rugnux and viewer ... Until now nothing in the tree said what analysis a run performed. The answer was composed at each site out of four independent scalars - the detector type, two separate "spot finding off" switches, an indexing flag and a rotation flag - so what was configured and what actually ran were different things, and no single place could be read to find out which. AnalysisMode {None, MXRotation, MXStills, Azint, Grid, PowderCalibration} is that statement, in common/ because all three programs configure the DiffractionExperiment that carries it. AnalysisSettings sits on the experiment beside IndexingSettings, outside the DatasetSettings member, which is the one thing a /start replaces wholesale - so the mode is persistent by construction rather than by a rule someone has to remember. The mode does not label a run, it decides it. AnalysisModeStages() is a table - modes as rows, pipeline stages as columns - and every gate reads that table instead of testing the mode: spot finding in DiffractionExperiment::IsSpotFindingEnabled, indexing (and with it prediction and integration, which never run without a lattice) in one gate inside IndexAndRefine that serves all three front ends, azimuthal integration where the CPU engine is built. Two rows carry a judgement worth reviewing: powder calibration keeps spot finding, because --calibration spots fits the pooled spots; grid does not index, because a raster is thousands of frames and the per-image scoring it ranks on deliberately avoids an indexer that fires on ice. There is deliberately no Auto value. GetIndexingAlgorithm() resolves Auto at read time, which is exactly why an indexing setting cannot be read back off the configuration; removing that kind of implicitness is the point here, so the mode getter stays a plain accessor. MXStills is the default because None would silently switch analysis off on every deployment whose configuration predates the field. Rotation MX is absent from the OpenAPI schema rather than present and refused: jfjoch_broker has no rotation analysis path, so the REST and configuration-file routes cannot express it at all. The shared enum can still carry the value from elsewhere, so CheckAnalysisSettingsOnline refuses it on both routes with a message naming rugnux. A sweep collected under an MX mode is not refused - collecting rotation data online is normal and live spot counts are useful - but it is said out loud in the log, since the mistake worth preventing is the silence about what was done to it, not the acquisition. Powder calibration forces azimuthal integration onto the CPU and supplies 32 sectors where fewer than four were asked for. The FPGA integration core holds 2048 bins in total, so 32 sectors would leave 64 q bins - far too coarse to fit a ring. Frame rate is what this costs and a calibration exposure does not need it. The two existing "no analysis" switches, per-dataset dataset_settings.spot_finding and persistent SpotFindingSettings::enable, are interfaces in too many places to remove now. They are marked deprecated in the schema and in both headers, and the mode takes precedence over them: a mode that analyses no spots wins outright, while under a mode that does find spots they remain the finer control. The precedence is written where it is enforced. rugnux's ProcessMode is gone, replaced by the shared enum; RugnuxMode stays as the CLI spelling layer and no existing spelling changes. --mode gains mx_rotation and mx_stills, which are spellings of -R and --force-still rather than new switches; plain mx still chooses between them from the goniometer. scale keeps no shared counterpart, since it runs no analysis over images at all. The mode reaches the CBOR start message and /entry/MX/analysis_mode in the HDF5 master, so a written file records which analysis produced it. It is read back as provenance only - what a stored file was produced by is not what the next run should do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:16:26 +02:00
leonarski_fandClaude Opus 5	89962574ef	spindle: the severity no longer rides on the indexing seed or on which indexer is configured ... The score gated on 60 spots, but the seed escalation stops at the leanest seed that indexes - 30 spots on precisely the clean frames a grid scan produces - so the value was absent exactly where beamline automation most needs it, and absence maps to "engage": the protocol would have fired on every good frame, which degenerates the trigger into "always". The floor itself stays where it was calibrated; what changes is what it gates. When no escalation pass could answer, one severity-only pass runs over the full spot list - the row search alone, no reduction, no refinement - purely to produce the number. The same was true of the indexer choice: only the FFT family computes a row shortlist, so a deployment configured with the known-cell indexer - the ordinary online stills path - never produced the score at all. Where the severity-only pass has no row search to run, the severity is read off the rows of the winning lattice instead, which any indexer produces: the lattice's shortest few distinct directions, as many as the FFT shortlist resolves in practice, fed through the same window and scoring with equal magnitudes. The count parity is load-bearing - a worst case over every enumerable lattice direction fires on 100% of harmless mounts of a generic triclinic cell against 74% for this selection at theta_max = 15 deg, and an always-firing trigger decides nothing - while the diad-detection rate stays 1.00 on the monoclinic classes either way, a dropped axis row being recovered by the pair normals exactly as an invisible one is. A frame that neither indexed nor reached the spot floor still reports nothing, which is the honest answer and maps to the recoverable error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	c477cf0be3	spindle: a row perpendicular to the axis is as blind as a row on it ... The blind region a sweep leaves is a double cone, so a 2-fold sends it to two places: 2*beta away, and 180-2*beta away. The severity took only the first, and scored a row perpendicular to the spindle as 0 - "symmetry repairs everything" - when such a 2-fold in fact carries the cone onto its opposite lobe, which the sweep leaves equally unmeasured. Folding the miss-angle to min(beta, 90-beta) covers both images and reproduces a Monte-Carlo of the true overlap to 0.002. The failure was silent and in the dangerous direction, and it fired on the more common geometry: for a random axis the perpendicular band is several times wider than the aligned one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	91775cc4fb	geometry refinement: at coarse slicing, forgive the miss the exposure could have supplied ... The acceptance gate that decides which spots enter the geometry refinement compares a fractional-index miss against a fixed tolerance, and that comparison assumes the spot diffracted at the frame's midpoint. It did not. The rotation coordinate is bracketed, not observed - the spot diffracted somewhere inside the exposure - and profiling that unknown angle out of the least squares has a closed form. Writing u = m x q for the direction a rotation delta moves the observation along, the component of the miss along u is free up to the exposure's rms half-width and only the excess is charged; no other direction is touched, so |q|, and with it every d-spacing, is unaffected. Without it the gate is a resolution cut that tightens with the frame width, the miss growing as a/d. Where the benefit lives is the per-frame refinement, which holds one frame and therefore deliberately does not back-rotate - the frame's angle is a gauge its orientation block absorbs - and passes no goniometer at all. The spots on that frame still span the exposure, and that spread does not go away with the gauge, so the spindle and the wedge are handed over separately, for the gate alone. The spindle is also written into the rotation vector the frame constants carry, which was left at the lab x axis whenever no goniometer was passed; with a zero angle that vector is multiplied out and the block is held constant, so nothing changes by it today, but it was a trap for the next reader. The width is k = 1/sqrt(12), the rms of a rotation coordinate uniform over the frame, which is what a least-squares is calibrated on. Half the exposure - the worst case a spot could sit at - degrades accuracy against a ground truth, and so does forgiving the direction outright with no bound: the bound is load-bearing, and what it protects is fine slicing. Measured on 112 paired datasets, scored against deposited structure factors because neither ISa nor R_meas can arbitrate this - both move opposite to accuracy along the neighbouring partiality knob. At 0.5 degrees per image or coarser, n = 16: CC to the deposited data better on 9 of 12, median dISa +0.87%, and at the top of the a x oscillation lever better on 9 of 10 with the per-shell CC improving in all ten shells, most in the outer half. The win rises with that lever and the population below it is inert. It is applied only at 0.5 degrees per image and above. The band is principled - the dead zone matters once the exposure's rms rotation ambiguity, 0.29 of the wedge, is comparable to the crystal's own along-u rocking spread, which measures around 0.26 degrees, putting the boundary between 0.25 and 1.0 degrees - while the point inside that band is empirical and taken at the conservative end. It has to be there: the two datasets the corpus lost outright both sit at 0.2 degrees or finer, one of them finely sliced, and the response is not monotone in the width, so no bound on the width alone makes them safe. Below the trigger the block is skipped and the gate computes what it always computed; a finely sliced sweep reprocessed across this change is byte for byte identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:08:31 +02:00
leonarski_fandClaude Opus 5	28a98e1a14	stills: a per-image number for how much of a sweep's blind cone this orientation loses ... A rotation sweep never records a double cone of half-angle asin(lambda/2d) about the spindle. That loss is normally repaired by the point group; it is not repaired when an operator's axis lies inside the cone, because the cone then maps onto itself. A still cannot know the point group, but it can see where the crystal's short lattice rows are, and a symmetry axis is always one of them - measured over 107 solved cells its length is 1.4x the shortest row at the median. The number is the fraction of the blind cone that a 2-fold about the nearest short row carries back into the blind cone: two equal caps of angular radius theta_max whose centres are 2*beta apart, which is the circle-lens area in beta/theta_max and within 0.035 of the spherical value even for a 55 deg cone. 0 means one sweep reaches everything symmetry could give; 1 means the row is on the spindle and the whole cone is lost coherently. No goniometer geometry enters, so the number describes the problem and leaves the remedy to the beamline. Free: it rides on the FilterFFTResults shortlist the indexer already builds, needing only the spindle, the wavelength and the frame's own resolution. FilterFFTResults gains an optional out-parameter for each row's peak prominence, which the length window is taken over. Measured on 384 stills of 22 solved crystals against the frames' own symmetry axes, with the same procedure re-run along five decoy directions per frame as the null: 0.93 of severe orientations reported severe, 0.010 of harmless ones reported severe, AUC 0.948, and the value tracks the true severity to 0.10 at the 90th percentile. Below 60 spots the misses triple, so that is where it stops answering. It is blind to a symmetry axis much longer than the crystal's shortest row: measured on synthetic stills the search grid finds a 150 A row every time, a 200 A row half the time and a 300 A row once in sixteen. Two cheap consistency tests refuse to answer for part of that regime and cost nothing on real frames, but they do not cover it - the reach is a documented property of the number, not something every frame can detect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-03 23:57:12 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	4878318c27	v1.0.0-rc.139 (#49) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start` * jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start' * jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time * jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received * jfjoch_broker: Use nanoseconds internally for frame/image/readout time * jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished) * jfojch_broker: Improve profiling of different data analysis steps * jfjoch_broker: Record integration reflection count * jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds) * jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked * jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets **Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time. Reviewed-on: #49	2026-04-29 09:50:50 +02:00
leonarski_f	c1c170112c	v1.0.0-rc.136 (#45) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress). Reviewed-on: #45	2026-04-20 11:54:33 +02:00
leonarski_f	6133da1377	v1.0.0-rc.133 ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Use httplib for HTTP server instead of Pistache * jfjoch_broker: Drop OpenSSL support * jfjoch_broker: Base work for multi-lattice support in the future * Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib) Reviewed-on: #41	2026-03-26 20:50:33 +01:00
leonarski_f	64002f1e29	v1.0.0-rc.129 (#36) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files) * jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction * jfjoch_broker: Image pusher statistics are accessible via the REST interface * jfjoch_writer: Supports TCP image socket and for these auto-forking option Reviewed-on: #36 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-03-05 22:13:12 +01:00
leonarski_f	3036b1d37f	v1.0.0-rc.128 (#35) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Handle properly reuse of image buffer locations * jfjoch_broker: Fix bug in counting idle slots * jfjoch_broker: Force obtuse angle for monoclinic cells * jfjoch_process: Change scaling refinement tolerance Reviewed-on: #35 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-03-03 22:24:44 +01:00
leonarski_f	f3e0a15d26	v1.0.0-rc.127 (#34) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Default EIGER readout time is 20 microseconds * jfjoch_broker: Multiple improvements regarding performance * jfjoch_broker: Image buffer allows to track frames in preparation and sending * jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer * jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets * jfjoch_writer: Fixes regarding properly closing files in long data collections * jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code Reviewed-on: #34	2026-03-02 15:57:12 +01:00
leonarski_f	1ab257af6c	v1.0.0-rc.125 (#32) ... This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-18 16:17:21 +01:00
leonarski_f	07fe4dd3bb	v1.0.0-rc.124 (#31) ... This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-01 13:29:33 +01:00
leonarski_f	27496b8207	v1.0.0-rc.122 (#29) ... This is an UNSTABLE release. * jfjoch_broker: Add thresholding to prefer shorter vectors after FFT * jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress) * jfjoch_viewer: Display file opening errors * jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available Reviewed-on: #29 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-16 15:27:40 +01:00
leonarski_f	b02c412d59	v1.0.0-rc.118 (#25) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_viewer: Fix issue when HTTP sync silently disconnected when it was enabled when the broker was starting measurement. * jfjoch_broker: Add protections on time of geometry optimization and reduce rotation recalculations Reviewed-on: #25 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-06 20:05:30 +01:00
leonarski_f	31a357fa57	v1.0.0-rc.113 (#19) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_broker: Improve handling of rotation indexing * jfjoch_broker: More information saved in CBOR end message (WIP) * jfjoch_writer: Save rotation indexing lattice parameters and Niggli class * jfjoch_viewer: Remove (for now) primitive cell information Reviewed-on: #19 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-02 09:29:22 +01:00
leonarski_f	06949caf1a	v1.0.0-rc.112 (#18) ... This is an UNSTABLE release and not recommended for production use (please use rc.11 instead). * jfjoch_broker: Experimental rotation (3D) indexing * jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values Reviewed-on: #18 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-30 17:39:22 +01:00