Completes the rename the C++ identifiers already carried. Producers now emit ice_ring_ratio /
ice_ring_ratio_mean over CBOR and write /entry/MX/iceRingRatio and iceRingRatioMean; both readers
accept the retired ice_ring_score / iceRingScore spellings as well, so nothing that exists on disk
or on the wire stops being readable. The REST plot_type gains ice_ring_ratio and KEEPS
ice_ring_score, both mapping to the same plot, so no client breaks. --ice-min-score is deliberately
unchanged: it is a threshold on the ratio, it is the one surface a user types, and this program has
91 long options and no aliases.
The fallback is not a formality. rugnux --mode scale reads the stored per-image value to reproduce
the ice gate the writing run applied, and an unread dataset does not fail loudly - ice_n == 0 sends
the gate to its fail-safe branch, "ice present". Demonstrated on a clean crystal whose stored file
uses the old names: with the fallback the gate reads 1.06 against a 1.50 threshold and skips ice
handling; with the fallback removed the same file has 272056 of 1032493 reflections (26%) excluded
from the scale fit, and ISa, R_meas, I/sigma, SIGANO and both error-model terms all move. That is a
silent change to merged intensities on files already written, which is why the two new CBOR test
cases were each checked to FAIL when their fallback is taken out rather than merely to pass.
Verified: same binary on an old-name and a new-name copy of one file gives byte-identical .hkl,
.mtz and unmerged .mtz and an identical report; a file written after the rename scales to the same
bytes again; the master and the data-file read paths were each exercised with both spellings; and
the live API serves the same plot under either name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
Both lanes added a per-image scalar to the same eleven files, so every conflict was two
additions competing for one line. All were resolved by keeping both, with three that needed
more than that:
- ScanResultGenerator: rc167 changed the per-image float vectors to resize(n, NAN) so a frame
that never arrived does not read back as a real 0. v_protein_score and v_ice_score are exactly
that case - 0 is a real answer ("nothing detected here") - so they take the NAN default too.
- HDF5MetadataSource: rc167 established that NaN in a stored per-image array means "no value" and
the optional must come back absent. The two detection scores now follow it, which they did not
before the merge; without the guard a missing score would come back as a NaN that a threshold
would silently compare against.
- CBORTest: designated initialisers must follow member declaration order, so spindle_blind_fraction
precedes the two scores in the DataMessage aggregate.
Verified after the merge that every CBOR key that is encoded is also decoded (198 encoded keys,
one intentional exception: the "type" discriminator), that both lanes' fields survive in the
writer, the reader, the plots and the API, and that a stored file still round-trips.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
proteinScore and iceScore are saturating scores in [0,1] that answer presence, not quality: a superb
crystal and a barely-diffracting one both read near 1, and neither the spot count nor the resolution
enters either of them as a term. iceRingScore already reports a magnitude - a ratio, unbounded - which
is not a number that can be thresholded; these are.
The protein score counts distinct d SHELLS above 5 A rather than spots, so a parasitic ring in the
low-resolution band cannot accumulate evidence, and weights each spot against the frame's own median
so a scattering of the weakest detections cannot fill a shell either. The ice score carries two
channels and takes the stronger: a radial one over the azimuthal profile, which runs the hexagonal
and the CUBIC phase as separate hypotheses and decides between them at the end (flash-cooled loops
show cubic or stacking-disordered ice at least as often as hexagonal, the two share only three
lines, and dropping the cubic hypothesis costs about 5 pp on iced loops), and a spot one that reads
an excess on the ice radii against the same band slid to every ice-free offset, which is what
catches ice arriving as discrete crystallites and leaving the radial profile flat.
Both read d out of the geometry, so both move with a beam-centre error; the centre is not fitted
here, and the one they were computed with is written beside them as scoreBeamCenterX/Y so a later
rescoring can tell an algorithm disagreement from a geometry one.
Ported from validated prototypes and checked against them frame by frame on stored data: mean
absolute difference 2.7e-5 (protein), 1.3e-8 (ice radial) and 3.4e-4 (ice spots). On a 41-loop
battery the protein score reaches 98.4% of confirmed-protein frames and 0.00% of water frames, and
finds no cluster on any water or ice raster. Cost is 0.01 ms/frame for the protein score and
0.08-0.32 ms/frame for the ice score.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
A reflection arriving at an angle to the detector normal crosses D/cos(alpha) of
whatever lies between the sample and the sensor, not D, so it is attenuated more than
one arriving head-on and reads low. That is the same geometry as the sensor crossing
already corrected here and the opposite sign, and it was missing.
The factor is exp(D/L*(1/cos(alpha)-1)) from the NIST attenuation coefficient of the
medium, the stated distance and the stated wavelength. Nothing in it is fitted, and
it is not justified by any measured amplitude: the flight path and the sensor
crossing are collinear to better than 0.998 over the angular range any single
experiment samples, so no fit of one can be evidence for the other. It is the
tabulated absorption of a known thickness of a known material over a known path.
The medium cannot be detected. No field of the NXmx application definition describes
it, none of the masters this program reads carries one, and it cannot be inferred
from the implied transmission either - in this corpus a station confirmed to use
helium sits at 51% implied air transmission and one confirmed to use air at 63%, so
any rule separating them is a threshold fitted between two points. It is therefore
assumed, stated, and overridable: --flight-path air|helium|vacuum, defaulting to air.
Helium is its own material rather than an alias for vacuum, attenuating about a six
hundredth of air rather than nothing.
On an untilted detector the correction is a function of resolution alone, so its
entire effect on merged data is a shift in the Wilson B - which is what the report
now prints beside the assumption, accurate to better than a tenth of an angstrom
squared against measurement from 0.05 up to 28. Where that shift is large the report
warns, because a wrong medium is then the largest number in the run: applied to data
from the confirmed helium station it returns a B of 14 A^2 at 3.0 A resolution, which
is not a value a crystal can have.
The corpus contains its own control. One crystal, one station, three collections a
quarter of an hour apart at falling energy through the same air: corrected, the
Wilson B rises monotonically with the dose, as it must.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
LP is a column other programs read and un-apply. It is documented as the
Lorentz-polarization factor, and until the sensor efficiency was folded into the
same product that is what it held. Measured on our own unmerged file, LP spanned a
factor of 1.1343 across the detector where pure L/P spans nothing of the sort - the
excess is the efficiency, 13% end to end at 13 keV and 34% at 18 keV.
Both reference implementations keep it out. Recomputing pure L/P from a stored XDS
file's own geometry over its 124k reflections leaves RLP flat to 0.1% from 8.6 to
33.7 degrees, where a folded-in efficiency would have shown a 7% trend - and XDS has
the sensor numbers in hand. DIALS fills LP from lorentz and polarization alone and
writes QE as a separate column, even a column of ones.
Split them: LP is L/P again, QE is the efficiency, and the intensity is unchanged,
so a reader that un-applies LP recovers what it expects and one that wants raw
counts divides by LP and multiplies by QE. Only the unmerged file moves - every
other column is bit-identical and the file grows by exactly one float per reflection
plus one header record. The merged files are byte for byte what they were.
The process file gains an optional qe dataset beside the existing one rather than
changing what that one means, so a file written before the efficiency existed still
loads, and reads back as a correction of exactly one - which is what it was. It is
stored rather than recomputed on read because the writer has no geometry to
recompute it from, and because recomputing would have written a radial trend into
every stored file that never had one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
The factor multiplied into each integrated intensity was called rlp, for reciprocal
Lorentz-polarization, and until this week that is all it held. It now also carries
the sensor efficiency at the angle the beam arrives, and on the stills path it holds
that efficiency and the polarization with no Lorentz term at all - correctly, since
the Lorentz factor of a still is one. Three different products under one name that
promises exactly one of them, in code where the neighbouring member is the total
correction.
Rename it prescaling_corr: multiplicative, applied before scaling, therefore not a
scale, and silent about its contents - which is the point, since the contents have
now grown twice. It is also what DIALS calls the same product. The stills refinement
member spelled "1 / rlp" becomes inv_corr, and the comments and usage text that
promised "the Lorentz-polarization factor and nothing else" now say what is actually
there.
The Lorentz term keeps its own name where it is computed, because that name is
correct. The two external spellings are untouched: the CBOR key and the reflection
dataset are a published format, and a reader that meets an unknown key would take
the factor as zero, which both the merge key and the ingest treat as a reflection to
drop - so every reflection would vanish and the run would still exit zero.
No output changes: the merged and unmerged files of two full runs are byte for byte
what the previous binary wrote, four stored files from before the efficiency
correction still re-scale identically, and the reflection datasets of the process
file are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
Offline, a merged rotation run has what a still lacks - a determined point group and an exact
indexed orientation - so the run-level number no longer needs the pessimistic presumed-diad
bound, and it no longer uses the nearest-axis angle either. That 15-deg warning heuristic was
wrong in both directions: an aligned in-plane 2-fold of a dihedral group is repaired by the
principal axis, a cubic group is never severe in any orientation, and a lone diad perpendicular
to the spindle is severe with no axis anywhere near the spindle at all.
The group's proper rotations are applied to the sweep's blind double cone in the crystal's
actual orientation, and what no operator maps onto measured territory is counted, weighted by
each shell's own cone width so the result is a fraction of unique reflections to this run's
resolution limit. Friedel and the improper operators need no separate handling - the cone and
the measured region are both inversion-symmetric. The number is machine-readable on purpose:
SPINDLE_LOST_UNIQUE_FRACTION in the report (0-1, a bare number a pipeline can act on) and
/entry/MX/spindleLostUniqueFraction in the master, with the warning prose only on top of it,
fired when the group recovers less than half the cone's content. REPORT_VERSION stays 7: the
format's own rule is that adding a key does not move it.
This also settles what the nearest-axis keys hedged: with the measured group the mounting is
cleared or convicted exactly, so their documentation now calls them descriptive and points at
the new key for the verdict. Verified against Monte Carlo: P1 loses 2.0% of unique reflections
at theta_max = 15 deg with nothing repaired; a lone diad on or perpendicular to the spindle
repairs nothing; an axis of order >= 3 perpendicular to the spindle repairs everything; 622
with an in-plane diad on the spindle loses nothing; cubic loses nothing in any orientation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
The score was computed and then thrown away - carried on the per-image message but transported
nowhere - so the automation it exists for could not read it. It now flows like bkg_estimate at
every layer: CBOR (per-image key, END-block run mean and per-image array), HDF5 (per-image
/entry/MX/spindleBlindFraction in the data files, the array and spindleBlindFractionMean in the
master), read-back into a re-opened dataset, the scan result, the receiver plots, the REST plot
and scan_result schemas, and the viewer and frontend plot menus.
Absence is load-bearing and every transport keeps it distinguishable from a measured zero: the
CBOR key is simply missing, the HDF5 array holds NaN, and read-back turns NaN back into an
absent optional rather than a value. A pipeline that read 0 where the truth is "no value" would
take exactly the wrong action - a measured 0 says one sweep loses nothing, absence says nobody
could look, and the second must engage the recovery protocol while the first must not. The
round-trip tests pin all three states through CBOR and through a written-and-reopened file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N
* `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands.
* `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion.
* Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants.
* `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing.
* A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed.
* `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing.
* Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences.
* The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to.
* The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after.
* The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution.
* `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have.
* Twinning is no longer reported when the L-test contradicts it.
* The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's.
* `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots.
* The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area.
Reviewed-on: #76
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.
* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.
**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.
---------
Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.
**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.
**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.
**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**
Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release and not recommended for production use (please use rc.111 instead).
* jfjoch_broker: Improve handling of rotation indexing
* jfjoch_broker: More information saved in CBOR end message (WIP)
* jfjoch_writer: Save rotation indexing lattice parameters and Niggli class
* jfjoch_viewer: Remove (for now) primitive cell information
Reviewed-on: #19
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
leonarski_fandClaude Opus 5