70 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	cb1d88dd4d	Merge branch 'worktree-agent-a4c58d76a0be29dd6' into grid-scan-integration ... # Conflicts: # rugnux/rugnux_cli.cpp	2026-09-08 07:55:31 +02:00
leonarski_fandClaude Opus 5	748d14d2cd	file format: the ice ring ratio is stored and sent under its own name, and the retired one is still read ... Completes the rename the C++ identifiers already carried. Producers now emit ice_ring_ratio / ice_ring_ratio_mean over CBOR and write /entry/MX/iceRingRatio and iceRingRatioMean; both readers accept the retired ice_ring_score / iceRingScore spellings as well, so nothing that exists on disk or on the wire stops being readable. The REST plot_type gains ice_ring_ratio and KEEPS ice_ring_score, both mapping to the same plot, so no client breaks. --ice-min-score is deliberately unchanged: it is a threshold on the ratio, it is the one surface a user types, and this program has 91 long options and no aliases. The fallback is not a formality. rugnux --mode scale reads the stored per-image value to reproduce the ice gate the writing run applied, and an unread dataset does not fail loudly - ice_n == 0 sends the gate to its fail-safe branch, "ice present". Demonstrated on a clean crystal whose stored file uses the old names: with the fallback the gate reads 1.06 against a 1.50 threshold and skips ice handling; with the fallback removed the same file has 272056 of 1032493 reflections (26%) excluded from the scale fit, and ISa, R_meas, I/sigma, SIGANO and both error-model terms all move. That is a silent change to merged intensities on files already written, which is why the two new CBOR test cases were each checked to FAIL when their fallback is taken out rather than merely to pass. Verified: same binary on an old-name and a new-name copy of one file gives byte-identical .hkl, .mtz and unmerged .mtz and an identical report; a file written after the rename scales to the same bytes again; the master and the data-file read paths were each exercised with both spellings; and the live API serves the same plot under either name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:42:23 +02:00
leonarski_fandClaude Opus 5	0f00b76a9a	image analysis: the ice score takes the pipeline's own band width, and the ice quantities are named for what kind of number they are ... Two things, both about telling one ice quantity from another. The ice score's spot channel had its own band half-width of 0.02 A^-1 while the spot finder marks ice rings at 0.03 (ice_ring_width_Q_recipA). The 0.02 was justified by a 5 pp specificity gain measured on the PYTHON PROTOTYPE, which used a fitted beam centre and a mask-derived coverage table; the shipped port, which takes the geometry's centre and the azimuthal profile's own live pixel count, does not reproduce it. Measured over the corpus by truth class rather than by directory label, at 0.02 vs 0.03 on the combined score: ice loops 62.13/62.19%, _icy protein 89.03/89.79%, _clean protein 16.51/16.31%, water 17.19/20.03%. The widths are indistinguishable except on water, where one of the four loops is independently known to carry a full hexagonal pattern. So the width is now a parameter and the pipeline's own value is passed in - one band width, not two. The 0.012 tolerance in the radial channel is NOT a second band width, and is renamed CENTRE_SMEAR_Q to say so: it is how far either side the channel looks for the bin a mis-set beam centre moved the ring to. The rest is naming. Three kinds of number were all called score, or built from things called count, and a reader could not tell from the name whether 1 meant "none" or "certain" - which are opposite. The convention, now stated in docs/CPU_DATA_ANALYSIS.md: *_score is bounded [0,1] and 1 is certainty, *_ratio is unbounded and 1 is nothing, *_count is a count. The C++ identifiers for the ice ring ratio follow it (ice_ring_score -> ice_ring_ratio, GetIceRingScore -> GetIceRingRatio, PlotType::IceRingScore -> IceRingRatio), and the local in the scaling gate that shadowed the new ice_score while meaning the ring ratio is renamed with them. Nothing outside the source moved: the CBOR keys ice_ring_score and ice_ring_score_mean, the datasets /entry/MX/iceRingScore and iceRingScoreMean, the ice_ring_score plot type and the --ice-min-score flag are all unchanged, and were checked to be after the rename. Renaming those changes stored files, the stream format, the REST API and a CLI flag, and is a separate decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 07:19:03 +02:00
leonarski_fandClaude Opus 5	550f7c7d7b	scan result: a grid scan's crystals and a calibration's geometry reach the stream, the file and the API ... A grid scan and a powder calibration each produce an answer that is not a per-image number, and neither had anywhere to go. ScanResult gains two top-level optionals, grid and calibration, in the same idiom the three rotation_* members already use: a mode's payload is present when that mode ran and absent otherwise. images stays required, so every existing /result/scan client - the python client and the beamline scripts on it - is untouched. No oneOf or discriminator. The spec contains none today, so polymorphism is unexercised across all three generators at once, and the only thing it would buy is making "exactly one payload" structurally true rather than true by convention. powder_calibration_output has been fully specified and referenced by nothing since it was written; the calibration member reuses it. The crystal list travels the same three hops a per-image quantity does. In the CBOR end message grid_crystals is an array of maps keyed by field name - the shape a spot already uses - so a reader that does not know a field skips it and the record survives a field being added; the deserializer steps over an unknown key rather than reading its payload as the next one. In the HDF5 master it is parallel 1-D datasets under /entry/MX/crystals, one per field, because crystals are numbered rather than named and a group per crystal would put the index in the path. The reader reads them back so a stored raster re-opens with what it found. Nothing caps the list at one. A raster can find several crystals and the ordering carries the answer: sorted by score, best first. angle_deg is an axis, not a direction. It lies in [0, 180) and wraps, so 179 and 0 are adjacent and two of these do not average arithmetically - a pair of nearly parallel needles averaged across the wrap comes out as a right angle. That is stated at every layer it passes through, since it is the kind of field a consumer will reach for without reading the producer. The producer side is not wired yet: nothing fills ScanResult::grid or EndMessage::grid_crystals here. The seam is receiver/JFJochReceiver.cpp, where rotation_lattice is filled in EndDataCollection and GetFinalStatistics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:17:10 +02:00
leonarski_fandClaude Opus 5	a7d3ada3ab	analysis: what runs over the images is one stated mode, shared by broker, rugnux and viewer ... Until now nothing in the tree said what analysis a run performed. The answer was composed at each site out of four independent scalars - the detector type, two separate "spot finding off" switches, an indexing flag and a rotation flag - so what was configured and what actually ran were different things, and no single place could be read to find out which. AnalysisMode {None, MXRotation, MXStills, Azint, Grid, PowderCalibration} is that statement, in common/ because all three programs configure the DiffractionExperiment that carries it. AnalysisSettings sits on the experiment beside IndexingSettings, outside the DatasetSettings member, which is the one thing a /start replaces wholesale - so the mode is persistent by construction rather than by a rule someone has to remember. The mode does not label a run, it decides it. AnalysisModeStages() is a table - modes as rows, pipeline stages as columns - and every gate reads that table instead of testing the mode: spot finding in DiffractionExperiment::IsSpotFindingEnabled, indexing (and with it prediction and integration, which never run without a lattice) in one gate inside IndexAndRefine that serves all three front ends, azimuthal integration where the CPU engine is built. Two rows carry a judgement worth reviewing: powder calibration keeps spot finding, because --calibration spots fits the pooled spots; grid does not index, because a raster is thousands of frames and the per-image scoring it ranks on deliberately avoids an indexer that fires on ice. There is deliberately no Auto value. GetIndexingAlgorithm() resolves Auto at read time, which is exactly why an indexing setting cannot be read back off the configuration; removing that kind of implicitness is the point here, so the mode getter stays a plain accessor. MXStills is the default because None would silently switch analysis off on every deployment whose configuration predates the field. Rotation MX is absent from the OpenAPI schema rather than present and refused: jfjoch_broker has no rotation analysis path, so the REST and configuration-file routes cannot express it at all. The shared enum can still carry the value from elsewhere, so CheckAnalysisSettingsOnline refuses it on both routes with a message naming rugnux. A sweep collected under an MX mode is not refused - collecting rotation data online is normal and live spot counts are useful - but it is said out loud in the log, since the mistake worth preventing is the silence about what was done to it, not the acquisition. Powder calibration forces azimuthal integration onto the CPU and supplies 32 sectors where fewer than four were asked for. The FPGA integration core holds 2048 bins in total, so 32 sectors would leave 64 q bins - far too coarse to fit a ring. Frame rate is what this costs and a calibration exposure does not need it. The two existing "no analysis" switches, per-dataset dataset_settings.spot_finding and persistent SpotFindingSettings::enable, are interfaces in too many places to remove now. They are marked deprecated in the schema and in both headers, and the mode takes precedence over them: a mode that analyses no spots wins outright, while under a mode that does find spots they remain the finer control. The precedence is written where it is enforced. rugnux's ProcessMode is gone, replaced by the shared enum; RugnuxMode stays as the CLI spelling layer and no existing spelling changes. --mode gains mx_rotation and mx_stills, which are spellings of -R and --force-still rather than new switches; plain mx still chooses between them from the goniometer. scale keeps no shared counterpart, since it runs no analysis over images at all. The mode reaches the CBOR start message and /entry/MX/analysis_mode in the HDF5 master, so a written file records which analysis produced it. It is read back as provenance only - what a stored file was produced by is not what the next run should do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:16:26 +02:00
leonarski_fandClaude Opus 5	14e2bf3c0f	Merge rc167 into the detection-score branch ... Both lanes added a per-image scalar to the same eleven files, so every conflict was two additions competing for one line. All were resolved by keeping both, with three that needed more than that: - ScanResultGenerator: rc167 changed the per-image float vectors to resize(n, NAN) so a frame that never arrived does not read back as a real 0. v_protein_score and v_ice_score are exactly that case - 0 is a real answer ("nothing detected here") - so they take the NAN default too. - HDF5MetadataSource: rc167 established that NaN in a stored per-image array means "no value" and the optional must come back absent. The two detection scores now follow it, which they did not before the merge; without the guard a missing score would come back as a NaN that a threshold would silently compare against. - CBORTest: designated initialisers must follow member declaration order, so spindle_blind_fraction precedes the two scores in the DataMessage aggregate. Verified after the merge that every CBOR key that is encoded is also decoded (198 encoded keys, one intentional exception: the "type" discriminator), that both lanes' fields survive in the writer, the reader, the plots and the API, and that a stored file still round-trips. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-08 00:13:51 +02:00
leonarski_fandClaude Opus 5	62ca2b24f6	image analysis: two per-image detection scores say whether there is protein and whether there is ice ... proteinScore and iceScore are saturating scores in [0,1] that answer presence, not quality: a superb crystal and a barely-diffracting one both read near 1, and neither the spot count nor the resolution enters either of them as a term. iceRingScore already reports a magnitude - a ratio, unbounded - which is not a number that can be thresholded; these are. The protein score counts distinct d SHELLS above 5 A rather than spots, so a parasitic ring in the low-resolution band cannot accumulate evidence, and weights each spot against the frame's own median so a scattering of the weakest detections cannot fill a shell either. The ice score carries two channels and takes the stronger: a radial one over the azimuthal profile, which runs the hexagonal and the CUBIC phase as separate hypotheses and decides between them at the end (flash-cooled loops show cubic or stacking-disordered ice at least as often as hexagonal, the two share only three lines, and dropping the cubic hypothesis costs about 5 pp on iced loops), and a spot one that reads an excess on the ice radii against the same band slid to every ice-free offset, which is what catches ice arriving as discrete crystallites and leaving the radial profile flat. Both read d out of the geometry, so both move with a beam-centre error; the centre is not fitted here, and the one they were computed with is written beside them as scoreBeamCenterX/Y so a later rescoring can tell an algorithm disagreement from a geometry one. Ported from validated prototypes and checked against them frame by frame on stored data: mean absolute difference 2.7e-5 (protein), 1.3e-8 (ice radial) and 3.4e-4 (ice spots). On a 41-loop battery the protein score reaches 98.4% of confirmed-protein frames and 0.00% of water frames, and finds no cluster on any water or ice raster. Cost is 0.01 ms/frame for the protein score and 0.08-0.32 ms/frame for the ice score. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-07 23:59:46 +02:00
leonarski_fandClaude Opus 5	5b8ce26c83	integration: the flight path between the sample and the detector is corrected for, and named ... A reflection arriving at an angle to the detector normal crosses D/cos(alpha) of whatever lies between the sample and the sensor, not D, so it is attenuated more than one arriving head-on and reads low. That is the same geometry as the sensor crossing already corrected here and the opposite sign, and it was missing. The factor is exp(D/L*(1/cos(alpha)-1)) from the NIST attenuation coefficient of the medium, the stated distance and the stated wavelength. Nothing in it is fitted, and it is not justified by any measured amplitude: the flight path and the sensor crossing are collinear to better than 0.998 over the angular range any single experiment samples, so no fit of one can be evidence for the other. It is the tabulated absorption of a known thickness of a known material over a known path. The medium cannot be detected. No field of the NXmx application definition describes it, none of the masters this program reads carries one, and it cannot be inferred from the implied transmission either - in this corpus a station confirmed to use helium sits at 51% implied air transmission and one confirmed to use air at 63%, so any rule separating them is a threshold fitted between two points. It is therefore assumed, stated, and overridable: --flight-path air|helium|vacuum, defaulting to air. Helium is its own material rather than an alias for vacuum, attenuating about a six hundredth of air rather than nothing. On an untilted detector the correction is a function of resolution alone, so its entire effect on merged data is a shift in the Wilson B - which is what the report now prints beside the assumption, accurate to better than a tenth of an angstrom squared against measurement from 0.05 up to 28. Where that shift is large the report warns, because a wrong medium is then the largest number in the run: applied to data from the confirmed helium station it returns a B of 14 A^2 at 3.0 A resolution, which is not a value a crystal can have. The corpus contains its own control. One crystal, one station, three collections a quarter of an hour apart at falling energy through the same air: corrected, the Wilson B rises monotonically with the dose, as it must. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 17:55:43 +02:00
leonarski_fandClaude Opus 5	6140de7249	rugnux: the unmerged MTZ carries the sensor efficiency in its own column, not inside LP ... LP is a column other programs read and un-apply. It is documented as the Lorentz-polarization factor, and until the sensor efficiency was folded into the same product that is what it held. Measured on our own unmerged file, LP spanned a factor of 1.1343 across the detector where pure L/P spans nothing of the sort - the excess is the efficiency, 13% end to end at 13 keV and 34% at 18 keV. Both reference implementations keep it out. Recomputing pure L/P from a stored XDS file's own geometry over its 124k reflections leaves RLP flat to 0.1% from 8.6 to 33.7 degrees, where a folded-in efficiency would have shown a 7% trend - and XDS has the sensor numbers in hand. DIALS fills LP from lorentz and polarization alone and writes QE as a separate column, even a column of ones. Split them: LP is L/P again, QE is the efficiency, and the intensity is unchanged, so a reader that un-applies LP recovers what it expects and one that wants raw counts divides by LP and multiplies by QE. Only the unmerged file moves - every other column is bit-identical and the file grows by exactly one float per reflection plus one header record. The merged files are byte for byte what they were. The process file gains an optional qe dataset beside the existing one rather than changing what that one means, so a file written before the efficiency existed still loads, and reads back as a correction of exactly one - which is what it was. It is stored rather than recomputed on read because the writer has no geometry to recompute it from, and because recomputing would have written a radial trend into every stored file that never had one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 14:29:45 +02:00
leonarski_fandClaude Opus 5	db9cc9106f	rugnux: the per-reflection correction factor is named for what it is, not for what it once held ... The factor multiplied into each integrated intensity was called rlp, for reciprocal Lorentz-polarization, and until this week that is all it held. It now also carries the sensor efficiency at the angle the beam arrives, and on the stills path it holds that efficiency and the polarization with no Lorentz term at all - correctly, since the Lorentz factor of a still is one. Three different products under one name that promises exactly one of them, in code where the neighbouring member is the total correction. Rename it prescaling_corr: multiplicative, applied before scaling, therefore not a scale, and silent about its contents - which is the point, since the contents have now grown twice. It is also what DIALS calls the same product. The stills refinement member spelled "1 / rlp" becomes inv_corr, and the comments and usage text that promised "the Lorentz-polarization factor and nothing else" now say what is actually there. The Lorentz term keeps its own name where it is computed, because that name is correct. The two external spellings are untouched: the CBOR key and the reflection dataset are a published format, and a reader that meets an unknown key would take the factor as zero, which both the merge key and the ingest treat as a reflection to drop - so every reflection would vanish and the run would still exit zero. No output changes: the merged and unmerged files of two full runs are byte for byte what the previous binary wrote, four stored files from before the efficiency correction still re-scale identically, and the reflection datasets of the process file are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-05 13:41:16 +02:00
leonarski_fandClaude Opus 5	307987c865	rugnux: the mounting's cost is computed exactly from the measured group, not guessed from an angle ... Offline, a merged rotation run has what a still lacks - a determined point group and an exact indexed orientation - so the run-level number no longer needs the pessimistic presumed-diad bound, and it no longer uses the nearest-axis angle either. That 15-deg warning heuristic was wrong in both directions: an aligned in-plane 2-fold of a dihedral group is repaired by the principal axis, a cubic group is never severe in any orientation, and a lone diad perpendicular to the spindle is severe with no axis anywhere near the spindle at all. The group's proper rotations are applied to the sweep's blind double cone in the crystal's actual orientation, and what no operator maps onto measured territory is counted, weighted by each shell's own cone width so the result is a fraction of unique reflections to this run's resolution limit. Friedel and the improper operators need no separate handling - the cone and the measured region are both inversion-symmetric. The number is machine-readable on purpose: SPINDLE_LOST_UNIQUE_FRACTION in the report (0-1, a bare number a pipeline can act on) and /entry/MX/spindleLostUniqueFraction in the master, with the warning prose only on top of it, fired when the group recovers less than half the cone's content. REPORT_VERSION stays 7: the format's own rule is that adding a key does not move it. This also settles what the nearest-axis keys hedged: with the measured group the mounting is cleared or convicted exactly, so their documentation now calls them descriptive and points at the new key for the verdict. Verified against Monte Carlo: P1 loses 2.0% of unique reflections at theta_max = 15 deg with nothing repaired; a lone diad on or perpendicular to the spindle repairs nothing; an axis of order >= 3 perpendicular to the spindle repairs everything; 622 with an in-plane diad on the spindle loses nothing; cubic loses nothing in any orientation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_fandClaude Opus 5	26a82ddc72	spindle: the per-image severity travels the whole data path, and absence travels with it ... The score was computed and then thrown away - carried on the per-image message but transported nowhere - so the automation it exists for could not read it. It now flows like bkg_estimate at every layer: CBOR (per-image key, END-block run mean and per-image array), HDF5 (per-image /entry/MX/spindleBlindFraction in the data files, the array and spindleBlindFractionMean in the master), read-back into a re-opened dataset, the scan result, the receiver plots, the REST plot and scan_result schemas, and the viewer and frontend plot menus. Absence is load-bearing and every transport keeps it distinguishable from a measured zero: the CBOR key is simply missing, the HDF5 array holds NaN, and read-back turns NaN back into an absent optional rather than a value. A pipeline that read 0 where the truth is "no value" would take exactly the wrong action - a measured 0 says one sweep loses nothing, absence says nobody could look, and the second must engage the recovery protocol while the first must not. The round-trip tests pin all three states through CBOR and through a written-and-reopened file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-04 10:59:05 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	4878318c27	v1.0.0-rc.139 (#49) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start` * jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start' * jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time * jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received * jfjoch_broker: Use nanoseconds internally for frame/image/readout time * jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished) * jfojch_broker: Improve profiling of different data analysis steps * jfjoch_broker: Record integration reflection count * jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds) * jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked * jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets **Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time. Reviewed-on: #49	2026-04-29 09:50:50 +02:00
leonarski_f	c981e1b91c	v1.0.0-rc.137 (#46) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Better track time for each operation in the processing stack * jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress) * jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress) * jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0) Reviewed-on: #46	2026-04-25 19:59:21 +02:00
leonarski_f	c1c170112c	v1.0.0-rc.136 (#45) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress). Reviewed-on: #45	2026-04-20 11:54:33 +02:00
leonarski_f	6133da1377	v1.0.0-rc.133 ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Use httplib for HTTP server instead of Pistache * jfjoch_broker: Drop OpenSSL support * jfjoch_broker: Base work for multi-lattice support in the future * Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib) Reviewed-on: #41	2026-03-26 20:50:33 +01:00
leonarski_f	f3e0a15d26	v1.0.0-rc.127 (#34) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Default EIGER readout time is 20 microseconds * jfjoch_broker: Multiple improvements regarding performance * jfjoch_broker: Image buffer allows to track frames in preparation and sending * jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer * jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets * jfjoch_writer: Fixes regarding properly closing files in long data collections * jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code Reviewed-on: #34	2026-03-02 15:57:12 +01:00
leonarski_f	1ab257af6c	v1.0.0-rc.125 (#32) ... This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-18 16:17:21 +01:00
leonarski_f	07fe4dd3bb	v1.0.0-rc.124 (#31) ... This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-01 13:29:33 +01:00
leonarski_f	27496b8207	v1.0.0-rc.122 (#29) ... This is an UNSTABLE release. * jfjoch_broker: Add thresholding to prefer shorter vectors after FFT * jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress) * jfjoch_viewer: Display file opening errors * jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available Reviewed-on: #29 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-16 15:27:40 +01:00
leonarski_f	31a357fa57	v1.0.0-rc.113 (#19) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_broker: Improve handling of rotation indexing * jfjoch_broker: More information saved in CBOR end message (WIP) * jfjoch_writer: Save rotation indexing lattice parameters and Niggli class * jfjoch_viewer: Remove (for now) primitive cell information Reviewed-on: #19 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-02 09:29:22 +01:00
leonarski_f	06949caf1a	v1.0.0-rc.112 (#18) ... This is an UNSTABLE release and not recommended for production use (please use rc.11 instead). * jfjoch_broker: Experimental rotation (3D) indexing * jfjoch_broker: Minor fix to error in optimizer potentially returning NaN values Reviewed-on: #18 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-30 17:39:22 +01:00
leonarski_f	061152279c	v1.0.0-rc.91	2025-10-20 20:43:44 +02:00
leonarski_f	5d9d2de4a4	v1.0.0-rc.81	2025-09-21 19:27:51 +02:00
leonarski_f	caf2a9de4f	v1.0.0-rc.79	2025-09-15 21:38:58 +02:00
leonarski_f	c67337cfe1	v1.0.0-rc.72	2025-09-08 20:28:59 +02:00
leonarski_f	bb32f27635	v1.0.0-rc.70	2025-08-27 06:21:10 +02:00
leonarski_f	fa95858008	v1.0.0-rc.69	2025-08-17 21:21:20 +02:00
leonarski_f	20973792e4	v1.0.0-rc.68	2025-08-16 19:59:27 +02:00
leonarski_f	4da526c51b	v1.0.0-rc.63	2025-07-09 14:55:28 +02:00
leonarski_f	0b5f28f50d	v1.0.0-rc.62	2025-07-08 20:19:59 +02:00
leonarski_f	e4ec9fb38f	v1.0.0-rc.61	2025-07-08 07:16:59 +02:00
leonarski_f	fbaf15aa07	v1.0.0-rc.60	2025-07-05 18:52:42 +02:00
leonarski_f	0d1b388f4d	v1.0.0-rc.50	2025-06-20 09:52:26 +02:00
leonarski_f	fb87f376c8	v1.0.0-rc49	2025-06-18 15:19:18 +02:00
leonarski_f	a9c7d23b9e	v1.0.0-rc.42	2025-06-11 19:53:33 +02:00
leonarski_f	41a3e671f4	v1.0.0-rc.41	2025-06-10 18:14:04 +02:00
leonarski_f	53c90ee5d8	v1.0.0-rc.40	2025-05-28 18:49:27 +02:00
leonarski_f	040cf08386	v1.0.0-rc.36	2025-05-05 19:32:22 +02:00
leonarski_f	9bec33290c	v1.0.0-rc.35	2025-04-22 14:42:14 +02:00
leonarski_f	b0607ab3ca	v1.0.0-rc.34	2025-04-14 11:52:06 +02:00
leonarski_f	ddf4c75645	v1.0.0-rc.31	2025-03-02 13:15:28 +01:00
leonarski_f	b3e745a8dd	version 1.0.0-rc.26	2024-11-26 16:04:38 +01:00