77 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	a1b48e9454	Mark unreadable frames in the reprocessing virtual dataset too ... The master's own virtual dataset fills with the error marker, so a source file that cannot be resolved reads as masked rather than as zero counts. The virtual dataset rugnux writes into _process.h5 was left at HDF5's default fill of zero, which is a legitimate count - the same silent failure, one file along. The helper moves above its first user; it has to be set before SetVirtual. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:42:47 +02:00
leonarski_fandClaude Opus 5	583da3c6a0	Derive the rotation width for a chain that was sent ... A chain carried in the END message was written verbatim and stopped there, so AXISNAME_end and the rotation width - which the writer produces when it builds the chain itself - were simply absent. A one-image sweep sent that way imported as a still, since dxtbx prefers AXISNAME_end and only falls back to np.diff; omega_range_average is what DECTRIS-oriented tooling reads for the oscillation. They are derived here rather than added to the wire format: for a constant step they follow from the values, which is every case there is today, so carrying them would cost an array per axis and say nothing new. The step is taken over the endpoints, because the values arrive as floats and a single difference puts that noise straight into the reported width. The test now compares the two routes on the files. It could not have caught this before: it went through the reader, and the reader reads neither of these. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 11:20:34 +02:00
leonarski_fandClaude Opus 5	ce11cade84	Tell a Smargon head position from the spindle by equipment_component ... The reader recognised chi and phi by name. phi is an ordinary spindle name in MX, so a file whose rotation axis is called phi had it read back as a head position as well as the spindle - and writing that experiment out again threw, because the sample chain then tried to create phi twice. In the other direction a still with a head position had chi, its alphabetically first stationary axis, adopted as the goniometer. Both are now settled by the file: the axes jfjoch writes for a Smargon carry equipment_component="smargon", the reader takes a head position only from a tagged axis, and skips tagged axes when looking for the spindle. NXmx defines equipment_component as an identifier of the component of the equipment a transformation belongs to, which is what this is; there is no "equipment" attribute in NeXus at all. Adds HDF5Object::AttrExists, since the tag is absent on every file from anywhere else. The two tests assert on the written file - the axis length and the attribute - because the reader cannot see either: it does not look at a shape, and it did not look at the tag. That is the same gap that let the one-image shape through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 10:49:47 +02:00
leonarski_fandClaude Opus 5	8dd3ee6576	Give the Smargon axes one entry per image ... A reader takes the number of images from the innermost axis of the sample chain when no axis varies, and chi/phi are innermost whenever they are present. Written as scalars, a still or a grid scan with a recorded head position imported as ONE image however many were collected - dxtbx falls back to nxsample.depends_on and takes num_images = len(scan_axis). NXmx has no attribute that would say otherwise: there is no "equipment", and equipment_component identifies a rigid assembly ("detector_arm", "detector_module"), which dxtbx reads only for the detector module hierarchy. The axis length is what carries the image count. Both writer paths are fixed, and the goniometer in BuildTransformationChain now takes its container whenever the image count is known, as the writer already did - a stationary spindle sent over CBOR had the same one-image shape. Verified against DIALS 3.27: same file, chi/phi as scalars imports as 1 image, as per-image arrays as 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 10:43:26 +02:00
leonarski_fandClaude Opus 5	e4edcd6fa9	Carry the sample transformation chain as DetectorTransformation ... Replaces the start-message TransformationAxis of the previous commit, which was the wrong shape in two ways. DetectorTransformation (common/) mirrors a NeXus NXtransformations axis and holds nothing else: name, type, units, vector, offset, depends_on and the positions themselves. Deliberately without cleverness - the values are either a single number for an axis that does not move or one per image, and nothing derives a position from a start and an increment. That is the point: a producer will later want to report where a stage actually WENT rather than where it was told to go, and a structure that stores start+increment cannot express that. A million images cost 4 MB per axis, which is not a reason to be clever. Hence also the move to the END message: measured positions are only known once the run is over. And hence no metadata version bump, which the previous commit did make. The chain is optional; when it is absent the writer builds the identical chain from the start message, exactly as before. Nothing on the wire changes for a producer that does not send it, so a broker and a writer of different releases still interwork - the constraint the previous version stated is withdrawn. The writer transcribes a chain it is given, without recomputing an angle, which is what makes measured positions possible end to end. JFJochReader_TransformationChain_SentAndBuilt writes the same run both ways and checks the two files read back the same, chi/phi included. CBORSerialize_End_Transformations covers the wire 1:1, asserting the order survives and that a moving axis keeps one value per image while a stationary one keeps a single value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 23:36:50 +02:00
leonarski_fandClaude Opus 5	7efcf631de	Write module_offset as a float, and declare offset_units ... Two latent traps in the NXtransformations attributes, both inert today and both wrong the moment they are not. module_offset was an int32 with @vector = (0,0,0). NXmx types the field NX_FLOAT, and a translation needs a unit vector to be well formed - the direction of a zero-magnitude translation is arbitrary, not absent. Now a float with (0,0,1); the transformation it describes is unchanged, since the magnitude is still zero. Every transformation that carries an @offset wrote it without @offset_units, and every caller passed an empty string. A reader then falls back to the axis's own `units` - which on a rotation axis is degrees - and converts a length from degrees to millimetres. nxmx only performs that conversion when the offset is non-zero, so ours have never triggered it, but the first non-zero offset on a goniometer axis would. The helper now always declares it, rather than leaving it to a caller to remember. Measured after the change: module_offset is H5T_IEEE_F32LE, a rotation axis carries offset_units "m" beside units "deg", dials.import still reads the file, and the tilted-geometry cross-check is unchanged at 1.6e-6 mm. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 23:14:40 +02:00
leonarski_fandClaude Opus 5	d2f57975e8	Record whether the image is mirrored in Y, and label the .poni orientation ... Which way the detector's rows run was decided once, in the module assembly, and never stated again: not on the wire, not in the file, nowhere a consumer could read it. mirror_y was consumed inside the DetectorGeometryModular constructor and discarded. It is now a declared property of the detector setup, carried into the start message, written to HDF5 under detectorSpecific, and read back. Absence means true, which is the MX convention and the only thing Jungfraujoch has ever produced. Deliberately a boolean and not a corner enum: the assembled image can only be flipped in Y, so a four-corner value would encode states that cannot occur. DECTRIS stream2 has no field for this - checked against the specification - so the key is new rather than an extension of theirs, and a consumer that does not know it skips it and behaves exactly as before. The .poni file gains pyFAI's orientation. Without it pyFAI applies its own default, 3 (bottom left), and believes increasing row means physically upwards. The numbers still agreed - a mirror preserves 2theta, so radial integration was never affected - but the azimuth came out with the opposite sense, which matters for cake and sector integration. Declaring orientation 2 is not a one-line addition: it re-anchors Poni1 to the top edge and reverses rot2 and rot3, a row flip being improper. Measured against pyFAI 2026.5.0 by searching all four orientations, both Poni1 anchorings and all eight sign combinations: exactly two combinations reproduce the lab position DiffractionGeometry computes to 1.4e-17 m - the unlabelled form written before, and (orientation 2, Poni1 = height-1-beam_y, +rot1/+rot2/-rot3), which is now written. Calibration_PoniFileAxisConvention pins it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:54:50 +02:00
leonarski_fandClaude Opus 5	5e05515165	Write the grid stage as a base stage, not head-mounted ... The grid translations went innermost, i.e. mounted on the head, so a grid position turned with the spindle. At SLS the grid is an Aerotech xyz that the spindle is mounted ON, so the mounting order is base -> grid -> omega -> chi -> phi -> sample and a grid position is independent of omega. Identical to the previous chain at omega = 0, which is every grid scan collected so far, and correct rather than incorrect when it is not. A head-mounted stage exists too - the Smargon translates, and that is what helical uses - and would sit on the other side of omega. Only the base stage is modelled for now, which is the one actually used; the comment says so. Measured: dials.import reads a master with the new chain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:41:51 +02:00
leonarski_fandClaude Opus 5	cc334c5c55	Name the right rotation in the NXmx frame comment ... The comment said internal-to-McStas is 180 degrees about x. It is 180 degrees about z: internal (a,b,c) maps to McStas (-a,-b,+c), which is what makes the written vectors correct - internal +y becomes (0,-1,0), the -x of Rx(-rot2) becomes (1,0,0), and the -z of Rz(-rot3) is unchanged. 180 degrees about x is the internal-to-imgCIF relation, one step further on, and it is the frame the verification was done in - which is why the vectors are right and only the prose was wrong. Re-measured after the change: still 1.6e-6 mm over nine tilt settings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:33:30 +02:00
leonarski_fandClaude Opus 5	7039aa4e45	Stop treating a goniometer axis and a grid scan as alternatives ... They are not alternatives: a grid is usually collected at a particular head position, so an axis and a grid describe different parts of the same setup. The exclusion was enforced independently in four places - the API converter, the CBOR serializer, the writer and the reader - and each silently dropped the grid scan when an axis was present. Nothing warned. The writer now builds one chain from the base outwards, spindle -> chi -> phi -> helical -> grid translations, instead of two branches. NXmx applies the deepest dependency first, so the sample ends up innermost, which is what it physically is: the grid stage rides on the head and the head rides on the spindle. The grid translations consequently move inside the rotation - identical to before at omega = 0, and right rather than wrong when it is not. A grid scan with no axis at all now writes a stationary omega. NXmx has no way to say "there is no rotation", and a sample chain of translations alone is not something readers accept: dxtbx raises outright on it, so every grid-scan master we have written so far cannot be opened by DIALS. Measured on a file matching the new chain: dials.import reads it. At 0 degrees the rotation is the identity whatever the axis points along, so the conventional vector carries no geometric claim - it only has to be well formed. The API change is deliberately not breaking: no field changes type or cardinality, only the prose saying the two were exclusive, and a request that set both used to lose one silently and now does not. JFJochReader_GridScan asserted the absence of a goniometer; it now asserts the axis is present and stationary, which is the contract that matters - a grid scan must not read back as a sweep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:25:08 +02:00
leonarski_fandClaude Opus 5	259b43154e	Writer: refuse a mistyped stream, and mark unreadable VDS frames ... Two ways the written files could misdescribe themselves without anyone noticing. The pixel format is stated twice and the two were never compared: each image carries its own type as a CBOR tag, which is what the data files are written with, while the master is typed from the start message. A stream whose header contradicts its images produced data files of one type under a master declaring another, and with NXmxVDS, HDF5 then converts silently on every read. HDF5DataFile::CreateFile now checks the two agree and refuses the run otherwise - the point where the values first meet, so it covers every path into the writer. Two test fixtures were relying on exactly that inconsistency. The HDF5 writer tests wrote uint16 buffers under a JUNGFRAU experiment, which converts to photon counts by default and so declares int16; they never read the pixels back, so it went unnoticed. The receiver-lite tests feed frames from compression_benchmark.h5, which really are signed int16, through a DECTRIS experiment, which declares unsigned by default - the same class of bug the pixel_signed propagation fixed on the live path. Both now declare what they send. Second: a virtual dataset whose source file is absent reads as the fill value, and HDF5 defaults that to zero, so a data file that was not copied alongside the master is indistinguishable from frames of genuine zero counts. Measured with DIALS on a four-file set with one file removed: 25 frames of pure zeros, no error and no warning. The image VDS is now filled with the error marker instead, which sits outside underload_value..saturation_value, so a reader masks those frames. Same measurement after the change: -32768 throughout, which DIALS excludes. Only the images ask for a fill value; the per-image metadata datasets keep the default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 21:44:04 +02:00
leonarski_fandClaude Opus 5	03e481ff2a	Export the detector tilt correctly in the NXmx transformation chain ... Three independent errors, all invisible while the tilt is zero - which it is in every test and every CI file, and which is why this survived. 1. rot1 and rot2 carried each other's axis. Jungfraujoch holds the tilt in the PyFAI PONI convention, poni_rot = Rz(-rot3)*Rx(-rot2)*Ry(+rot1), written in the internal frame (x along increasing column, y along increasing row, z along the beam). NXmx uses McStas, which is that frame turned 180 degrees about x - a proper rotation, NOT a mirror - so rotations about y and z reverse sense and those about x do not. Correct vectors are rot1 (0,-1,0), rot2 (1,0,0), rot3 (0,0,-1); only rot3 was already right. 2. The depends_on chain composed the rotations in the reverse order. A chain applies the deepest dependency first, so rot3 has to sit at the root for the product to be R_rot3*R_rot2*R_rot1. Second-order: it only shows up when two angles are non-zero at once. 3. The tilt pivoted about the wrong point. With translation at the root, a reader takes the panel origin as the unrotated vector and merely reorients the panel, while Jungfraujoch rotates the whole sample->pixel vector including the distance. Moving translation inside the rotations fixes the pivot; this was the largest of the three and is invisible to any test that only checks axes. Verified against DIALS 3.27 by comparing the lab position of nine pixels spread over the detector against poni_rot applied to the sample->pixel vector, over nine tilt settings including combined and mixed-sign angles: max error 1.6e-6 mm, which is float32 rounding of the stored metadata. The shipped encoding gives 40.7 mm - 542 pixels - at a (0.2, 0.3, 0.15) rad tilt. Note when comparing by hand that dxtbx applies a parallax correction in its pixel->mm conversion, which Jungfraujoch does not model; get_pixel_lab_coord is therefore not the right comparison point and shows a ~0.1 mm radius-dependent offset that is not a geometry error. Jungfraujoch's own reader is unaffected: it reads rot1/rot2/rot3 by dataset path and never consults the vectors or the chain, which is why the round trip stayed self-consistent throughout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 19:04:43 +02:00
leonarski_fandClaude Opus 5	77c6d0f4ec	Report the stored image depth in bit_depth_readout, and write underload_value ... NXmx has no field for the depth of the stored image - only bit_depth_readout, "how many bits the electronics record per pixel". The two diverge exactly when summation is used: the readout keeps the detector's native width while the summed image must be wider to hold the sum. Every NXmx reader nonetheless takes bit_depth_readout as the width of the stored pixel. dxtbx ignores the non-standard bit_depth_image entirely for a generic NXmx file, derives its masking markers from bit_depth_readout, and raises "Unsupported integer dtype uint32" for a 32-bit image when the field is absent. Reporting the electronic value there would mislead precisely where it differs. So report the image depth in both fields, and drop the machinery that existed to carry the electronic one for a DECTRIS detector: the SIMPLON read, the DetectorSetup setter, and the receiver-side propagation of a key that the DECTRIS stream2 protocol does not even define. JUNGFRAU and PSI EIGER keep their readout depth, which the FPGA acquisition genuinely needs. Also write NXmx underload_value, the lowest valid value. Without it a reader takes the trusted minimum to be -0x7FFFFFFF, so the error-pixel marker sits inside the trusted range and is consumed as an intensity. Measured with DIALS 3.27 on a written file: trusted_range goes from (-2147483647, 32766) to (-32767, 32766), so the INT16_MIN gap pixels are now masked. Third fix in the same area: JFJochReceiverLite::Configure took the image width from the incoming stream but not the sign, while the image itself is forwarded byte-for-byte. A detector sending int32 was re-declared uint32, and the VDS master was typed unsigned over signed data files. Take pixel_signed from the stream too - it and the width are both carried by the one image_dtype key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 18:15:44 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62	2026-06-17 20:36:24 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	caef26873e	v1.0.0-rc.145 (#55) ... This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144. * **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future. * XDS plugin: Improve performance of VDS reading. * jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD. * jfjoch_writer: Clean-up of HDF5 routines to better handle issues. Reviewed-on: #55	2026-05-06 21:50:02 +02:00
leonarski_f	4a852b4d6b	v1.0.0-rc.134 (#43) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long * jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata * XDS plugin: Add generation of Jungfraujoch plugin for XDS * CI: Add tests with XDS and DIALS (`xia2.ssx`) Reviewed-on: #43	2026-04-09 13:30:47 +02:00
leonarski_f	1ab257af6c	v1.0.0-rc.125 (#32) ... This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-18 16:17:21 +01:00
leonarski_f	31a357fa57	v1.0.0-rc.113 (#19) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_broker: Improve handling of rotation indexing * jfjoch_broker: More information saved in CBOR end message (WIP) * jfjoch_writer: Save rotation indexing lattice parameters and Niggli class * jfjoch_viewer: Remove (for now) primitive cell information Reviewed-on: #19 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-02 09:29:22 +01:00
leonarski_f	95acf3aba3	v1.0.0-rc.104 (#9) ... This is an UNSTABLE release. jfjoch_writer: Fix and improve the way grid scan geometry is saved (non-NXmx extension makes it way easier) jfjoch_viewer: Display grid scan results in 2D (work in progress) jfjoch_viewer: Improve auto-scaling on start of images (work in progress) jfjoch_viewer: Add B-factor and resolution estimate to the dataset info plots Reviewed-on: #9 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-19 17:28:10 +01:00
leonarski_f	8b356a7001	v1.0.0-rc.96 (#1) ... This is an UNSTABLE release. * Fixes in CI pipeline * jfjoch_broker: Remove PNG preview, no dependency on libpng * jfjoch_writer: Fix UTC timestamp being generated wrong (mix between milli- and microseconds) * jfjoch_viewer: Show data collection time in dataset tooltip * jfjoch_viewer: Allow to choose the calibrant (presets for LaB6 and silver behenate) * jfjoch_viewer: Auto foreground value * Use external libjpeg-turbo and libtiff: simpler build stack, these are built and linked statically in automated Docker builds * Remove OpenBLAS dependency Reviewed-on: #1 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-02 13:45:57 +01:00
leonarski_f	061152279c	v1.0.0-rc.91	2025-10-20 20:43:44 +02:00
leonarski_f	5bb8898a0e	v1.0.0-rc.85	2025-09-24 22:52:31 +02:00
leonarski_f	5d9d2de4a4	v1.0.0-rc.81	2025-09-21 19:27:51 +02:00
leonarski_f	caf2a9de4f	v1.0.0-rc.79	2025-09-15 21:38:58 +02:00
leonarski_f	20cd8dc4ac	v1.0.0-rc.76	2025-09-11 14:37:16 +02:00
leonarski_f	c67337cfe1	v1.0.0-rc.72	2025-09-08 20:28:59 +02:00
leonarski_f	bb32f27635	v1.0.0-rc.70	2025-08-27 06:21:10 +02:00
leonarski_f	fa95858008	v1.0.0-rc.69	2025-08-17 21:21:20 +02:00
leonarski_f	7ad4a3337b	v1.0.0-rc.58	2025-07-03 15:10:04 +02:00
leonarski_f	99de57fe6f	1.0.0-rc.54	2025-06-29 10:56:38 +02:00
leonarski_f	c41858f80d	v1.0.0-rc.53	2025-06-24 16:43:47 +02:00
leonarski_f	0d1b388f4d	v1.0.0-rc.50	2025-06-20 09:52:26 +02:00
leonarski_f	fb87f376c8	v1.0.0-rc49	2025-06-18 15:19:18 +02:00
leonarski_f	a9c7d23b9e	v1.0.0-rc.42	2025-06-11 19:53:33 +02:00
leonarski_f	41a3e671f4	v1.0.0-rc.41	2025-06-10 18:14:04 +02:00
leonarski_f	53c90ee5d8	v1.0.0-rc.40	2025-05-28 18:49:27 +02:00
leonarski_f	b245967df3	v1.0.0-rc.38	2025-05-12 14:17:24 +02:00
leonarski_f	040cf08386	v1.0.0-rc.36	2025-05-05 19:32:22 +02:00
leonarski_f	9bec33290c	v1.0.0-rc.35	2025-04-22 14:42:14 +02:00
leonarski_f	b0607ab3ca	v1.0.0-rc.34	2025-04-14 11:52:06 +02:00
leonarski_f	ddf4c75645	v1.0.0-rc.31	2025-03-02 13:15:28 +01:00
leonarski_f	50ef79e621	version 1.0.0-rc.26 quick fixes	2024-11-27 11:46:58 +01:00