40 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	54f2de31bb	grid scan: say how the stationary spindle angle is stated, and pin it ... A grid scan is a set of stills at a stationary spindle, and the angle it stood at is what relates one grid to another taken elsewhere on the circle. Users were finding an all-zero omega in the file and concluding the angle could not be recorded at all. It already can, and has since the goniometer and the grid scan stopped being alternatives: send the axis with step 0 and its start angle, and that angle is written per image into the NXmx sample chain, read back by reader/, and taken by dials.import as a set of stills. Measured on a generated 12-image grid: the placeholder file carries omega = 0 x 12, the same file with the axis sent at step 0 carries omega = 90 x 12, and dials.import reports "still: 1, sweep: 0" for both. Nothing in the code needed changing, so nothing was; what was missing was that nobody could tell, and that no test held the behaviour down. So: the API and the HDF5 documentation now say it in as many words, and three tests pin the three legs the value crosses - the OpenAPI request (which used to drop the grid scan whenever an axis was present, unpinned until now), the CBOR start message, and the file round trip. Also corrects a claim two comments and the HDF5 page were making. NXmx can express "no rotation" perfectly well - a sample may depend_on "." - so the placeholder is not there for the standard's sake. It is there because dxtbx cannot read a sample chain of translations alone: strip the rotation axis from a grid scan master and dials.import dies in get_dxtbx_goniometer with a matmul dimension mismatch. Recorded so nobody removes the placeholder on the strength of the standard. One thing the change does not fix, because it cannot: a stationary angle is invisible to DIALS when a grid scan is present. dxtbx picks the first varying axis as the scan axis, which is a grid translation, so the oscillation reads (0, 0); and with exactly one rotation axis in the chain it builds a single-axis goniometer whose fixed rotation is the identity, never consulting the angle. The same angle IS visible when it is the only candidate (oscillation reads (90, 0)) or when a Smargon head puts a second rotation axis in the chain (the setting rotation then carries it). The value is in the file and correct either way.	2026-09-02 11:04:29 +02:00
leonarski_fandClaude Opus 5	7c3ab72fed	reader: refuse a CBF that is not a detector image instead of misreading it ... Three ways a miniCBF opened silently wrong. A byte-offset CBF with no PILATUS header at all was accepted: Count_cutoff then defaulted to 0, SaturationLimitFromValue(0) is 1, and every pixel at or above one count was flagged saturated - the integration accept gate drops the whole reflection, so the run comes out empty for a reason nothing reports. The pixel size defaulted to 0 with no validation anywhere downstream, which collapses every resolution, every scattering vector and the beam centre in millimetres. XDS writes its correction files in exactly this shape, so this is not hypothetical. A pixel size is now required to claim the file at all, and a missing Count_cutoff leaves the saturation limit unset - falling back to the container's own overflow, which can only fail to call a pixel saturated - with a warning saying so. And the header captures are character classes, not number grammars: "[\d.eE+-]+" matches a bare "." and "(\d+)" matches a digit string too long for int64. std::stod and std::stoll answer both with a raw std:: exception, which escaped the format probe - CanRead catches JFJochException only - so merely LOOKING at a corrupt file threw out of the viewer's open path. A header field that does not parse is now reported as a malformed header. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:40 +02:00
leonarski_fandClaude Opus 5	26fc4b02b3	symmetry: carry the space group as the group, not as its number ... The adopted space group travelled the pipeline as a bare int and was rebuilt downstream with find_spacegroup_by_number, which returns the reference setting. So every setting a number cannot name was destroyed one line after it was determined: P 1 1 2 came back as P 1 2 1, I 1 1 2 as C 1 2 1, R 3:R as R 3:H. DatasetSettings now holds the gemmi::SpaceGroup itself, DiffractionExperiment exposes it as GetGemmiSpaceGroup() / GetSpaceGroupOrP1(), and everything that used to take an int - HKLKeyGenerator (its int constructor is gone, so the compiler finds the callers), the merge, the R-free flags, French-Wilson, the reindexing ambiguity, the completeness enumeration, the MTZ and mmCIF exports, the model validation - takes the group. -S keeps the setting the symbol names rather than reducing it to a number. The end message carries both spellings and a reader prefers the name, since only the name keeps the setting while the number is what a reader written before the name understands. It carries them over CBOR too: the determined group was never serialised at all, so a group rugnux chose reached the master file only when the same process wrote it, and an online writer fell back to whatever the user had supplied at the start. Both keys are optional additions, so an older reader skips them and a newer one reads an older sender. On disk the master's /entry/sample/space_group carries the extended Hermann-Mauguin name and is what the reader takes the group from, so a setting survives a _process.h5 and the --mode scale that re-reads it; the number stays beside it and is the fallback for files written before. Every one of the 230 reference settings the old writer could produce reads back as itself, so older files are unaffected. Stage A and Stage B of the search still enumerate reference settings only, so this determines no group differently today - it is what the enumeration needs before it can be widened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:16:43 +02:00
leonarski_fandClaude Opus 5	a27c4cf26f	reader: take a miniCBF's mounting from the imgCIF axis table its header states ... A miniCBF header states three things about how the instrument is put together that the reader was assuming instead: which laboratory direction the image's columns run along, which its rows run along, and which the spindle turns about. Some beamlines append a CBF template block holding the full imgCIF axis table, which says all three outright. Two instruments in the corpus are not what was assumed, in two different ways. One mounts its detector a quarter turn round, so the image's columns run vertically. Another turns its spindle about the VERTICAL, with the image mounted the usual way; its table says so, and its "# Oscillation_axis" line says so a second way, by naming the image direction the spindle runs along rather than a vector. Either error leaves the spindle 90 degrees from the image. That is not a sign, so the run's axis-sign rescue cannot reach it, and no refinement recovers it: all three affected sweeps indexed nothing usable. So the table is read. The element axes give the image orientation, matched against the eight discrete mountings exactly as the NXmx module directions already are - the match itself moves to DetectorOrientation, so both readers share one definition rather than two copies. The goniometer axis with no parent gives the spindle DIRECTION; its sign stays the rescue's business, which is the part a convention can legitimately differ on. The detector axis with no parent gives the 2theta arm, replacing the assumption that the arm shares the spindle's axis - the one header stating both states them with the same vector, so this changes no answer, only what it rests on. imgCIF's frame differs from the internal one by a half turn about x, a rotation and not a mirror, as writer/HDF5NXmx.cpp already records from the other side. Where a header carries no table, a "+SLOW" on the Oscillation_axis line still says the spindle runs along the image's slow direction. That is the only thing one of the three affected sets says about it. The axis NAME on that line stays unusable - the header that carries both says "X.CW" where its own table says Y - but the direction token is not: where both are present they agree, which is what makes reading it evidence rather than a guess. Also: naming a frame with no directory at all now finds its sweep. parent_path() of a bare filename is empty and iterating an empty path finds nothing, so running from inside the data directory reported that no images were found. Measured, with nothing on the command line. The vertical-spindle protein set goes from no usable lattice to 100% indexed, P 6(3) 2 2 with a cell 0.43% from deposited, 87846 reflections at 86.3% completeness and CC(1/2) 0.995. Its companion from the same detector, which has no table and only the +SLOW token, goes from a spurious monoclinic cell at 2.3% completeness and I/sigma 0.21 to the right orthorhombic lattice, 97.7% indexed, 59.7% complete, CC(1/2) 0.996. The quarter-turned set's three sweeps, at three arm positions, now all index without the hand-passed quarter turn they needed and agree on one cell to 0.03 A. Six miniCBF sets that state no table and no +SLOW - including one whose Oscillation_axis line names an axis in a third dialect - are byte-identical in .hkl, .mtz, .cif and the image statistics, as are two NXmx sets, which is the shared orientation matcher moving nothing on that path either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:03:32 +02:00
leonarski_fandClaude Opus 5	5c44e544dc	reader: place a detector swung out on a 2theta arm where the file says it stands ... Chemical crystallography reaches high angle by swinging the detector out on a 2theta arm. Both readers had the number and neither used it: the miniCBF header's Detector_2theta was parsed into a struct member nothing ever read, and on the NXmx side the rotation was in the depends_on chain, which was not followed at all. A sweep taken at 30 degrees was therefore processed with its detector plane 30 degrees from where it stood, and nothing indexed. The geometry could already express it, and needed no change: the arm turns the detector about the sample, so the distance is still measured along the detector normal and the beam centre is still the point of normal incidence - which is exactly the PONI convention, and a swung detector is one PONI rotation. What moves is the direct beam, by distance*tan(2theta), off the beam centre and often off the detector. NXmx is the harder half, because the swing has no field of its own: it is one rotation in the chain the detector's position depends on, and "two_theta" is only one beamline's name for that dataset. So the chain is followed and its rotations composed, rather than a field of one name being looked for - each transformation states its vector in the frame of the one it depends on, which is why the product is the whole placement. Translations are skipped; they are the distance and the beam centre, which the file states separately in the square-on frame. Vectors come from McStas through the same 180-degree turn about z the module directions already use, a proper rotation, so an axis carried through it turns the same way. The three rotations a file this system writes ARE that chain, and are also read as the PONI angles - so those three paths are skipped, or every tilted file we have ever written would come back tilted twice. That is the one way this change could have broken existing data, and the test for it writes a tilted file and reads it back. For miniCBF the arm turns about the base spindle axis: on the four-circle geometry those headers describe the two are one axis, and the imgCIF axis table such a header carries states them with the same vector. Both now come from one constant, so a later correction to the frame moves them together. Measured. On a swung NXmx sweep the chain gives rot2 = -0.34907 rad for the 20 degrees it states, and the sweep goes from "nothing was integrated" to 25000 reflections at 82.2% completeness and CC(1/2) 0.9993, in the same space group and the same cell to 0.03 A as the square-on sweep of that crystal; the opposite sign indexes nothing. A miniCBF sweep at 30 degrees goes the same way, to 0.585 A, and a second sweep of that crystal at 55 degrees reaches 0.476 A and reproduces the cell again - with a low-resolution limit of 2.36 A rather than 13 A, which is what a detector swung that far records. On all of them post-refinement recovers the header's own beam centre and distance, and the beam stop shadow sits within four pixels of where the swung geometry puts the direct beam, 417 and 537 pixels from where the unswung one does. Seven sets whose detector is square to the beam, three of them carrying a chain whose 2theta is zero, are byte-identical in .hkl, .mtz, .cif and the image statistics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 12:36:46 +02:00
leonarski_fandClaude Opus 5	8e9ca1f6d2	tests: open a third-party NXmx master, in the shape a Diamond-written one takes ... Covers all five ways such a master differed from a DECTRIS one - lengths in millimetres, no detectorSpecific, the distance one level up in NXinstrument, a pixel_mask linked into a file not holding it, and per-file links naming a plain /data - and the reader's refusal to report a master whose data files cannot be read as a dataset with no images. Verified to fail on the parent commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:27:53 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	caef26873e	v1.0.0-rc.145 (#55) ... This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144. * **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future. * XDS plugin: Improve performance of VDS reading. * jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD. * jfjoch_writer: Clean-up of HDF5 routines to better handle issues. Reviewed-on: #55	2026-05-06 21:50:02 +02:00
leonarski_f	d760b12a18	v1.0.0-rc.141 (#51) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times * frontend: Fix selection of FFTW in indexing settings Reviewed-on: #51 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 13:04:54 +02:00
leonarski_f	239a441ee6	v1.0.0-rc.140 (#50) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection * jfjoch_broker: Minor bug fixes for rare conditions Reviewed-on: #50	2026-04-29 21:40:22 +02:00
leonarski_f	4a852b4d6b	v1.0.0-rc.134 (#43) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long * jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata * XDS plugin: Add generation of Jungfraujoch plugin for XDS * CI: Add tests with XDS and DIALS (`xia2.ssx`) Reviewed-on: #43	2026-04-09 13:30:47 +02:00
leonarski_f	166fcdb68f	v1.0.0-rc.131 (#39) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Fix bug in saving JUNGFRAU calibration (pedestal/pedestalRMS) * jfjoch_viewer: Fix calibration (pedestal) images being open flipped * jfjoch_process: Add space group detection (EXPERIMENTAL) Reviewed-on: #39	2026-03-07 11:34:04 +01:00
leonarski_f	36d0507758	v1.0.0-rc.119 (#26) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_broker: Add binary export of data analysis plots over OpenAPI * jfjoch_broker: Minor fixes to HTTP error handling * jfjoch_viewer: Prefer binary plots over JSON plots * jfjoch_viewer: Change foreground with F button + wheel * jfjoch_viewer: Change way how angles are displayed * jfjoch_viewer: Display resolution of the mouse cursor in top left corner Reviewed-on: #26 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-08 19:53:35 +01:00
leonarski_f	31a357fa57	v1.0.0-rc.113 (#19) ... This is an UNSTABLE release and not recommended for production use (please use rc.111 instead). * jfjoch_broker: Improve handling of rotation indexing * jfjoch_broker: More information saved in CBOR end message (WIP) * jfjoch_writer: Save rotation indexing lattice parameters and Niggli class * jfjoch_viewer: Remove (for now) primitive cell information Reviewed-on: #19 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-02 09:29:22 +01:00
leonarski_f	f44c6520a8	v1.0.0-rc.107 (#12) ... This is an UNSTABLE release. * jfjoch_viewer: Minor polishing of new functionality * jfjoch_broker: User NaN for empty azimuthal bins Reviewed-on: #12 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-24 07:07:32 +01:00
leonarski_f	a6b5cb5629	v1.0.0-rc.105 (#10) ... This is an UNSTABLE release. * jfjoch_viewer: Clean-up widgets slightly * jfjoch_viewer: Limit right panel to 600 pixels * jfjoch_viewer: Parse crystal symmetry type * jfjoch_viewer: Grid scan view takes color map and can be fit to zoom Reviewed-on: #10 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-20 09:46:25 +01:00
leonarski_f	95acf3aba3	v1.0.0-rc.104 (#9) ... This is an UNSTABLE release. jfjoch_writer: Fix and improve the way grid scan geometry is saved (non-NXmx extension makes it way easier) jfjoch_viewer: Display grid scan results in 2D (work in progress) jfjoch_viewer: Improve auto-scaling on start of images (work in progress) jfjoch_viewer: Add B-factor and resolution estimate to the dataset info plots Reviewed-on: #9 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-19 17:28:10 +01:00
leonarski_f	4dbbf0e365	v1.0.0-rc.97 ... This is an UNSTABLE release and not recommended for production use (please use rc.96 instead). * jfjoch_broker: For DECTRIS detectors add dark data collection during initialization for bad pixel mask * jfjoch_broker: Refactor of calibration logic for more clear code (likely to introduce problems) * jfjoch_viewer: Add option to handle user pixel mask (experimental) * jfjoch_viewer: More options for ROI * jfjoch_viewer: Add window to display calibration Reviewed-on: #2 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-09 12:42:27 +01:00
leonarski_f	5bb8898a0e	v1.0.0-rc.85	2025-09-24 22:52:31 +02:00
leonarski_f	c0bd8e1215	v1.0.0-rc.82	2025-09-22 19:49:39 +02:00
leonarski_f	5d9d2de4a4	v1.0.0-rc.81	2025-09-21 19:27:51 +02:00
leonarski_f	c67337cfe1	v1.0.0-rc.72	2025-09-08 20:28:59 +02:00
leonarski_f	fa95858008	v1.0.0-rc.69	2025-08-17 21:21:20 +02:00
leonarski_f	a32cec3404	v1.0.0-rc.67	2025-08-11 11:13:59 +02:00
leonarski_f	fbaf15aa07	v1.0.0-rc.60	2025-07-05 18:52:42 +02:00
leonarski_f	0d1b388f4d	v1.0.0-rc.50	2025-06-20 09:52:26 +02:00
leonarski_f	41a3e671f4	v1.0.0-rc.41	2025-06-10 18:14:04 +02:00
leonarski_f	53c90ee5d8	v1.0.0-rc.40	2025-05-28 18:49:27 +02:00
leonarski_f	b245967df3	v1.0.0-rc.38	2025-05-12 14:17:24 +02:00
leonarski_f	040cf08386	v1.0.0-rc.36	2025-05-05 19:32:22 +02:00
leonarski_f	b0607ab3ca	v1.0.0-rc.34	2025-04-14 11:52:06 +02:00
leonarski_f	ddf4c75645	v1.0.0-rc.31	2025-03-02 13:15:28 +01:00