10 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	ef5da29319	calibration: a fit that hands back the file's own tilt is not a calibration ... A powder calibration is run because the file's geometry is in doubt, so a fit that quietly returns part of that file has answered nothing - and it is indistinguishable from one that worked, down to the residual and the sigmas arranged around it. On one of four LaB6 exposures of one detector the tilt came out at 2.93x its own sigma, a hundredth under the significance gate, so it was declined and pinned - at the master's hardcoded rot1 -0.08, rot2 -0.22 deg. That is eight times the tilt just refused, on no evidence, and worth 10 px of PONI at 190 mm. rugnux printed it to four decimal places, wrote the .poni, and exited 0. Judge the result on provenance instead of on any residual: a geometry is a measurement only if every parameter in it came from this data. Two ways out of the fits do not qualify - a covariance that never conditioned, so the fit cannot say what it determined, and a declined tilt pinned at a non-zero value from the file. A declined tilt over a file stating no tilt still qualifies, because reporting no tilt is then exactly what was measured; so does --no-refine-tilt, because a hold that was asked for is a stated choice and not a silent substitution. No single number separates the four. rms is 2.465 px against 1.44-1.64; the significance of all four lies between 2.93 and 4.47, so the gate is nearly a coin flip at these distances and moving it would only recalibrate on one population; and the failed fit has the TIGHTEST parameter sigmas of the set, because pinning the tilt removes the tilt/centre correlation that inflates a good fit's. The spot cross-check reads 13.5 px against 0.98-2.66, but 10.4 px of that is the pinned tilt moving the PONI - the same defect one step downstream, not independent evidence. On a failure rugnux says so, writes no .poni - a PONI file states where the detector is and has no field in which to say it does not know - writes the JSON with converged false and the reason beside it, and exits non-zero. The re-binning pass now prefers a converged refit over a non-converged one whatever its residual, so a tilt an earlier pass measured is not what a later one gets pinned at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 09:37:46 +02:00
leonarski_fandClaude Opus 5	bd6e933f04	api: name the calibration schemas powder_, not calibration_ ... "Calibration" already means something else in this broker: the JUNGFRAU pedestal and gain measurement, which owns /statistics/calibration, calibration_statistics and JFJochState::Calibration. Four schemas called calibration_* would have sat beside it meaning a completely different procedure, and the confusion is cheapest to remove now, before an endpoint returns them. calibration_output, calibration_quality, calibration_fit_sigma and calibration_spot_check become powder_calibration_output, powder_calibration_quality, powder_calibration_fit_sigma and powder_calibration_spot_check. All four, not only the outer one - the collision is in the word, and half a rename would read as though the inner three belonged to the other kind of calibration. Rename only. Every client regenerated from the spec: the C++ server model, the TypeScript frontend client, redoc-static.html, and the python client, which is gitignored but was checked to still decode the file rugnux writes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:58:24 +02:00
leonarski_fandClaude Opus 5	6fe3f30ab4	api: define the powder calibration result as calibration_output ... The JSON a calibration writes was a shape invented at its writer, described only by the comments around it. That is enough for a file somebody reads with jq and not enough for anything else: a client cannot type it, and an endpoint returning it later would have to declare the shape a second time and keep the two in step by hand. So declare it where every other shape in this system is declared. calibration_output holds dataset_settings and a calibration member; the latter is calibration_quality, which nests calibration_fit_sigma and calibration_spot_check. The descriptions carry what a reader has to know to use the numbers rather than only what they are named - that beam_x_pxl is the PONI and the direct beam is elsewhere, that the rotations travel together because a body omitting them states a flat detector, that a tilt below about three sigma was declined and pinned, and that the two correlations approach 1 as the tilt stops being separable from the beam centre. Nothing references it yet. It is declared now because /powder_calibration will return exactly this, and because the file rugnux already writes is decodable today: jfjoch_client's CalibrationOutput.from_dict reads it as it stands, with o.calibration.fit_sigma.correlation_beam_x_rot1 and the rest typed. Generated clients regenerated from the spec, as the spec requires: the C++ server model (four new pairs under broker/gen/model), the TypeScript frontend client, and broker/redoc-static.html. Both regenerations are purely additive - no existing generated file changed except to export the new names. The python client regenerates from the same spec and is gitignored. The test now validates the WHOLE file against the generated Calibration_output rather than only its geometry member against Dataset_settings, so the quality block is under the same contract: a field renamed or newly required in jfjoch_api.yaml fails here rather than at a client. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:54:09 +02:00
leonarski_fandClaude Opus 5	8691bf4876	calibration: write the result as JSON, not only as a PONI file ... A calibration run produced one file, and it was a pyFAI PONI - which pyFAI and its neighbours read and nothing in this system does. Carrying the answer back into the instrument meant a person reading numbers off a printed report and retyping them into a dataset_settings body, and the report is where the two points a "beam centre" can mean are easiest to confuse. So write <prefix>.json beside it. Its "dataset_settings" member holds the geometry under the property names broker/jfjoch_api.yaml gives them and holds nothing else, so it is a valid dataset_settings body as it stands: curl -X POST -H 'Content-Type: application/json' \ -d "$(jq -c .dataset_settings det.json)" http://broker:5232/start beam_x_pxl is the PONI, as everywhere here. The three poni_rot*_rad ride along whenever any is non-zero and are left out when all are zero: a body without them does not leave the tilt unstated, it states a FLAT detector, so they travel together or not at all - the same rule the report's JFJOCH_DATASET_SETTINGS block already follows. The "calibration" member holds what the run knows about that geometry: the residual, the fit's own sigmas and the correlation between the tilt and the beam centre, whether the tilt cleared its significance test or was declined and pinned, where the direct beam lands, and where the spots independently put the beam. A calibration that has gone wrong looks exactly like one that has not until those are read, and a machine-readable file that carried only the geometry would be the easiest possible way to feed a bad one into an instrument. Tested against the model generated from the spec rather than against a list of field names written out by hand: the file's dataset_settings member is parsed into org::openapitools::server::model::Dataset_settings and validated, so a field renamed or newly required in jfjoch_api.yaml fails here rather than at someone's POST. The tilted and untilted branches are both covered, and the PONI/direct-beam distinction is asserted rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:47:51 +02:00
leonarski_fandClaude Opus 5	d275bbcae7	calibration: -C overrides the calibrant, with absences from -S ... The five named standards are a convenience, not the limit of what a powder calibration can be run against. A unit cell given with -C now IS the standard in --mode calibration, and its rings are enumerated from that cell. A cell alone does not give a ring list, though: the centring and any glide decide which hkl the lattice actually diffracts into, and the fit pairs the innermost OBSERVED ring with the innermost LISTED one - so a list opening with a reflection that is not there scales the whole calibration by the ratio between them. Where -S is given, the absences come from the space group itself via gemmi, which covers centring, glides and screws in one mechanism rather than the three hand-written conditions the built-in table uses. The two agree exactly on LaB6 and CeO2, which is the cross-check that says the gemmi route is safe to hand a user's cell. They do NOT agree on silicon, and the test now pins that: Fd-3m's symmetry absences are only the F centring, while silicon's 222 and its relatives are extinguished by its two-atom basis - a structure-factor absence, not a symmetry one, so no symmetry handler can know it. gemmi offers 24 rings where the diamond condition gives 18. The extra rings do not move the first one, so the distance is not scaled, but they are rings with no intensity offered to the matcher - which is why --calibrant si stays, and why the usage text says -S supplies symmetry absences only. Without -S the cell is taken as primitive and the log says "assumed primitive". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:23:37 +02:00
leonarski_fandClaude Opus 5	5a80d2df53	calibration: fix four ways the powder fit quietly loses its input ... None of these changes the answer on a well-separated cubic standard - the LaB6 distance series is bit-identical by both methods - but each one is a case where input is dropped or mis-assigned without saying so. The circumcentre vote grid was a fixed 4000x4000 box, and the caller never passed anything else. That allocated 128 MB whatever the detector, and on a detector larger than 4000 px in either direction it put the beam centre outside the grid, so every vote was discarded and the guess failed with "Beam center not found". Span the spots' own bounding box instead: a powder ring encloses its centre, so that is where the answer has to be. uint32 votes while there - the most any bin can take is C(500,3). Spots were assigned to the FIRST calibrant ring within a fixed 0.1 1/A, not the nearest. Silver behenate's orders sit 0.108 1/A apart and hexagonal ice has three rings inside 0.06, so for those two standards the window reaches the neighbour and every point lands on the lower-q ring of the pair, biasing the distance. Take the nearest ring, and clamp the window to half the gap to the neighbour - which is what the profile path already did inline, now shared as RingMatchWindow and covered by a test that checks it actually narrows on the crowded standards and not on LaB6. A profile bin no pixel fell in is NaN. SectorPeakQ dropped such a sector by accident, through NaN comparisons falling false; check the four background bins and return explicitly. Ice-ring handling is switched off in calibration mode. Flagged spots are sorted last by the spot budget and so discarded first, which for --calibrant ice throws away exactly what is being calibrated on. The two per-ring std::cout lines in GuessGeometry are gone: a library has no business writing to a terminal, and constructing a Logger to keep them would emit a version banner from inside a fit. What matched belongs in the result struct, which the quality gating still to come needs anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:18:27 +02:00
leonarski_fandClaude Opus 5	f34d9fe62b	calibration: move the powder fit into image_analysis/geom_refinement ... CalibrateFromProfile/CalibrateFromSpots/WritePoniFile sat in rugnux/, so the only way to reach them was to link the Rugnux library - which drags in JFJochWriter and gemmi. Nothing in them needs either: the includes are all common/ and image_analysis/geom_refinement/, next to the RingOptimizer and RingsFromProfile they call. Moving them to image_analysis/geom_refinement/PowderCalibration.{h,cpp} puts the powder fit beside the rest of the geometry refinement and makes it reachable from anything that already links JFJochImageAnalysis - the receiver and so the broker included, which is what an online geometry calibration would need. Pure move: the file contents differ from their previous form only in the include paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:09:42 +02:00
leonarski_fandClaude Opus 5	6516bc96af	Ice rings: carry the list past 1.5 A, where ice does not stop ... The eleven measured bands end at 1.522 A because their source says so in its own words - "pure hexagonal ice has 11 diffraction rings between 4 and 1.5 A resolution" - and its subject was detecting ice in deposited data, not masking it. On a detector that reaches further, the rings it does not list are the ones left in the data: on the strong rotation set just added to the battery, 44% of every image's spots sit in ice bands, and beyond 1.5 A the spot list is ice and nothing else, which is why the resolution estimate read the ice rather than the crystal. There is nothing measured to copy below 1.522 A, so the eight added bands are calculated. Enumerating hkl is not enough and the code already said so: ice Ih is P6_3/mmc with O on 4f, and most of what enumeration emits is extinguished by the OXYGEN SUBLATTICE rather than by the space group - which is why (004) at 1.830 A and (104) at 1.657 A are missing from the measured list although they sit inside its range and its reflection conditions allow them (for (00l) the structure factor goes as cos(2*pi*l*z), and z ~ 1/16 kills l = 4). So compute structure factors - oxygen only, the hydrogens being half-occupancy disordered and weak to X-rays - and keep the lines reaching 3% of the strongest. That rule REPRODUCES THE MEASURED ELEVEN EXACTLY and every line it drops inside their range computes to zero, which is what makes it trustworthy below 1.522 A. It stops at 1.170 A: below that the real lines fall to 2-3% while the extinct ones rise to about 1%, and an oxygen-only calculation cannot separate them honestly. Every added band was independently confirmed in the data - the spot-count histogram of the strong set peaks at each of them and is empty between - and every line the rule calls extinct is absent there too. Costs, measured. The bands are inert above 1.6 A: on 38 of 39 battery sets the profile ice score does not move at all, and the one that appeared to (a jet set, 1.25 -> 2.60) does not on the peak-excluded profile the score actually uses - that was Bragg peaks in the plain profile, which is what the peak exclusion is for. Where a detector does reach past 1.5 A the bands cover more of reciprocal space: unchanged at 1.6 A, +7.4 points at 1.4 A, +16.5 at 1.18 A. On the strong set that is 17% -> 27% of reflections held out of the scale fit, and it shows: the spot resolution estimate improves from 1.33 to 1.46 A against a truth near 1.42, while CC1/2 falls 98.5 -> 97.6% and ISa 3.58 -> 3.37. Ice handling only runs at all on a run that trips the ice gate, so a clean crystal pays nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 11:27:28 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00