v1.0.0-rc.171 #81

Merged

leonarski_f merged 13 commits from rc171 into main 2026-09-17 14:42:53 +02:00

Conversation 0 Commits 13 Files Changed 211

+2857 -775

leonarski_f commented 2026-09-17 09:34:05 +02:00

Owner

Copy Link

Copy Source

• Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF.
• jfjoch_viewer: opens the CCD formats, and fixes to the dataset plots.
• Documentation updates.

* Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates.

leonarski_f added 9 commits 2026-09-17 09:34:05 +02:00

rugnux reads marCCD sweeps natively ... a1889c45e9

A decade of deposited CCD data is archived as marCCD - what Rayonix MX-series and
mar Mosaic detectors write - and rugnux could not open any of it. reader/ had two
formats, NXmx/HDF5 and PILATUS miniCBF, and rugnux_cli dispatched on the one
CanRead(); this adds the third.

The format needs no new dependency: a marCCD file is an ordinary uncompressed TIFF
whose 3072-byte instrument header sits in the gap between the TIFF header and the
pixels, so libtiff - already fetched for JFJochPreview in every build mode - reads
the image, and the header is a fixed-offset block of little-endian int32.

Two things differ from the miniCBF path and are worth naming:

* The pixel size is NOT rounded to whole micrometres. A PILATUS pixel is exactly
  172 um so the existing reader can afford lround(); a MAR300 pixel is 73.242 um,
  and rounding it to 73 is a 0.33% scale error on every cell edge reported.
* The sweep template is the last run of digits in the whole file name rather than
  in the stem, which covers both schemes these detectors use - a numbered stem
  (xtal_1_00042.mccd) and the frame number as the extension (D1.042).

A CCD frame marks no untrusted pixels, so the sweep starts with nothing masked, and
the format has nowhere to state the rotation axis' direction, so the run settles its
sign from the data exactly as it does for a miniCBF carrying no axis table.

Measured on one deposited 300-frame Rayonix MX-300 sweep, de novo with no flags:
100% indexing, the deposited point group, cell within 0.045%, 99.5% complete at
multiplicity 3.4, in 26 s. The chosen sweep is confirmed against the instrument
header before it is opened, so a directory of ordinary TIFFs is refused rather than
read with a pixel size of zero - a unit test covers that, both naming schemes, and
the geometry conversion.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rugnux and the viewer read SMV and gzipped miniCBF ... 2ab8c55dfa

Two more of the formats deposited data actually arrives in, found by processing a
corpus of it: every PETRA III EMBL set is .cbf.gz, and NSRRC and the whole ADSC
Quantum era are SMV. Both were previously "no native input".

SMV is an ASCII "KEY=value;" block between braces, then the pixels - no container,
no compression, nothing to decode by offset - so reader/SMV.{h,cpp} and
JFJochSMVReader are a smaller job than the marCCD pair they sit beside, and need no
new dependency at all. Two things the format does not give us, both said out loud
rather than papered over:

* It states no saturation value, so overloads are judged on the 16-bit container
  alone. That can only fail to call a pixel saturated, never condemn a good one,
  but a CCD at the top of its range does saturate, so the reader warns once.
* Its beam centre is in MILLIMETRES and which of X/Y is the fast direction is a
  convention rather than a rule. Measured on one ALS ADSC sweep the file's value is
  TRANSPOSED: as stated it indexes 2/60 frames, and the run's own beam-centre
  measurement (which adopts the right one automatically) indexes 60/60. Swapping it
  here would fit that writer and might break another, so the header is read as the
  format defines it and the measurement stays the arbiter. Revisit with a second
  vendor's SMV in hand.

.cbf.gz needed only Slurp() in MiniCBF.cpp, through which every read already passes:
it sniffs the two-byte gzip magic - not the file name - and takes a zlib path when it
is there, leaving the plain path free of zlib's buffer copy. zlib-ng is already in the
build, so this is a link line, not a dependency. The sweep template grew a suffix,
because ".cbf" and ".cbf.gz" are separate sweeps and std::filesystem cannot split the
double extension on its own.

The viewer's single cbf_reader becomes three, dispatched by CanRead() in the same
order as rugnux. Dispatch is by CONTENT in both: ".img" is used by miniCBF, marCCD
AND SMV depending on the writer, and a PDB detector label has now been wrong about
the format four times, so an extension decides nothing.

Measured, de novo, no flags: 9fcg (1800 gzipped frames) gives P4 and a cell 0.06%
from the deposited one at 1.37 A against a deposited 1.54; 6oel (ADSC SMV) gives
F4132 - 96 operations, the most a protein space group can have - and a cell 0.05%
out, 100% indexed. Tests cover both formats and the transposed-beam-centre case with
fixtures written byte for byte, so they need no external data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

marCCD: take the distance from the start/end pair when the first field is zero ... 28f222f51d

Four of the twelve marCCD sets in the new corpus were refused with "states no
detector distance". Their headers carry xtal_to_detector (offset 640) as zero and
the real distance only in start_xtal_to_detector and end_xtal_to_detector - 70, 170
and 200 mm on three BESSY Rayonix MX-225 sweeps, against a set where all three
fields agree at 300 mm. Reading the first field alone is what refused them.

The guard itself was right: a distance of zero collapses every resolution and every
scattering vector, so refusing beats processing silently. It just fired on data that
does state a distance, in the field the format also defines for it.

With the fallback, all three process de novo and match their depositions - P2_1 at
cell 0.06%, I422 at 0.40%, P1 at 0.07% - and each reaches finer than its deposited
resolution (0.889 A against 1.09, 1.440 against 1.69, 1.757 against 1.93).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Documentation sweep: Rugnux naming, repository citations, defect pass ... 70c4d871b6

- Capitalize Rugnux as a proper noun throughout the prose; the command
  stays lowercase `rugnux` in code font. RUGNUX_OVERVIEW.md is retitled
  "What Rugnux does".
- ACKNOWLEDGEMENT.md cites the raw-data repositories only: dataset counts
  and DOI prefixes moved out (EXTERNAL_TEST_DATA.md owns them), the ESRF
  data portal gains its citation (Dimper et al. 2019), and MXRDR remains
  name + link - it has no canonical citation paper.
- RUGNUX_FORMATS.md: the CCD formats (marCCD, SMV) are supported as-is
  with very limited scope, and per-panel XFEL data is not read.
- Fix wrong facts a reader would act on: nonexistent `make jfjoch`
  targets, invalid udev rules, PUSH sockets documented as PULL, swapped
  writer width/height, underload semantics, the transposed pixel-mask
  numpy example (the server checks width and height separately), the
  Durin/Neggia mask-bit table, FPGA threshold register addresses and the
  mailbox bit field, the I2C core's document number (PG090), an inverted
  MODEL_FIT_SIGMA formula, a self-inconsistent worked report example,
  and 11 cross-page anchors whose slugs carry MyST section numbers.
- Unify CC1/2 spelling in prose; math notation and report keys unchanged.
- Sweep grammar, typos and editing residue across the FPGA, deployment,
  streaming and analysis pages, including historical CHANGELOG typos.
- rugnux_cli.cpp: the -S usage/error examples pair 96 with P43212;
  92 names a different group.
- Root THIRD_PARTY_NOTICES.md: scope the GPL-compatibility claim (CUDA
  EULA) and the vendored-table intro (traccc); the docs copy regenerates
  via update_version.sh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs: the 51 second-round external datasets, and XRDa ... 7fecd11337

EXTERNAL_TEST_DATA.md gains a row per dataset in the existing form - deposited
beamline, resolution, space group, cell and title from the RCSB API, source
repository with its citable DOI, and the detector read out of the image files
themselves. Every DOI was resolved before it was written down; one differs from the
pattern the rest follow (5EPE is 10.18430/m3159c, not the m3<pdbid> form), which is
the reason the page insists on resolving them rather than constructing them.

Seven of the 51 name a detector in the PDB entry that the files contradict, now
listed with the others. Two are not merely a model or size: one deposited as a
"MAR CCD 165 mm" writes a 225 mm plate, and one deposited as a "RAYONIX MX300-HS"
writes SMV with 315 mm geometry. marCCD and SMV headers name no model at all, so
the Detector column carries what those headers do state - plate size, serial - and
the page explains that.

A new section gives the composition of the round: repository, file format as the
files are on disk, facility and crystal system. It is metadata about the
depositions, not measurement: no quantity measured by this software appears on the
page, as the page's own preamble promises. The format spread is the point of the
round, and is why Rugnux now reads marCCD, SMV and gzipped miniCBF natively.

ACKNOWLEDGEMENT.md adds XRDa, the archive three of these came from, in the form the
preceding documentation sweep settled on for a repository with no canonical citation
paper: name, link and what it asks to be cited, and no substitute reference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

viewer: the stacked spots/background plots hold together at any font size ... 8288d4cbe6

The two half-height plots were matched once per data load, from
JFJochViewerDatasetInfo: it read both y-label widths and pushed the larger
back into both. A font change never goes through that path - each chart
re-measures its own labels in its FontChange handler - so the padding was
still the width measured in the old font, collapsed to zero, and the two
curves started at different x at 125 % and 150 %.

The pairing now lives in the chart view: each compact plot holds a peer
pointer, and every rebuild re-matches the pair, so a font change fixes
itself. Matched along with the label width:

- the minimum height, taken from the width of the rotated y-axis title
  (Qt elided "Spots" to "Sp..." once the font outgrew the 70 px that was
  hardcoded for it), and given to both plots so the layout does not hand
  the wider-titled one more of the dock;
- the plot area height: the upper plot now paints its x labels transparent
  instead of hiding them, so Qt reserves the same label band on both.

The background plot also stops snapping its y axis to zero - it is read
for its shape, not its level, and half a dock is not much to draw it in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

marCCD/SMV: keep the default frame time when the header's exposure is not a time ... 39a98692e2

One deposited marCCD sweep carries -2093438692 in its exposure field. Passed on as
written that is a negative count time, DiffractionExperiment refuses it, and the
whole dataset fails to open - over a number that feeds no geometry, no indexing and
no merge. With the field ignored when it is implausible, that sweep processes and
matches its deposition: P2_12_12_1, cell within 0.12%, and 1.375 A against a
deposited 2.0.

Neither format guarantees the field. The same guard is in the SMV reader, whose TIME
key is no better attested, and both say what they did rather than passing over it
silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs: condense the rc.171 changelog ... 29a90416b7

Three lines at the level the rest of the file keeps: what a user gains, not how it
works. The detail lives in RUGNUX_FORMATS.md and in the commits.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

v1.0.0-rc.171 ... 4bff5c2b01

VERSION bumped and update_version.sh run: the OpenAPI spec and the three clients it
generates, the frontend and docs version strings, the Redoc page, and the FPGA and
PCIe-driver version strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-09-17 11:34:23 +02:00

viewer: draw a live grid scan as a grid, and the compact axis names upright ... 20977e89bc

Two fixes in the dataset-info panel.

The live (HTTP) reader took the goniometer axis and the grid scan as
alternatives, so a raster collected at a head position - both fields set,
the axis with step 0 - lost its grid and was drawn as a line plot, with
the Grid button disabled. The broker had the same bug and lost it in
rc.162 (OpenAPIConvert.cpp); the HDF5 read-back path already read both.
The reader now sets both, like the other two.

Qt Charts always draws a vertical axis title rotated, which reads
awkwardly beside the two numbers it names in a half-height plot and is
what forced that plot's minimum height (Qt elided "Spots" to "Sp..." once
the font outgrew it). The title is switched off and the view paints the
name itself: upright, one letter under the next, in the series' colour,
centred on the plot area, with the left margin reserving its width. It is
re-measured on every rebuild, so a font change still fixes itself, and
plotAreaChanged re-centres it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-09-17 12:24:34 +02:00

indexing: the cap search always puts the direction grid back, and every FFT upload is checked ... 4b5c6e234d

A production broker on rc.170 segfaulted again in the FilterFFTResults length
sort, in a binary that does contain the rc.170 non-finite guard (verified from
its disassembly: the two isfinite tests are compiled in and skip the insert).
With magnitude and length guaranteed finite, the only remaining way that sort
sees a NaN key is a NaN direction vector - and the one place that writes the
direction grid is SearchCap.

SearchCap swaps in a temporary cap grid but restored it around ExecuteFFT only.
The fill can run out of memory, the upload can fail, and the .at() that reads
the peak back throws on exactly the corrupted result row this search has to
survive - any of those left the cap grid installed on an indexer the pool hands
the next image. If the axis was ever not a direction, axis.Normalize() is NaN
and the leaked grid is 16384 NaN directions, which is that crash on every image
afterwards. The restore now covers the whole body, and an axis that is not
finite and non-zero is refused instead of normalized into NaN. The peak is also
tested against the minimum length before the .at() rather than after, so an
unusable peak with a corrupted index no longer throws at all.

DirectionsChanged uploaded the grid with three unchecked cudaMemcpy calls, and
the spot upload in ExecuteFFT had three more: a silently failed upload leaves
the device holding the previous grid, so the directions the host reads the
results against are not the ones the kernel used. Checked like the rest of the
file.

Tests: [Indexing] (9 cases, 115 assertions) and RotationIndexer pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-09-17 13:02:32 +02:00

tests: the FFT indexers on a frame of pure noise ... 51f51ec5bf

The frame that works the indexer hardest is junk, not a crystal: a real lattice
prunes the FFT shortlist to four or five distinct directions, while noise leaves
dozens, which is what drives the length sort and the degenerate-plane fallback -
where a production broker segfaulted twice. There was no test in that regime.

Two cases, both over every FFT back-end the build has (GPU FFT under CUDA, CPU
FFTW always), so a non-CUDA job covers them as well:

- a cloud of 1500 reciprocal-space vectors of random direction and length, with
  no periodicity in it. Both back-ends return 70 candidate lattices from such a
  frame, which is the designed behaviour - Run() offers candidates and the caller
  scores them - so what the test pins is that none of them looks like a crystal:
  each takes about 1% of the cloud against the ~100% a real lattice takes, and
  the assertion allows 5%.
- the same noise flattened onto a tilted plane. A coplanar shortlist cannot close
  a cell, so this is the branch that goes looking for the missing row in a 3 deg
  cap (SearchCap), verified with a temporary probe to enter it on both back-ends
  and not to enter it on the isotropic frame. The SAME indexer object then has to
  index a clean lattice correctly, which is the regression test for the cap search
  putting the direction grid back.

The cloud is drawn from a fixed seed, and from the engine scaled by hand rather
than through uniform_real_distribution, whose output is not specified to be the
same in every standard library: a crash that needs one particular junk frame is
no use as a regression test if the frame is redrawn on each machine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-09-17 13:59:31 +02:00

broker: error responses carry the content type the OpenAPI spec declares ... d6a2b9d19e

The 500 handlers built a correct error_message object but sent it through
send_plain(), which labels the body text/plain. The spec declares 500 as
application/json + error_message, and the generated clients key their
deserialization off the content type: the Python client kept the body as a
raw str, handed it to ErrorMessage.from_dict(), and pydantic raised - the
exception swallowed by the finally: in response_deserialize, so the caller
got a ServiceException with data=None and only the raw text in .body.

send_plain becomes send_error, taking the content type alongside the code
and the body; the two exception handlers now return it. 400 stays a
plain-text exception string, exactly as the spec says; 500 is JSON.

Two further 500s did not carry an error_message object at all and now do:
the generic std::exception branch of handleParsingException, which returned
a bare what(), and ProcessOutput's output-validation failure, which returned
the validation dump. A catch-all set_exception_handler covers anything that
escapes a route's own handler - httplib would otherwise answer with a
bodyless 500, which fails to deserialize the same way.

Verified against a running broker: POST /pedestal in the wrong state returns
application/json and the generated Python client parses it into
ErrorMessage(msg=..., reason='WrongDAQState'); a malformed body still
returns 400 text/plain.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

leonarski_f merged commit 30b6800289 into main 2026-09-17 14:42:53 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) mx-gitea perl_d (perl_d) takaba_k (takaba_k) wang_m (wang_m)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#81