v1.0.0-rc.168 #78

Merged

leonarski_f merged 18 commits from rc168 into main 2026-09-10 13:51:16 +02:00

Conversation 0 Commits 18 Files Changed 174

+1097 -488

leonarski_f commented 2026-09-10 11:40:43 +02:00

Owner

Copy Link

Copy Source

• rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results.
• A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell.
• rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; ACKNOWLEDGEMENT.md now ships in every package beside LICENSE and THIRD_PARTY_NOTICES.md.

* rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`.

leonarski_f added 18 commits 2026-09-10 11:40:43 +02:00

rugnux: acknowledge the field and the open-source stack ... 11c15837c6

The crystallographic methods rugnux implements were developed and published by
the X-ray research community, and the program is built on other people's
open-source software. Both were credited only in files that never left the
source tree: jfjoch_viewer shows LICENSE and ACKNOWLEDGEMENT.md from its own
resources, but rugnux said nothing and the packages shipped no
ACKNOWLEDGEMENT.md at all.

One text, defined once, printed by the CLI after the licence banner and written
at the foot of every <prefix>_report.txt, so a result carries the
acknowledgement wherever it travels and names the files that hold the detail.
ACKNOWLEDGEMENT.md joins LICENSE and THIRD_PARTY_NOTICES.md in
JFJOCH_NOTICE_FILES, which is installed per component - so it reaches the Linux
.tgz, the Windows .zip, the NSIS installer, the macOS image and the DEB/RPMs
alike, next to the two files it links to.

Also drops the SUMMARY block's explanatory paragraph. It was written to whoever
maintains the report format, not to the crystallographer reading their results,
and the greppability contract it described is already in docs/RUGNUX_REPORT.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018FeM2CtemFrbhoK1G9n7J5

performance: the French-Wilson quadrature stops recomputing what its grid already fixes ... e74517c2fb

The posterior is integrated on j = (i + 1/2) * dj, so sqrt(j) and log(j) each
split into a per-reflection factor and a term that depends only on i:
sqrt(j) = sqrt(dj) * sqrt(i + 1/2) and log(j) = log(dj) + log(i + 1/2). The
i-dependent halves are the same for every reflection and every intensity, so
they are tabulated once for the run rather than recomputed at each of the 400
grid points. The exponential in the second pass is left alone, but a point more
than 37 below the peak carries a weight under 1e-16 and cannot move a
normalised sum of doubles, so it is skipped instead of exponentiated - and the
posterior is sharply peaked, so most of the grid is skipped. Finally the Wilson
prior and the centric flag belong to the reflection rather than to one of its
three intensities, so the two symmetry lookups are made once and shared instead
of three times over.

None of this changes the arithmetic: on five datasets spanning the corpus's
speed range the space group, the unique-reflection count, R_meas and CC1/2 are
identical to the digit.

Measured over those five, each run twice and the two binaries interleaved so
machine load cannot favour one arm: 771.2 s to 730.7 s, 5.3 % of total wall,
between 3.2 % and 7.8 % per dataset and in the same direction on all five. The
work removed is largest where the run merges most - the quadrature runs on
every weak intensity of every merge, and a large cell at high resolution reaches
a few million of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

beam centre: the background fit stops at its fixed point, and reads the pixels on every thread ... 7db76560a6

The background ring fit is the largest single phase of an offline run on a large
detector: 626 s of a 5403 s hundred-dataset battery, against 127 s for the
beam-stop projection it reads. The cost is not spread over its invocations - the
worst single invocation per dataset accounts for 516 s of that 626 s - and the
worst one is always the second pass, at the post-refined geometry, where the fit
starts essentially at the answer.

Instrumented per iteration, that invocation is not a truncated walk. It reaches
its answer at iteration 4 and then enters a period-2 limit cycle: the step
alternates in sign every iteration at a fixed 0.30 px, and the centre oscillates
by 0.08 px about a value it knows to 1.75 px, for the 96 iterations left in the
budget. CONVERGED_PXL sits below the cycle amplitude, so the run can never meet
it; whether a dataset costs 5 s or 55 s is decided by whether its cycle happens
to fall under 0.02 px. Travel looks nothing like that - a fit forced to walk
339 px keeps its step direction through all 22 of its travelling iterations and
takes steps of 75 px down to 0.1 px - so the sign of the step against the one
before it separates the two without any length to compare against.

Two changes, measured apart.

The walk now also stops when its step reverses the previous one twice running,
which is a crossing of the fixed point rather than a step toward it.
MAX_ITERATIONS stays at 100, so the long-travel budget is untouched: the forced
339 px walk arrives as before, in 27 iterations instead of 29, 0.013 px away. A
monotone fit is bit-identical. The two cycling invocations measured go from 100
iterations to 7 and 9, moving 0.11 and 0.16 px - under 6% of the sigma each
reports, and the hundredth iteration was itself an arbitrary point on the cycle.

The per-pixel passes - a rotation, a sqrt, two atan2 and two divisions each, over
18 Mpx, three times an iteration - now run on the threads the caller was given,
split over a fixed 64 row blocks folded in block order, so the grouping of the
sums is a property of the image and not of the machine. Serial against six
threads on the same projection in the same process: 4.2-5.3x, with beam_x, beam_y
and sigma identical to four decimals on all five fits measured, and a test that
asserts one thread and eight give the same answer.

Together, measured on the same phase boundary the battery numbers use: 57.7 s to
2.4 s and 54.5 s to 3.0 s on the two worst datasets, 0.23 s to 0.10 s on a small
one. The beam-stop mask is untouched - ShadowFinder is not modified and the fit
runs after it and writes nothing back - and both masks come out at exactly the
pixel counts the unmodified binary logged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

merge: the half-set cell array is the one the flat fields already hold ... b43f72a676

The merge built a per-observation cell array with the same rule the flat
merge fields had just been filled with - `usable_merge(o) ? o.group : -1` -
but serially, over the eighty-byte observation records, so a large P1 pass
dragged more than a gigabyte through the cache to rebuild an array that
was already in hand. Hand the existing one to the half-set assignment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

merge: the report-only error-model diagnostics are measured only where they are read ... 58a9e24484

The strong-reflection ISa asymptote and the reported chi2 median are both
report-only - the header already states that a merge run with full_stats
false skips the diagnostics, because its numbers are never read - yet both
ran on every merge, including the geometry pre-pass and the pre-correction
CC1/2 merge that is discarded. The asymptote is the expensive one: a
per-group scatter array value-initialised on one thread (hundreds of
megabytes at a million ASU groups) plus a serial random-scatter walk over
every full. Gate both on full_stats, and leave them out of the error-model
line on the merges that no longer measure them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

scaling: the decay / relative-B passes read a compact array, and their per-group references run on all threads ... 2e5c0df0a5

The three step-4b passes - the global decay slope, the per-batch relative-B
and the radiation-damage monitor - were plain serial walks over the
eighty-byte observation records, a dozen of them, each reading twenty bytes
of each record and each allocating and zeroing a pair of per-group vectors.
The decay fit alone runs seven of those passes (three slope fits, four
held-out scores) on one thread while the workers and the GPU idle.

Give them the treatment the correction surfaces below them already have:
pull the fields they read out of the records once into a compact array in
fulls order, and keep a second copy of the same terms in ASU-group order
(stable counting sort) so a thread can own whole groups. The per-group
inverse-variance references then run on all threads and still add each
group's terms in fulls order, and the remaining scalar reductions keep their
serial order over the compact array - so every sum is formed from the same
terms in the same order as before, and the numbers are unchanged rather than
merely close. The build is shared by the three curve fits and the three
held-out scores of the relative-B pass instead of being repeated per call.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JFJochStateMachine: remove confusing indexing_possible setting dcc6d5e4fc

reader: a worker reads every frame into the same raw image ... 4bac6d3f21

GetRawImage() allocated a fresh JFJochReaderRawImage per frame, and on a
miniCBF sweep its buffer holds the whole decoded image: 72.6 MB on a 4150 x
4371 detector. That is over glibc's 32 MB mmap ceiling, so the block is mmapped
and munmapped every image and the decoder faults in 17 700 untouched pages as it
writes them. Measured on one 18.1 Mpx frame, decoding into a fresh buffer takes
61 ms against 16 ms into one that has been written before - the allocation costs
nearly three times what the decode does, and the per-image loop pays it twice
per sweep.

ReadRawImage() fills a raw image the caller owns, so a worker declares one
outside its loop and keeps the pages. GetRawImage() stays as the allocating
wrapper for the single-image callers. The two loops that hand the image to the
process-file writer keep a shared_ptr and recycle it only while the writer
thread is not still reading the previous frame's pixels.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

miniCBF: read the compressed file through a buffer the caller keeps ... 391736425f

Slurp() returned a fresh std::vector per call, so every frame allocated the
whole compressed file - 18 MB on a 16 Mpx detector - and value-initialised it
before the read overwrote every byte. Measured on one such frame that is 9.2 ms
against 5.4 ms into a buffer that is already there.

ReadInto() now takes the scratch, and the raw image carries it, so a worker
reading a sweep reads every file into the same bytes. Slurp() fills a buffer
instead of returning one, which is also what makes the resize cost nothing when
the next file is the same size as the last.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

preprocessing: page-lock the reader's bytes for an uncompressed image ... ffbdaa71e4

PinInputBuffer() page-locks the buffer an image is decompressed into, which an
uncompressed image never uses: it is read straight out of the reader's own
buffer, so the upload came from pageable memory. Measured on this card, 72.6 MB
crosses at 5.6 GB/s pageable and 14.6 GB/s registered - 12.9 ms against 5.0 ms,
on every image of a sweep that stores its frames uncompressed.

PinInputRegion() page-locks a region the caller owns and remembers it, so a
worker that reads every frame into the same buffer registers it once. The
registration is dropped with the engine, so the three workers that build one
declare their raw image before it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

miniCBF: read the sweep's rotation angles on several threads ... 10435ef588

Opening a sweep parses the header of every file to get its start angle, and one
header is two dozen std::regex searches: measured at 0.33 ms on a 16 Mpx sweep
and 1.0 ms on one with a longer header, so a 7200-frame sweep spent 7.2 s on one
thread before it read an image. The files are independent, so the scan is shared
out over up to eight of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

miniCBF: write the decoded pixels past the cache ... 03a534fb66

Nothing on the host reads a decoded image back: it is DMAed to the GPU, or a CPU
engine walks it once and drops it. An ordinary store pays for that twice - it
reads every cache line before overwriting it, and it evicts 72 MB of live cache
to make room - and the decode is bound by exactly that traffic, which is why
neither an eight-pixel-at-a-time scalar loop nor an SSE2 prefix sum moved it at
all. Storing the pixels non-temporally takes an 18 Mpx frame from 27 ms to
11 ms, and 240 frames from six sweeps decode byte for byte as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

broker: the second use of the removed indexing capability flag goes too ... 10ccc26db2

Removing the flag left one caller behind, so the broker did not compile. The
setter's gate was the same one the constructor had already dropped: with the CPU
indexer always compiled in there is no configuration where indexing is
unavailable, so a setting that asked for it never has to be refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

beam centre: the metric symmetry the two centres disagree about is decided on the merge ... 73e2382be5

The beam-centre check indexes a second first pass at the centre the scattered
background measures. Where the two centres return the same cell but a different
metric symmetry read off it, the run kept the file's and said the difference did
not decide the cell. It does not decide the cell; it decides the CLASS, and the
class is handed to GeometryRefiner, which holds the cell to it for the whole
run - so a tetragonal promotion the centre manufactured averages two axes that
differ, and the run refines, predicts, integrates and merges on an equality the
crystal does not have. Measured on a small-molecule rotation set whose two
in-plane axes differ by 2.35 %: 3.06 px of centre error, one edge 1.6 % wrong,
R_meas 57 %, ISa 1.2, and the axial row that carries the third screw never
measured.

Whether the centre could have decided it is arithmetic, and needs no new
constant. The promotion holds a pair of reduced-cell axes equal to
LatticeSearch's dist_tolerance; a centre wrong by d px moves an axis of length a
by d*p*a/(D*lambda) relative, and both axes of the pair carry their own error.
Where their sum reaches the tolerance the higher class is a statement about
where the beam was assumed to be, not about the crystal.

Nothing before integration can settle which centre is right - both index the same
frames, and the frame count is the one statistic this check already refuses to
arbitrate on - and the disagreement is two-sided, the file's centre finding the
higher class about as often as the measured one does over the corpus. So both
lattices are carried forward: the first pass is run again at the other centre and
the run adopts whichever arm MERGES better, on each arm's own search merge (P1,
whole range, before any correction surface - the terms the two passes are already
compared in). The alternative arm is a whole first pass, so it gets the short-axis
pass and the geometry post-refinement of its own; on a crystal with an axis under
the FFT floor the check runs before that pass and compares supercells, and the
arbitration has to be carried past it. Moving off the geometry the file gave
takes the same CC1/2 band the pass-2 quality guard uses.

On the target set the rescue is outright: cell 5.787 5.940 12.263 against
5.838 5.838 12.251, 100 % indexed, R_meas 57.0 -> 2.3 %, CC1/2 0.865 -> 0.993,
I/sigma 16.2 -> 84.1, P212121 adopted on its own axial rows rather than on a tie
between two empty zones. Of the eight other corpus runs that reach this branch,
one does not trigger and six keep the file's centre with bit-identical output;
one adopts the measured centre and returns the same space group and cell to
0.07 % with R_meas 11.6 -> 11.5 % and ISa 25.1 -> 25.2. The branch costs one
extra first pass on the runs that trigger.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

preprocessing: a buffer someone else page-locked is already what we wanted ... bc816eeccb

The uncompressed-image upload page-locks the caller's bytes, which is right when
the caller allocated them for us and wrong when it did not: the online receiver
registers its image buffer once and hands out ranges inside it, and registering
a range a second time is an error rather than a no-op, so the first frame
aborted the collection.

An already-registered range needs nothing done to it - the upload is a DMA
either way - and it is not ours to unregister when this object is destroyed, so
it is deliberately not remembered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

release: version 1.0.0-rc.168 ... 31422ddf54

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

beam centre: a walk that adopts nothing is a walk not worth taking ... 672840dfb8

The beam-centre hypothesis ladder runs up to 48 full first-pass indexing
attempts, a pixel at a time, and adopts the first rung that indexes half the
validation frames. When it adopts nothing the centre is restored and the run is
bit-identical to one that never searched, so on those datasets the whole walk is
pure cost - measured at 347 s of the 429 s the loop spends over a 100-dataset
corpus.

Two exits, both measured against the six corpus datasets that enter the walk.
The walk is skipped outright when the background already measures a centre that
is significant at three sigma and further away than the ladder can reach: it
cannot get there, so it only spends the time finding that out. And it stops
after the second ring when the running maximum, the starting centre included, is
below a third of the adoption bar, which separates cleanly on this corpus - the
three walks that adopt nothing peak at 0, 2 and 4 frames, the two that succeed
sit at 17 and 26 by that point.

Every dataset that adopted a centre before still adopts the same one, with the
same cell and the same reflection count; the two that fail still fail with the
same message. The measured saving is 226 s of the 429.

rotation passes: a pass already known to be superseded does not write the report ... 0a349026bb

A rotation run can integrate more than twice: a pass whose post-refined geometry
loses the quality comparison, or whose integration radius has to be refixed, is
followed by another that overwrites it. The superseded pass nonetheless computed
a Wilson B, an anisotropy and twinning verdict, a radiation-damage and
sweep-quality report, ran model validation and wrote the P1 cross-check file -
all of it replaced minutes later. On a 100-dataset corpus that is 19 superseded
passes over 18 datasets and about 100 s.

A pass cannot know it is the last one, but it can know it is not: every trigger
that forces another pass is settled before the report is built - background
starvation at integration, a lattice that conflicts with a fixed space group at
indexing, and the quality comparison at the pass's own first merge. The one
exception compares cells rather than the pass's own numbers, so it is left out;
it fires on none of the corpus.

The quality comparison is hoisted into one function with two callers rather than
being written twice. The measurement the error model and the correction surfaces
are fitted on is not skipped: full_stats also gates the surface refinement, so
declining it would change the merged intensities rather than only the report.

Verified on nine datasets spanning seven space groups, two of them with a
superseded pass and two with none: the merged and unmerged files, the P1
cross-check and the image statistics are byte for byte identical, and the report
differs only in its date, prefix, command line and wall time.

leonarski_f merged commit 77bc0cfe52 into main 2026-09-10 13:51:16 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) mx-gitea perl_d (perl_d) takaba_k (takaba_k) wang_m (wang_m)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#78