v1.0.0-rc.168 #78

Merged

leonarski_f merged 18 commits from rc168 into main 2026-09-10 13:51:16 +02:00

Conversation 0 Commits 18 Files Changed 174

+1097 -488

18 Commits

Author	SHA1	Message	Date
leonarski_f	0a349026bb	rotation passes: a pass already known to be superseded does not write the report ... A rotation run can integrate more than twice: a pass whose post-refined geometry loses the quality comparison, or whose integration radius has to be refixed, is followed by another that overwrites it. The superseded pass nonetheless computed a Wilson B, an anisotropy and twinning verdict, a radiation-damage and sweep-quality report, ran model validation and wrote the P1 cross-check file - all of it replaced minutes later. On a 100-dataset corpus that is 19 superseded passes over 18 datasets and about 100 s. A pass cannot know it is the last one, but it can know it is not: every trigger that forces another pass is settled before the report is built - background starvation at integration, a lattice that conflicts with a fixed space group at indexing, and the quality comparison at the pass's own first merge. The one exception compares cells rather than the pass's own numbers, so it is left out; it fires on none of the corpus. The quality comparison is hoisted into one function with two callers rather than being written twice. The measurement the error model and the correction surfaces are fitted on is not skipped: full_stats also gates the surface refinement, so declining it would change the merged intensities rather than only the report. Verified on nine datasets spanning seven space groups, two of them with a superseded pass and two with none: the merged and unmerged files, the P1 cross-check and the image statistics are byte for byte identical, and the report differs only in its date, prefix, command line and wall time.	2026-09-10 10:01:29 +02:00
leonarski_f	672840dfb8	beam centre: a walk that adopts nothing is a walk not worth taking ... The beam-centre hypothesis ladder runs up to 48 full first-pass indexing attempts, a pixel at a time, and adopts the first rung that indexes half the validation frames. When it adopts nothing the centre is restored and the run is bit-identical to one that never searched, so on those datasets the whole walk is pure cost - measured at 347 s of the 429 s the loop spends over a 100-dataset corpus. Two exits, both measured against the six corpus datasets that enter the walk. The walk is skipped outright when the background already measures a centre that is significant at three sigma and further away than the ladder can reach: it cannot get there, so it only spends the time finding that out. And it stops after the second ring when the running maximum, the starting centre included, is below a third of the adoption bar, which separates cleanly on this corpus - the three walks that adopt nothing peak at 0, 2 and 4 frames, the two that succeed sit at 17 and 26 by that point. Every dataset that adopted a centre before still adopts the same one, with the same cell and the same reflection count; the two that fail still fail with the same message. The measured saving is 226 s of the 429.	2026-09-10 10:01:17 +02:00
leonarski_fandClaude Opus 5	31422ddf54	release: version 1.0.0-rc.168 ... Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-10 07:43:12 +02:00
leonarski_fandClaude Opus 5	bc816eeccb	preprocessing: a buffer someone else page-locked is already what we wanted ... The uncompressed-image upload page-locks the caller's bytes, which is right when the caller allocated them for us and wrong when it did not: the online receiver registers its image buffer once and hands out ranges inside it, and registering a range a second time is an error rather than a no-op, so the first frame aborted the collection. An already-registered range needs nothing done to it - the upload is a DMA either way - and it is not ours to unregister when this object is destroyed, so it is deliberately not remembered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-10 07:25:57 +02:00
leonarski_fandClaude Opus 5	73e2382be5	beam centre: the metric symmetry the two centres disagree about is decided on the merge ... The beam-centre check indexes a second first pass at the centre the scattered background measures. Where the two centres return the same cell but a different metric symmetry read off it, the run kept the file's and said the difference did not decide the cell. It does not decide the cell; it decides the CLASS, and the class is handed to GeometryRefiner, which holds the cell to it for the whole run - so a tetragonal promotion the centre manufactured averages two axes that differ, and the run refines, predicts, integrates and merges on an equality the crystal does not have. Measured on a small-molecule rotation set whose two in-plane axes differ by 2.35 %: 3.06 px of centre error, one edge 1.6 % wrong, R_meas 57 %, ISa 1.2, and the axial row that carries the third screw never measured. Whether the centre could have decided it is arithmetic, and needs no new constant. The promotion holds a pair of reduced-cell axes equal to LatticeSearch's dist_tolerance; a centre wrong by d px moves an axis of length a by d*p*a/(D*lambda) relative, and both axes of the pair carry their own error. Where their sum reaches the tolerance the higher class is a statement about where the beam was assumed to be, not about the crystal. Nothing before integration can settle which centre is right - both index the same frames, and the frame count is the one statistic this check already refuses to arbitrate on - and the disagreement is two-sided, the file's centre finding the higher class about as often as the measured one does over the corpus. So both lattices are carried forward: the first pass is run again at the other centre and the run adopts whichever arm MERGES better, on each arm's own search merge (P1, whole range, before any correction surface - the terms the two passes are already compared in). The alternative arm is a whole first pass, so it gets the short-axis pass and the geometry post-refinement of its own; on a crystal with an axis under the FFT floor the check runs before that pass and compares supercells, and the arbitration has to be carried past it. Moving off the geometry the file gave takes the same CC1/2 band the pass-2 quality guard uses. On the target set the rescue is outright: cell 5.787 5.940 12.263 against 5.838 5.838 12.251, 100 % indexed, R_meas 57.0 -> 2.3 %, CC1/2 0.865 -> 0.993, I/sigma 16.2 -> 84.1, P212121 adopted on its own axial rows rather than on a tie between two empty zones. Of the eight other corpus runs that reach this branch, one does not trigger and six keep the file's centre with bit-identical output; one adopts the measured centre and returns the same space group and cell to 0.07 % with R_meas 11.6 -> 11.5 % and ISa 25.1 -> 25.2. The branch costs one extra first pass on the runs that trigger. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 20:20:48 +02:00
leonarski_fandClaude Opus 5	10ccc26db2	broker: the second use of the removed indexing capability flag goes too ... Removing the flag left one caller behind, so the broker did not compile. The setter's gate was the same one the constructor had already dropped: with the CPU indexer always compiled in there is no configuration where indexing is unavailable, so a setting that asked for it never has to be refused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 20:17:16 +02:00
leonarski_fandClaude Opus 5	03a534fb66	miniCBF: write the decoded pixels past the cache ... Nothing on the host reads a decoded image back: it is DMAed to the GPU, or a CPU engine walks it once and drops it. An ordinary store pays for that twice - it reads every cache line before overwriting it, and it evicts 72 MB of live cache to make room - and the decode is bound by exactly that traffic, which is why neither an eight-pixel-at-a-time scalar loop nor an SSE2 prefix sum moved it at all. Storing the pixels non-temporally takes an 18 Mpx frame from 27 ms to 11 ms, and 240 frames from six sweeps decode byte for byte as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 20:14:04 +02:00
leonarski_fandClaude Opus 5	10435ef588	miniCBF: read the sweep's rotation angles on several threads ... Opening a sweep parses the header of every file to get its start angle, and one header is two dozen std::regex searches: measured at 0.33 ms on a 16 Mpx sweep and 1.0 ms on one with a longer header, so a 7200-frame sweep spent 7.2 s on one thread before it read an image. The files are independent, so the scan is shared out over up to eight of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 20:14:04 +02:00
leonarski_fandClaude Opus 5	ffbdaa71e4	preprocessing: page-lock the reader's bytes for an uncompressed image ... PinInputBuffer() page-locks the buffer an image is decompressed into, which an uncompressed image never uses: it is read straight out of the reader's own buffer, so the upload came from pageable memory. Measured on this card, 72.6 MB crosses at 5.6 GB/s pageable and 14.6 GB/s registered - 12.9 ms against 5.0 ms, on every image of a sweep that stores its frames uncompressed. PinInputRegion() page-locks a region the caller owns and remembers it, so a worker that reads every frame into the same buffer registers it once. The registration is dropped with the engine, so the three workers that build one declare their raw image before it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 20:14:04 +02:00
leonarski_fandClaude Opus 5	391736425f	miniCBF: read the compressed file through a buffer the caller keeps ... Slurp() returned a fresh std::vector per call, so every frame allocated the whole compressed file - 18 MB on a 16 Mpx detector - and value-initialised it before the read overwrote every byte. Measured on one such frame that is 9.2 ms against 5.4 ms into a buffer that is already there. ReadInto() now takes the scratch, and the raw image carries it, so a worker reading a sweep reads every file into the same bytes. Slurp() fills a buffer instead of returning one, which is also what makes the resize cost nothing when the next file is the same size as the last. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 20:14:04 +02:00
leonarski_fandClaude Opus 5	4bac6d3f21	reader: a worker reads every frame into the same raw image ... GetRawImage() allocated a fresh JFJochReaderRawImage per frame, and on a miniCBF sweep its buffer holds the whole decoded image: 72.6 MB on a 4150 x 4371 detector. That is over glibc's 32 MB mmap ceiling, so the block is mmapped and munmapped every image and the decoder faults in 17 700 untouched pages as it writes them. Measured on one 18.1 Mpx frame, decoding into a fresh buffer takes 61 ms against 16 ms into one that has been written before - the allocation costs nearly three times what the decode does, and the per-image loop pays it twice per sweep. ReadRawImage() fills a raw image the caller owns, so a worker declares one outside its loop and keeps the pages. GetRawImage() stays as the allocating wrapper for the single-image callers. The two loops that hand the image to the process-file writer keep a shared_ptr and recycle it only while the writer thread is not still reading the previous frame's pixels. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 20:14:04 +02:00
leonarski_f	dcc6d5e4fc	JFJochStateMachine: remove confusing indexing_possible setting	2026-09-09 19:33:35 +02:00
leonarski_fandClaude Opus 5	2e5c0df0a5	scaling: the decay / relative-B passes read a compact array, and their per-group references run on all threads ... The three step-4b passes - the global decay slope, the per-batch relative-B and the radiation-damage monitor - were plain serial walks over the eighty-byte observation records, a dozen of them, each reading twenty bytes of each record and each allocating and zeroing a pair of per-group vectors. The decay fit alone runs seven of those passes (three slope fits, four held-out scores) on one thread while the workers and the GPU idle. Give them the treatment the correction surfaces below them already have: pull the fields they read out of the records once into a compact array in fulls order, and keep a second copy of the same terms in ASU-group order (stable counting sort) so a thread can own whole groups. The per-group inverse-variance references then run on all threads and still add each group's terms in fulls order, and the remaining scalar reductions keep their serial order over the compact array - so every sum is formed from the same terms in the same order as before, and the numbers are unchanged rather than merely close. The build is shared by the three curve fits and the three held-out scores of the relative-B pass instead of being repeated per call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 19:06:40 +02:00
leonarski_fandClaude Opus 5	58a9e24484	merge: the report-only error-model diagnostics are measured only where they are read ... The strong-reflection ISa asymptote and the reported chi2 median are both report-only - the header already states that a merge run with full_stats false skips the diagnostics, because its numbers are never read - yet both ran on every merge, including the geometry pre-pass and the pre-correction CC1/2 merge that is discarded. The asymptote is the expensive one: a per-group scatter array value-initialised on one thread (hundreds of megabytes at a million ASU groups) plus a serial random-scatter walk over every full. Gate both on full_stats, and leave them out of the error-model line on the merges that no longer measure them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 19:06:40 +02:00
leonarski_fandClaude Opus 5	b43f72a676	merge: the half-set cell array is the one the flat fields already hold ... The merge built a per-observation cell array with the same rule the flat merge fields had just been filled with - `usable_merge(o) ? o.group : -1` - but serially, over the eighty-byte observation records, so a large P1 pass dragged more than a gigabyte through the cache to rebuild an array that was already in hand. Hand the existing one to the half-set assignment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 19:06:39 +02:00
leonarski_fandClaude Opus 5	7db76560a6	beam centre: the background fit stops at its fixed point, and reads the pixels on every thread ... The background ring fit is the largest single phase of an offline run on a large detector: 626 s of a 5403 s hundred-dataset battery, against 127 s for the beam-stop projection it reads. The cost is not spread over its invocations - the worst single invocation per dataset accounts for 516 s of that 626 s - and the worst one is always the second pass, at the post-refined geometry, where the fit starts essentially at the answer. Instrumented per iteration, that invocation is not a truncated walk. It reaches its answer at iteration 4 and then enters a period-2 limit cycle: the step alternates in sign every iteration at a fixed 0.30 px, and the centre oscillates by 0.08 px about a value it knows to 1.75 px, for the 96 iterations left in the budget. CONVERGED_PXL sits below the cycle amplitude, so the run can never meet it; whether a dataset costs 5 s or 55 s is decided by whether its cycle happens to fall under 0.02 px. Travel looks nothing like that - a fit forced to walk 339 px keeps its step direction through all 22 of its travelling iterations and takes steps of 75 px down to 0.1 px - so the sign of the step against the one before it separates the two without any length to compare against. Two changes, measured apart. The walk now also stops when its step reverses the previous one twice running, which is a crossing of the fixed point rather than a step toward it. MAX_ITERATIONS stays at 100, so the long-travel budget is untouched: the forced 339 px walk arrives as before, in 27 iterations instead of 29, 0.013 px away. A monotone fit is bit-identical. The two cycling invocations measured go from 100 iterations to 7 and 9, moving 0.11 and 0.16 px - under 6% of the sigma each reports, and the hundredth iteration was itself an arbitrary point on the cycle. The per-pixel passes - a rotation, a sqrt, two atan2 and two divisions each, over 18 Mpx, three times an iteration - now run on the threads the caller was given, split over a fixed 64 row blocks folded in block order, so the grouping of the sums is a property of the image and not of the machine. Serial against six threads on the same projection in the same process: 4.2-5.3x, with beam_x, beam_y and sigma identical to four decimals on all five fits measured, and a test that asserts one thread and eight give the same answer. Together, measured on the same phase boundary the battery numbers use: 57.7 s to 2.4 s and 54.5 s to 3.0 s on the two worst datasets, 0.23 s to 0.10 s on a small one. The beam-stop mask is untouched - ShadowFinder is not modified and the fit runs after it and writes nothing back - and both masks come out at exactly the pixel counts the unmodified binary logged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 18:40:45 +02:00
leonarski_fandClaude Opus 5	e74517c2fb	performance: the French-Wilson quadrature stops recomputing what its grid already fixes ... The posterior is integrated on j = (i + 1/2) * dj, so sqrt(j) and log(j) each split into a per-reflection factor and a term that depends only on i: sqrt(j) = sqrt(dj) * sqrt(i + 1/2) and log(j) = log(dj) + log(i + 1/2). The i-dependent halves are the same for every reflection and every intensity, so they are tabulated once for the run rather than recomputed at each of the 400 grid points. The exponential in the second pass is left alone, but a point more than 37 below the peak carries a weight under 1e-16 and cannot move a normalised sum of doubles, so it is skipped instead of exponentiated - and the posterior is sharply peaked, so most of the grid is skipped. Finally the Wilson prior and the centric flag belong to the reflection rather than to one of its three intensities, so the two symmetry lookups are made once and shared instead of three times over. None of this changes the arithmetic: on five datasets spanning the corpus's speed range the space group, the unique-reflection count, R_meas and CC1/2 are identical to the digit. Measured over those five, each run twice and the two binaries interleaved so machine load cannot favour one arm: 771.2 s to 730.7 s, 5.3 % of total wall, between 3.2 % and 7.8 % per dataset and in the same direction on all five. The work removed is largest where the run merges most - the quadrature runs on every weak intensity of every merge, and a large cell at high resolution reaches a few million of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-09 14:13:31 +02:00
leonarski_fandClaude Opus 5	11c15837c6	rugnux: acknowledge the field and the open-source stack ... The crystallographic methods rugnux implements were developed and published by the X-ray research community, and the program is built on other people's open-source software. Both were credited only in files that never left the source tree: jfjoch_viewer shows LICENSE and ACKNOWLEDGEMENT.md from its own resources, but rugnux said nothing and the packages shipped no ACKNOWLEDGEMENT.md at all. One text, defined once, printed by the CLI after the licence banner and written at the foot of every <prefix>_report.txt, so a result carries the acknowledgement wherever it travels and names the files that hold the detail. ACKNOWLEDGEMENT.md joins LICENSE and THIRD_PARTY_NOTICES.md in JFJOCH_NOTICE_FILES, which is installed per component - so it reaches the Linux .tgz, the Windows .zip, the NSIS installer, the macOS image and the DEB/RPMs alike, next to the two files it links to. Also drops the SUMMARY block's explanatory paragraph. It was written to whoever maintains the report format, not to the crystallographer reading their results, and the greppability contract it described is already in docs/RUGNUX_REPORT.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018FeM2CtemFrbhoK1G9n7J5	2026-09-09 13:39:14 +02:00