1165 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
jungfrauandClaude Opus 5	ca3ca7170e	Spread the scaling corrections and the space-group search over the cores ... Two thirds of a rotation run is one thread. The image loop is not the problem - on the heaviest crystal of the battery it is 1.8 s of 40 - and neither GPU nor CPU is saturated, because while the corrections and the space-group search run there is one core working and 47 idle. Mean occupancy over the whole run: 3.9 of 48. In the correction surfaces (absorption in the goniometer frame, detector-plane modulation, absorption against time and detector position - all one function): the per-cell accumulation, the score reduction and the final apply are now chunked, as are the three loops that assign a full to its cell, one of which spends a sine and a cosine per full de-rotating it into the crystal frame. Two full sorts of four million floats went with them: only the nine bin edges are wanted, so they are selected instead, each selection starting where the last one left off. The per-group pass is deliberately left serial. The terms of one group are spread all over the list, so the only way to give a thread groups of its own is to walk in group order, and that trades a near-sequential read of the fulls for a random one over a few hundred megabytes - the trade that already lost once in the combine kernel. The space-group search scores each candidate rotation by correlating I(h) against I(Rh) over the whole merge. Every operator it can ask about comes from a fixed list and none of them depend on each other, so they are scored up front, in parallel, and the search reads the cache. The scratch that stops a pair being counted twice is now per worker rather than shared. Worker counts are gated on how much work there is, not on how many cores the machine has (ThreadsForWork). Both parallel helpers start a thread per chunk, so a small dataset on a large node would otherwise pay for 48 thread starts to sum a few thousand terms - and this runs on 8-core laptops as well as on this node. Measured on the heaviest crystal, idle machine, two runs each, summed over both passes: those phases go 7.88 s -> 5.19 s. Whole-run wall time is the wrong ruler for it - it moves +-4 s between identical runs. Battery 9m45s -> 9m23s, space group 21/24, no failures; 16 of 24 crystals bit-identical to the previous run and the rest inside the noise floor of running one binary twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-23 08:25:43 -04:00
jungfrauandClaude Opus 5	01b619cfd0	Take the double precision out of the box integrator's inner loops ... boxsum summed its ring background in double and compared each ring pixel against a double threshold. The pixels are integers: a sum of at most a thousand int32 values is exact in a 64-bit integer AND exact in a double, so the two agree bit for bit, and comparing an integer against the floor of the threshold accepts exactly the same pixels as comparing it against the threshold itself. Both loops now do integer arithmetic. That was 39% of the card's double-precision pipe on the development machine and about three quarters of it on the production one, where the double rate is unchanged from Turing while the single rate has doubled - so this is worth more there than here. Alongside it, three things in the combine kernel. rr_nusable was computed by a whole extra walk over every observation and then never downloaded or read by anything. sum_wb and sum_cwb have no F in them, so they are the same in all three reweights and only the last round's values are ever used - two thirds of them were two divisions each, discarded. And CombineParams was the one parameter struct in the file without __restrict__, so the compiler could not assume the observation arrays and the freshly allocated fulls arrays were distinct. Measured on a crystal with 66 million partial observations: boxsum 12.2 s -> 8.3 s, the combine kernel 8.0 s -> 7.6 s, whole crystal 1m17s -> 1m12s. Battery 15m32s -> 9m59s. Same space group on all 24 crystals, none failed. Two things measured and NOT kept, recorded so they are not tried again: sorting the raw-hkl runs by length so a warp holds runs of similar length - it trades away the locality of neighbouring runs in the permutation and came out slower (7.6 s -> 8.8 s); and page-locking the integrator's host staging arrays individually - eleven separate registrations of small heap allocations overlap on shared pages and the driver refuses them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 08:24:21 -04:00
jungfrauandClaude Opus 5	a8cca3e5d4	Parallelise the incident-flux divide, drop a redundant sync ... DivideOutIncidentFlux was still the last fully serial pass in Ingest: a sweep over every observation to take each frame's mean background, and another to divide every rlp by its frame's flux. Ten gigabytes of traffic on one thread. The per-frame means go a frame at a time rather than an observation at a time, so each frame's running sum stays in one thread and in the order it had - splitting by observation would cut a frame across two threads and the partial sums would have to be recombined, which is a different sequence of roundings. The divide is per-element and splits anywhere. The adaptive spot finder synchronised after flagging strong pixels. The extractor that reads those pixels runs on the same stream, so the ordering already guaranteed the flagging had finished; the wait only idled the host, once per image. Measured on a crystal with 66 million partial observations: Ingest 8.5 s and 7.7 s -> 7.1 s and 6.6 s, whole crystal 1m24s -> 1m17s. Merged statistics unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 08:24:16 -04:00
jungfrauandClaude Opus 5	8ae53b7fdf	Say what actually makes the post-refine bucket sort reproducible ... The bucketing commit justified itself with "the partials order became total in an earlier commit", which is true of the scale/merge ingest and not of this sort: part_less ends at the image number, so two partials of one reflection on one image tie, exactly as they did before. Nothing is wrong with the result. The counting-sort prefix lays each bucket out chunk by chunk, and a chunk is a contiguous span of the gathered order, so every bucket arrives at std::sort in global gather order no matter how many threads scattered it - the order is reproducible run to run and identical across -N. Giving Partial a rank field to make the comparator total would settle those ties by index instead, at eight more bytes on an array that reaches tens of millions of elements, and would change nothing anyone can observe. So state the invariant where the comparator is, rather than leaving the next reader to trust a claim that does not hold for this half. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n8riB6X59oRjkrSHzNPAU	2026-08-23 08:23:57 -04:00
jungfrauandClaude Opus 5	01d16231b3	Sort the partials and the post-refine events in buckets, in parallel ... Both were one std::sort on one thread over tens of millions of elements, and together they were a third of a crowded crystal's run. Bucketing by h first makes them parallel. h is the comparator's leading key, so the sorted array is exactly the buckets laid end to end, and each bucket sorts on its own thread. In Ingest the keys are built straight into their bucket slot, so this replaces the build pass rather than adding one and the packed-key array is never duplicated; the extra memory is a few hundred kilobytes of histograms. Buckets are taken largest first, because the tail of the phase is whichever bucket finishes last. The run split falls out of the same structure for free: a run of equal (h,k,l) never crosses an h boundary, so each bucket counts its own runs, a scan over the buckets gives the offsets, and the arrays are sized exactly - which also removes the repeated growth the push_backs were paying for. The h range comes from the finiteness pass, which already reads every observation. The partials order became total in an earlier commit, when the observation index was added as the last key. That is what makes this safe rather than merely fast: the permutation is uniquely determined, so a bucket sort produces the same one a single sort would. Measured on a crystal with 66 million partial observations: Ingest 15.2 s and 14.3 s -> 8.3 s and 7.4 s, the post-refine event sort out of the top ten gaps entirely, the whole crystal 2m22s -> 1m24s. Battery 15m32s -> 10m05s. Same space group on all 24 crystals, none failed, and no crystal's R_meas moved by more than 0.3 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 08:22:40 -04:00
jungfrauandClaude Opus 5	c1b85c7e88	Reduce within the warp before the Bragg integration atomics ... fit and boxsum were 80% of GPU time on a crowded crystal - 78 s of it. Neither was bandwidth- or occupancy-bound: both sat at about an eighth of the issue rate the card can sustain, stalled. What stalls them is the block-wide accumulations. Every one has all 128 lanes of the block adding into one shared address, and a shared-memory atomicAdd on a float or a 64-bit integer has no instruction on either Turing or Ada - it compiles to a compare-and-swap retry loop. So those 128 lanes serialise into 128 retries, eighteen times per thread in fit. Summing across the warp first and letting one lane do the atomic leaves four per block instead of 128. That is the whole story: the arithmetic below was worth 2%, the atomics 5.6x. The arithmetic is still worth having, and is what was expected to matter: - compute_shell ran on all 128 threads of a block for a value that belongs to the reflection. It is two software double-precision divisions, on a card whose double throughput is a thirty-second (a sixty-fourth on the production one) of its single. One thread does it now. - The Kabsch inner loop divided by the same weight three times; the compiler emits the whole correctly-rounded sequence each time. One reciprocal now. Likewise the two Gaussian widths and the profile normalisation, which are constant over a reflection's cells and were divided per cell. - boxsum read the pixel before deciding whether it wanted it. The window is the bounding box of an ellipse, so nearly half of it is neither the signal disk nor the background ring, and those slots were fetching a cache line for nothing. Measured: fit 50.8 s -> 9.1 s, boxsum 27.5 s -> 12.1 s. A crowded crystal 2m22s -> 1m58s, a 16M-pixel one 39.5 s -> 37.2 s, the whole battery 12m30s -> 11m35s. Same space group on all 24 crystals, none failed. The integer sums are unchanged - addition is associative. The float ones move in their last bits and become more reproducible, since a fixed shuffle tree replaces whatever order the atomics arrived in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 08:22:16 -04:00
jungfrauandClaude Opus 5	52756273e1	Make the partials order total, and hoist 1/sigma out of the IRLS loop ... The sort that orders every observation by (h,k,l,image_number) was not a total order: two observations can genuinely share all four. The predictor emits BOTH intersections of a reflection's rotation circle with the Ewald sphere, and near the blind region - where zeta is smallest - the two are close enough in angle that both are accepted on the same frame. Which of them came first was then whatever the sort happened to produce. That was observable. The combine takes on_ice from the FIRST member of a rocking event, so the order decided whether a full was flagged as ice at all, and its per-event sums are floating point, so it moved intensities in their last bits. The observation's own index is now the final key, which orders them by arrival - and, more usefully, makes the order unique, so it no longer depends on which algorithm sorted it. sigma never changes once it is uploaded, so 1/sigma is the same in all thirty IRLS iterations of all three scaling iterations of all five scaling passes. It was being recomputed every time: a 64-bit reciprocal is a hardware estimate plus five refinement steps, and the profile put the three divisions in that loop at 21 of its 31 double-precision instructions. It is computed once now, in the pass that already streams every observation. The CPU has always hoisted it; this is the GPU catching up. Same expression on the same operand, so the value is what the loop used to compute, bit for bit. Also: PrepScaleObsKernel is not a grid-stride loop, but the scale-fulls path capped its grid at 65535 blocks like the grid-stride kernels around it. Above 16.8 million fulls that silently left the tail of sco_coeff/sco_ok stale. No dataset here reaches it; the cap is simply wrong for that kernel. And the AoS-to-SoA staging that feeds the GPU - the widest pass in Ingest, reading an 80-byte struct and writing fourteen arrays out of it - ran on one thread. Full 24-crystal battery: same space group on all 24, none failed, one crystal moved R_meas by 0.8 points with CC unchanged (it moves by that much between runs of an identical binary). 15m32s -> 13m35s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 08:22:16 -04:00
jungfrauandClaude Opus 5	594100accc	Make the rotation-scale fit reproducible, and stop refining past a float ... Two follow-ups to the closed-form fit. The five per-fifth sums were reduced under a mutex, so the order in which the chunks were added depended on which worker reached the lock first and the fitted scale moved in its last bits between runs of the same binary. Each chunk now folds into its own slot and the slots are summed in chunk order, which is the reduction pattern the rest of the analysis code uses. The split ParallelChunks makes is fixed, so the sum is now the same sequence every time. The golden section bracketed to 1e-9. The fit is narrowed to a float before it is applied, and a float's epsilon is 6e-8, so the last ten or so iterations - each a full parallel pass over every event - refined digits that are discarded on the next line. Bracket to 1e-7. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n8riB6X59oRjkrSHzNPAU	2026-08-23 08:22:00 -04:00
jungfrauandClaude Opus 5	c4c2d598d7	Fit the goniometer rotation scale in closed form ... The fit has ONE parameter, and it was handed to Ceres as one residual block per rocking event - 8 million of them on a large crystal. Each block is a functor, an auto-diff cost function and a loss object on the heap, and the solver then factorises an 8-million-by-one Jacobian on every iteration. It cost 13.7 s. The residual is closed-form in k. A rotation preserves length, so |p_lab| is |e_mid| whatever k is and only the z component moves; Rodrigues gives it exactly: r(k) = C + A cos(a k) - B sin(a k) = C + R cos(a k + psi) C = lambda |e|^2 / 2 + u_z (u.e), A = e_z - u_z (u.e), B = (u x e)_z with a the event's angle from the sweep centre. That is the same function the functor computes - Ceres uses the exact Rodrigues form here, so there is no small-angle branch to disagree with - and it reduces the fit to minimising a smooth function of one variable over the interval the solver was bounded to. It is scanned on a grid and then closed in by golden section; the objective's curvature jumps wherever an event crosses the Huber knee, which is why this is not a Newton iteration. The coefficients are computed in double and stored narrowed. Their rounding moves the minimiser by ~1e-10, and k is carried downstream as a float, so the committed value is the same to far more digits than anything reads. One pass over the events yields the five per-fifth partial sums, so the all-data fit and the five leave-a-fifth-out folds share it. That matters because the jackknife only runs when the fit is big enough to act on, and on a crystal that trips it the old code paid for six full solves. The partials gather ahead of it counted first and then filled instead of growing one vector by push_back tens of millions of times, which copied the whole thing on every doubling. Measured: unchanged verdict and k to five decimals on the regression crystals. Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 13m35s together with the scale/merge changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 08:20:48 -04:00
jungfrauandClaude Opus 5	c3d3161af0	Fold the beam-stop batch before its frame size changes ... ShadowAccumulatorGPU::Add sized the raw buffer before closing the pending batch, and EnsureRawCapacity assigned frame_bytes on entry. FoldPending strides `raw` by frame_bytes, so a frame of a different size arriving mid-batch made the already-decoded frames fold with the new stride: every pixel of the pending batch read from the wrong offset, silently, with no error. The depth-change branch that exists to handle exactly this ran one step too late to help. Fold first, then resize, then adopt the new stride. The batch also closes on a change of frame size, not only of pixel mode - a batch is one layout, and the mode alone does not fix the layout. The decoder was likewise built once from the first frame and never rebuilt, so it is now rebuilt when the frame size changes; without that the mixed-size path this commit repairs would still decode into a buffer of the wrong size. Also calls Gpu() once in ShadowFinder::AddImage instead of twice - it takes and releases a mutex each time, once per image. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011n8riB6X59oRjkrSHzNPAU	2026-08-23 08:20:37 -04:00
jungfrauandClaude Opus 5	ddf625d833	Decode and accumulate the beam-stop projection on the GPU ... The pre-scan decompressed its frames on the host and folded them into a per-pixel projection there. On a 16M-pixel detector that is 60 frames of 72 MB to decompress and 20 bytes per pixel to read and write back per frame - about 40 GB of memory traffic - and it was the whole cost of the phase once the mask was no longer the bottleneck. Only the compressed chunk crosses PCIe now. BSLZ4DecoderGPU already exposes the raw decoded bytes (Decode(), the path its own tests use), which is what this needs: the projection is defined on the RAW STORED COUNTS with the pixel type's sentinel skipped, not on the preprocessed image, so nothing here goes through the preprocessor. Sums, maxima and counts are integers, so the device result is identical to the host's rather than merely close. Frames are folded in batches of four. The fold reads and writes the whole accumulator whatever the batch holds, so per frame it was spending most of the bandwidth on the accumulator rather than on the data; four is where that stops mattering, and every frame beyond it is another full frame of device memory, which costs more in cudaMalloc - device-synchronizing - than it saves. The accumulator is built on a thread of its own. It allocates and clears several hundred megabytes, and doing that in the constructor stalled the caller before it had read its first frame. Frames the device cannot take - anything but bitshuffle+LZ4 - still go to a host shard, so a run mixing compressions needs no second code path, and a build without CUDA is unchanged. RotationScaleMergeGPU set the CUDA device in its constructor and never put it back. CUDA's current device is per-thread, so that silently re-pinned the calling thread for the rest of its life, and the destructor freed several gigabytes against whatever device happened to be current by then - CudaDevicePtr records no device of its own. Every entry point now sets the device on entry and restores it on exit. ParallelFor/ParallelChunks moved to common/ParallelFor.h; two files had copies and a third wants them. Measured on a 16M-pixel rotation dataset: pre-scan 4.78 s -> 2.37 s -> ~2.0 s, shadow unchanged at 139126 pixels (22143 on a 2M-pixel dataset). Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 14m49s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 08:19:20 -04:00
leonarski_fandClaude Opus 5	20edf55063	Changelog: note the spot-finding, indexing and GPU memory gains ... Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 13:16:02 +02:00
jungfrauandClaude Opus 5	312df30463	Score the first-pass validation frames together ... Two-pass rotation indexing picks between candidate lattices by forcing each one and counting how many of 60 validation frames it indexes. That count was a serial loop, and it is the single largest serial stretch outside scale/merge: 57% of the 4.9 s first-pass phase, a few Ceres solves per frame on one core while the other 47 and all four GPUs sit idle. The frames are scored together now. Each one's verdict is its own - the score is only how many index - and this is the same call the main image loop already makes from every one of its workers on this same IndexAndRefine, which writes nothing but unit_cells[] under its own mutex. With the candidate forced, GetLattice() returns it and the branch that would advance the indexer's own state is never reached. The offline solver stops on an iteration count rather than a clock, so a loaded machine cannot change a frame's verdict. The spot cache had to be filled first: it is a plain map filled on demand, and a lookup racing an insert is not something a map survives. Filling it stays serial and in frame order, so its contents do not depend on scheduling. First-pass scheme gaps on the heaviest crystal 4.93 s -> 2.06 s, with both schemes returning the same 60/60 they did before. Battery 9m23s -> 9m01s, space group 21/24, no failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-23 13:15:14 +02:00
leonarski_fandClaude Opus 5	83b33e19ee	Give each FFT direction a block and its histogram shared memory ... Cherry-picked from 2608-performance, restricted to the indexer: the same commit there also hoists a reciprocal out of a loop in the scaling code, which is not being touched on this branch. The histogram bins are unsigned integers and the counts stay below 2^24, so they convert to float exactly - the vote is bit-identical, and the indexing result with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 13:12:49 +02:00
jungfrauandClaude Opus 5	20ef59205b	Build the GPU engines a worker never uses on first use, not always ... Every worker thread built a full set of analysis engines. Two of them are never asked for on the offline path: the fixed-threshold spot finder, because detection is adaptive by default, and the azimuthal integrator, because the fused adaptive finder produces the profile as a by-product. They are still needed elsewhere - the broker defaults to non-adaptive detection, and --no-adaptive-spots asks for the finder - so they are built on first use rather than removed. A lazily built finder takes the current resolution mask on construction; without that it would find spots outside the limits it was never told about. The bitshuffle decoder sized its output buffer for the widest pixel type there is rather than the one the images actually have, holding a second full frame per worker on 16-bit data. It is sized from the image now and grows if a later frame needs more. The shared-table checksum runs over eight interleaved lanes. FNV's multiply is a loop-carried dependency, so one chain retires a byte every few cycles whatever memory bandwidth is spare, and every worker hashes tens of megabytes of geometry tables as it builds its engines - about 5% of all CPU samples on a 16M-pixel detector. Measured on a 16M-pixel rotation dataset: cudaMalloc 11314 -> 9474 calls and, with cudaFree, 117 s -> 78 s of aggregate thread time; both synchronise the whole device, so that time is spent blocking every other worker. Whole battery 15m32s -> 12m30s. Data quality against main, over 24 crystals and eight statistics each: the same space group on all 24, and every difference smaller than what two runs of an IDENTICAL binary produce (measured: 13 of 24 crystals reproduce exactly run to run, worst R_meas swing 5.5 points, against 4.6 points for main vs this branch). The float atomics in the reductions have always made this so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 13:12:36 +02:00
jungfrauandClaude Opus 5	495e2d752d	Read four pixels at a time in the ring reduction ... reduce_rings_shared was 69% of all GPU kernel time - 116.8 s of a 70 s run across four cards. It is not bandwidth bound: flag_strong streams the same two arrays through the same grid-stride loop and reaches 196 GB/s, while this reached 30. The difference is the shared-memory atomics. Lanes in a warp read consecutive pixels along a detector row, a ring is a few pixels wide, so most of a warp lands in a handful of rings and the atomics to each one serialise. Two changes. The block reads four pixels per thread as one 16-byte and one 8-byte transaction, and merges the ones that fall in the same ring in registers before touching shared memory. Consecutive pixels usually DO share a ring, so this is where the win is: a run costs one set of atomics instead of one per pixel. npix is not guaranteed to be a multiple of four - it is width x height on the converted path, and detectors are not obliged to be even - so the vector loop stops short and a scalar loop finishes the remainder. Reading past the end would not fault, which is worse than if it did: it would fold uninitialised device memory into the accumulators and move the detection threshold in a way that does not reproduce. And the grid is sized from the occupancy the device reports, per pass. The two passes have different shared footprints - the first carries the corrected rings as well - so they do not fit the same number of blocks, and a grid sized for one left the other running a second wave at a quarter occupancy. The comment that justified the old grid reasoned from 1536 threads per SM, which is an Ada number; the card it ran on holds 1024. The run totals are still exactly what they were. The accumulators are unsigned 64-bit, so summing a run in a register and adding it once is the same value as adding each pixel separately - addition mod 2^64 is associative, overflow included - which is what keeps the ring statistics, and therefore the detection threshold, independent of how the work was grouped. That is the property the integer accumulators exist for. (The run accumulators are unsigned for the same reason: signed overflow would be undefined, and four squares of a large pixel value reach 2^64.) The corrected float sums, which feed the reported profile rather than any decision, change in their last bits as they already did between runs. Measured on a 16M-pixel rotation dataset: the kernel 116.8 s -> 19.1 s (6.1x), no longer the largest; the whole run 70 s -> 39.5 s. Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 12m47s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 13:12:36 +02:00
jungfrauandClaude Opus 5	670831bad4	Stop allocating GPU and pinned memory nothing reads ... Three resource fixes and two latent bugs, none of which changes a computed number. The preprocessed image has a host copy that only a CPU engine ever reads. On the GPU path every engine reads the device buffer instead, and rugnux always runs the fused adaptive finder, so that host copy is allocated, zeroed and PAGE-LOCKED for nothing - 72 MB per worker, 3.5 GB over 48 of them, and a cudaHostRegister each, which the driver serializes. It is now skipped by the same condition that already decides whether the device copies the image back. ImagePreprocessorBuffer keeps the pixel count separately so size() still answers when the mirror was not allocated. ROIIntegrationGPU asked device 0 for the SM count it sizes its grid from, while workers are pinned round-robin across the GPUs - so on a multi-GPU node it could size a grid from a card it never launches on. It asks the current device now, like every other engine. ~CudaRegisteredVector called a function that throws out of a destructor, and the move-assignment did the same from a noexcept function. Either would abort the process rather than report the failure, and teardown - after a device reset, or while another exception unwinds - is exactly where cudaHostUnregister fails. Both now use an unchecked unregister, as every other destructor in that header already does for its own teardown call. The throwing form stays for rebind()/unregister(), which are called from live code. Measured on a 16M-pixel rotation dataset: unchanged space group, merged reflection count and merging statistics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 13:12:36 +02:00
leonarski_fandClaude Opus 5	b74d8f8545	Give ParallelFor a home and a test ... The two shapes - a fixed contiguous split, and work stealing off an atomic - had been copied into whichever file wanted them: an anonymous namespace in RotationScaleMerge.cpp, and another, byte for byte the same, in ShadowFinder.cpp, whose comment claimed it was "the only other file that wants it". The header is taken from the beam-stop GPU commit on the performance branch, which is not otherwise being picked. ShadowFinder now uses it, so the construct is exercised rather than shipped unused, and its own copy is gone. The beam-stop mask is unchanged, which its reference count already pins. Tested for the properties the callers rely on and which are easy to lose in a rewrite: the chunked slices tile the range in order with no empty one, work stealing visits every item exactly once, one thread means the caller's loop in order, an empty or negative count does nothing, an exception in a worker reaches the caller, and - the point of the whole thing - the answer is the serial answer bit for bit at every thread count. RotationScaleMerge.cpp keeps its own copy for now; consolidating it belongs with the scaling work, which is not being touched here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 13:05:31 +02:00
leonarski_fandClaude Opus 5	4e15fba98a	Pin what the beam-stop parallelisation must not change ... The mask rewrite replaces a BFS dilation with a separable box-max, a full-frame border flood with a bounded one, and three median passes with a single bin-and-rank - four separate equivalence arguments, none of them obvious by inspection. The reference count in the first test was taken from the serial implementation before any of it was picked, and is unchanged by all three commits. Two properties the reference alone cannot cover: the mask must not depend on how many threads split the per-pixel passes, and it must not depend on which shard a frame was accumulated into - including the maximum, which lives in a single shard when the reflection is on one frame. A four-armed scene is invariant under a quarter turn and so must its mask be, which is the sharpest probe available for the x and y passes being written differently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:51:53 +02:00
jungfrauandClaude Opus 5	7795ccb32b	Give the process file its own thread ... Writing an image to the process file takes the global HDF5 mutex, which is the same one every worker needs to find its next image. The write is short - the file holds the per-image analysis, not the pixels - but with a worker per hardware thread they were still taking turns at it. The workers now post to a bounded queue and one thread owns the file. A DataMessage does not own its pixels, it points into the reader's buffer, so the raw image is parked in the queue beside its message; without that the worker frees the pixels on its next iteration and the writer reads whatever landed there. The queue is bounded at four per worker so a run whose analysis outpaces its writer cannot accumulate every image it has ever processed, and a write that throws - out of space, above all - is held and rethrown when the loop drains it, before the end message is written and the file finalized. Worth 6.8 s -> 6.5 s on a 16 Mpx rotation dataset at 48 workers, on top of the much larger gain from taking the read out of the same lock. Both process files, written with and without the writer thread, re-scale to the same 101215 unique reflections at the same ISa. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-23 12:50:12 +02:00
jungfrauandClaude Opus 5	996cd20106	Make the beam-stop mask O(pixels) and parallel ... GetMask() was 3.28 s of the 4.78 s pre-scan on a 16M-pixel detector, all on one thread. Four changes, none of which alters the mask: dilate() was a multi-source BFS. On a full rectangle with no obstacles the 8-connected graph distance IS the Chebyshev distance - a path stepping towards the target never has to leave the frame - so the result is a dilation by the (2r+1) square clipped to the frame, which separates into a pass along x and a pass along y. That is O(1) per pixel whatever r is, with no queue and no 4-bytes-per-pixel distance array (72 MB, allocated and filled five times per call). The erode() case is the one that hurt: it dilates the COMPLEMENT, so on a detector whose shadow is under 1% of the pixels it seeded the BFS from essentially every pixel. fill_holes() floods the background from the border. It now floods the bounding box of the region grown by one: everything outside that box is background and the box's own ring is background, so the whole outside is one border-connected component and a background pixel inside the box is border-connected exactly when it reaches the ring. The three baseline iterations re-binned every pixel by radius and re-took a median each time. The iteration only ever excludes pixels whose background is below a cut, and dividing by a positive baseline is monotone, so a ring's excluded pixels are exactly its lowest ones and the next median is an order statistic of the same, unchanging ring. The rings are binned and sorted once; each iteration then picks a rank and counts a prefix. Nine full-image passes become one. box_sum's vertical pass walked one column at a time, striding a whole row per step and missing on every access; it now carries a strip of columns together. Each row's and each column's running sum keeps its terms in its order, so the floating-point rounding is unchanged - only the traversal differs. The pooled COUNT is a count of at most 25 pixels, so it is an exact integer box sum now rather than a floating-point one; the background itself stays in double, because its running sum adds and subtracts across a whole row and in float the two roundings would not cancel. The per-pixel passes then run on all threads, and GetMask takes a thread count. Measured on a 16M-pixel rotation dataset: GetMask 3.28 s -> 0.99 s, whole pre-scan 4.78 s -> 2.37 s, whole run 1m10s -> 1m03s. The mask is unchanged on both a 16M and a 2M-pixel dataset (139126 and 22143 shadow pixels), as are the space group, the merged reflection count and the merging statistics. Also corrected the comment on erode(): the dilation cannot seed outside the frame, so outside behaves as foreground, not as complement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 12:50:12 +02:00
jungfrauandClaude Opus 5	51c628af3b	Parallelize the beam-stop pre-scan ... The pre-scan read its sample of frames in a plain serial loop: one thread did the HDF5 read, the decompression and the full-detector accumulation for every frame. The cost is fixed per frame rather than per dataset, so it grew straight with detector area - measured at 0.9 s on a 2M-pixel detector and 7.9 s on a 16M-pixel one, where it was 11% of the whole run with 47 of 48 cores idle. Frames are now read on several workers. ShadowFinder keeps one projection per worker so nothing is locked while an image is added, and the projections are summed when the mask is read; the sums and counts are integers, so the result does not depend on how the frames were spread over the workers. A shard that never counted a pixel is skipped when the maxima are merged - it holds 0, which would otherwise beat a genuinely negative maximum. Worker count is capped (PRESCAN_MAX_WORKERS): a shard costs 20 bytes per pixel, and the accumulation is memory-bound, so a handful of workers already saturates it. The beam-centre spot pool is stitched together in sample order after the workers join, so frame numbering and the spot list are what the serial read produced regardless of how the workers interleaved. A frame still joins the pool only if it could be read. ShadowFinder::AddImage took its decompression scratch buffer BY VALUE, so the caller's buffer stayed empty and every frame allocated and zero-filled a fresh full-size uncompressed image (72 MB on a 16M-pixel detector) and freed it again. It takes a reference now, and each worker reuses one buffer. Measured on a 16M-pixel rotation dataset: pre-scan 7.9 s -> 4.8 s, whole run 69.2 s -> 65.1 s. Results are unchanged - same shadow pixel count, same space group, same merged reflection count and merging statistics on both a 16M and a 2M-pixel dataset, and the beam-centre path still commits the same centre. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-23 12:50:07 +02:00
leonarski_fandClaude Opus 5	6190787913	Test that the beam-stop finder finds a beam stop ... ShadowFinder has run by default in rugnux for a release and has never had a test. The scene is a beam stop - an opaque disk on the beam with an arm running off it to the edge - and the mask has to be that and nothing else: not the corners, and not a reflection recorded through the penumbra, which has to be given back. The last assertion pins the number of masked pixels as the serial implementation produces it. The detection is several passes of dilation, hole filling and a per-ring median, and a rewrite that moves the answer by a pixel would otherwise surface as a merging statistic several stages downstream, if at all. The scene is integer and noise-free so every mean is exact, and 257 is odd, square and not a multiple of 64 - the beam lands on a pixel, a cross is exactly 4-fold symmetric, and the column-blocked passes meet a short final block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:49:44 +02:00
leonarski_fandClaude Opus 5	308aa1e0de	Read raw images from several threads at once in a test ... The three GetRawImage cases are single-threaded, so none of them enters the path the change is for: the chunk address is taken under the HDF5 lock and the bytes are read outside it, which only means anything when several workers are inside the reader at once - which is how rugnux drives it. Eight of them pulling every image and comparing against the bytes handed to the writer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:45:01 +02:00
leonarski_fandClaude Opus 5	92f2e0309e	Keep the raw file alive while it is read without the lock ... GetRawImage takes the chunk address under hdf5_mutex, drops the lock, and then reads through a borrowed RawFile*. ReadFile() and Close() both take that same lock and call Clear(), which empties the dataset cache and closes the descriptor - so a read racing a close read through a freed object and a recycled fd. Not reachable today, since the only callers of GetRawImage are the rugnux workers and jfjoch_extract_hkl and neither closes concurrently, but the whole point of the change is that the read happens outside the lock. Share the RawFile rather than borrowing it, so the descriptor outlives a Clear() that races it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:44:04 +02:00
jungfrauandClaude Opus 5	c254081e58	Ask HDF5 where an image is, then read it without the lock ... Two things every worker thread of an offline run did inside the global HDF5 mutex, per image. It opened /entry/data/data and asked it for its dataspace, its datatype and its creation plist, then asked those for the rank, the dimensions, the chunking and the compression. All of that is a property of the file and identical for all of its images, so it is now resolved once when the file is first touched. And it read the pixels - megabytes of them, with the lock held, which is what turned a worker per hardware thread into a queue. HDF5 can say where a chunk lives instead - address and byte count, a lookup in the chunk index with no read attached - so that is all it is asked for now, and the bytes are fetched after the lock is dropped, with a positional read that any number of threads can make through one handle at once. Chunk addresses count from the end of the user block, so its size is added; zero for anything this project writes, not for every file. A file that is not one chunk per image, or a chunk that was never written and exists only as a fill value, still goes the old way - only HDF5 knows what those read as. On a 16 Mpx rotation dataset with the process file being written, the per-image loop at 48 workers goes 12.4 s -> 6.8 s, and stops getting slower as workers are added: 8 workers were faster than 48 before, and are not now. Where no process file is written the same loop only improves ~1%, because this machine has 1.5 TB of RAM and held the whole 7 GB test set in page cache - the read was never the expensive part here. It is where the cache is cold or the filesystem is remote. Battery 9m45s, space group 21/24, no failures, unchanged. The Windows path uses ReadFile with an OVERLAPPED offset for the same reason pread is used elsewhere: it takes the offset as an argument rather than moving a shared file position, so the viewer keeps building under MSVC and gets the same concurrency. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-23 12:42:47 +02:00
leonarski_fandClaude Opus 5	a1b48e9454	Mark unreadable frames in the reprocessing virtual dataset too ... The master's own virtual dataset fills with the error marker, so a source file that cannot be resolved reads as masked rather than as zero counts. The virtual dataset rugnux writes into _process.h5 was left at HDF5's default fill of zero, which is a legitimate count - the same silent failure, one file along. The helper moves above its first user; it has to be set before SetVirtual. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:42:47 +02:00
leonarski_fandClaude Opus 5	b2058d1a79	Feed the ENOSPC test the pixel format it declares ... DetJF4M is signed - a JUNGFRAU in photon-counting conversion is signed by default - and the fixture handed the writer uint16 images, so the pixel-format cross-check added in this branch refused them and the test failed before it reached what it is about. Nothing here reads a pixel value back. Missed when the other fixtures were corrected, because jfjoch_hdf5_enospc_test is a separate binary that jfjoch_test does not run; CI runs it as its own step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:39:00 +02:00
leonarski_fandClaude Opus 5	d0ac559e64	Open a file whose sample axis does not turn ... The reader looks for the goniometer by walking every leaf of /entry/sample/transformations and calling ReadAxis on each, stopping early only at an axis that is scanning. Not every leaf is an axis: the writer's own AXISNAME_end and the two rotation-width scalars carry units and nothing else, and ReadAxis threw when transformation_type was absent. A sweep survived on alphabetical order alone - omega sorts before omega_end, so the walk stopped before reaching it. A stationary axis never stopped, reached omega_end, and the open failed outright with "Cannot open attribute transformation_type". This branch is what made that reachable, by giving a still and a grid scan a spindle that stands still: rugnux read a grid-scan master, got a stationary axis, wrote _process.h5 through the goniometer path - which does emit omega_end - and could no longer open its own output. ReadAxis now treats a missing transformation_type as "not a transformation" and skips it, which is also what makes the search safe against anything a third party leaves in that group. JFJochReader_AxisRecovery covers what the reader has to recover: a sweep, a sweep about an axis that is not called omega, a spindle that does not turn, a grid scan alone and under a turning spindle, and a sweep with the head at a Smargon position. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:39:00 +02:00
leonarski_fandClaude Opus 5	d57f66a0b6	Docs: say what each downstream program actually does with our files ... SOFTWARE_INTEGRATION.md said little more than which plugin to prefer. It now carries the layout matrix, since no program reads all three, and the things that silently give wrong answers rather than errors: - Neggia mis-reads signed 16-bit images. It dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of -2 arrives as 65534 and the -32768 marker as 32768. JUNGFRAU in photon-counting conversion is signed by default, so this is the ordinary PSI case. Also noted in HDF5.md beside the fill-value description, where someone reading about the sentinel will meet it. - No XDS plugin reads saturation_value, so OVERLOAD has to be set by hand in XDS.INP. - XDS will not accept a negative MINIMUM_VALID_PIXEL_VALUE, so signed data cannot declare its negative counts valid at all. - The plugins act on different pixel_mask bits, so XDS and DIALS do not integrate the same pixels. - DIALS reads only the first data file of a multi-file NXmxLegacy set, and says nothing. - pyFAI learns no saturation value, marker or mask from a .poni and will integrate a sentinel as a count; the recipe given was checked against pyFAI's own NaN handling and matches it exactly. SECURITY.md was added to the tree but never to the toctree, so Read The Docs did not publish it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:13:06 +02:00
leonarski_fandClaude Opus 5	fae445c8ae	Changelog: lead rc.162 with what it means for users ... The block had grown into a list of field-level edits, several carrying rationale and measurements that belong in the commits. What a user needs from this release is one thing - files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI - so say that first and keep the rest to one line each. Also adds the security page, which shipped with no entry, and drops a test-only entry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 12:07:13 +02:00
leonarski_fandClaude Opus 5	eb0cb42355	Make the saturation limit convert once, in one place ... The limit is EXCLUSIVE inside Jungfraujoch - the first value that is no longer a count - and NXmx saturation_value is INCLUSIVE, the highest value that still is one. XDS OVERLOAD and the DIALS trusted_range read it inclusively too. The write side subtracted the count and no read side added it back, so the value fell by one on every write-read-write cycle, unbounded: four chained runs over one dataset gave 32766, 32765, 32764, 32763. It also fed the preprocessor, so one more real count was called saturated after each cycle. SaturationValueFromLimit / SaturationLimitFromValue now carry the conversion, used by the writer and by all three readers (HDF5, the lite receiver, the viewer). JFJochReaderImage's summation test moves from > to >= in the same commit: it was silently compensating for the missing count, and correcting one without the other would have shifted it instead. Two more places said the wrong thing about the same pixels: error_value was GetUnderflow(), which is -1 for an unsigned image - a value no unsigned pixel can hold. The marker those images really carry is UINTx_MAX, and GetImageFillValue() already returned it, so the class held two disagreeing definitions of one marker. bit_depth_readout is now written for unsigned images only. DIALS remaps the top two codes of 2^bit_depth_readout to -1 and -2 whenever the field is present, without looking at the pixel type. For an unsigned image those fall below underload_value and are masked, which is what we want. For a signed one they land INSIDE the trusted range, so a saturated pixel reached DIALS as a trusted count of -2 - on the strongest reflections. Verified with DIALS 3.27: an int32 file now masks both sentinels. The field stays where it earns its keep, since dxtbx cannot read unsigned 32-bit without it. Neither the values themselves nor the wire format change. Verified against NXmx, DECTRIS SIMPLON, Durin (Global Phasing fork), XDS and DIALS 3.27; the chained run now holds at 32766. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 11:58:40 +02:00
leonarski_fandClaude Opus 5	583da3c6a0	Derive the rotation width for a chain that was sent ... A chain carried in the END message was written verbatim and stopped there, so AXISNAME_end and the rotation width - which the writer produces when it builds the chain itself - were simply absent. A one-image sweep sent that way imported as a still, since dxtbx prefers AXISNAME_end and only falls back to np.diff; omega_range_average is what DECTRIS-oriented tooling reads for the oscillation. They are derived here rather than added to the wire format: for a constant step they follow from the values, which is every case there is today, so carrying them would cost an array per axis and say nothing new. The step is taken over the endpoints, because the values arrive as floats and a single difference puts that noise straight into the reported width. The test now compares the two routes on the files. It could not have caught this before: it went through the reader, and the reader reads neither of these. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 11:20:34 +02:00
leonarski_fandClaude Opus 5	c1b6030c4f	Set the azimuthal reference in the .poni file ... An image integrated in pyFAI through our .poni came out with every chi 180 degrees from where it belongs. pyFAI's in-plane axes are the negatives of ours, so Rot3 needs a half turn on top of the sign flip. Being a rotation about the beam it leaves 2theta alone - which is why radial integration was right all along and only the azimuth was wrong, and why a powder-ring check could never have caught it. The half turn is needed for the orientation-3 form written before rc.162 as well, so it is not an artefact of declaring the orientation - the file has been 180 degrees out for as long as it has been written. Verified against pyFAI 2026.5.0 on a tilted detector with an off-centre beam, against the lab positions of the NXmx chain: 2theta to 3.6e-15 deg and chi to 2.8e-14 deg. Then end to end, by integrating an image in jfjoch's own layout through a .poni the code actually writes: chi lands within 0.15 deg of physical truth on a 0.5 deg cake bin. Withdraws two changelog claims. The .poni does negate Rot3, and declaring orientation did not fix the azimuth: pyFAI's orientation is numerically inert here, so the file was relabelled and not corrected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 11:00:42 +02:00
leonarski_fandClaude Opus 5	ce11cade84	Tell a Smargon head position from the spindle by equipment_component ... The reader recognised chi and phi by name. phi is an ordinary spindle name in MX, so a file whose rotation axis is called phi had it read back as a head position as well as the spindle - and writing that experiment out again threw, because the sample chain then tried to create phi twice. In the other direction a still with a head position had chi, its alphabetically first stationary axis, adopted as the goniometer. Both are now settled by the file: the axes jfjoch writes for a Smargon carry equipment_component="smargon", the reader takes a head position only from a tagged axis, and skips tagged axes when looking for the spindle. NXmx defines equipment_component as an identifier of the component of the equipment a transformation belongs to, which is what this is; there is no "equipment" attribute in NeXus at all. Adds HDF5Object::AttrExists, since the tag is absent on every file from anywhere else. The two tests assert on the written file - the axis length and the attribute - because the reader cannot see either: it does not look at a shape, and it did not look at the tag. That is the same gap that let the one-image shape through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 10:49:47 +02:00
leonarski_fandClaude Opus 5	8dd3ee6576	Give the Smargon axes one entry per image ... A reader takes the number of images from the innermost axis of the sample chain when no axis varies, and chi/phi are innermost whenever they are present. Written as scalars, a still or a grid scan with a recorded head position imported as ONE image however many were collected - dxtbx falls back to nxsample.depends_on and takes num_images = len(scan_axis). NXmx has no attribute that would say otherwise: there is no "equipment", and equipment_component identifies a rigid assembly ("detector_arm", "detector_module"), which dxtbx reads only for the detector module hierarchy. The axis length is what carries the image count. Both writer paths are fixed, and the goniometer in BuildTransformationChain now takes its container whenever the image count is known, as the writer already did - a stationary spindle sent over CBOR had the same one-image shape. Verified against DIALS 3.27: same file, chi/phi as scalars imports as 1 image, as per-image arrays as 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 10:43:26 +02:00
leonarski_fandClaude Opus 5	9ed299798d	Stop tracking review notes and a generated version file ... docs/review/ holds working material for a single task - review reports, work plans, investigation notes. It goes stale as soon as the code moves, and docs/conf.py has an empty exclude_patterns, so Sphinx would publish all of it. common/GitInfo.cpp is configured from GitInfo.cpp.in into the binary dir; the tracked copy was the residue of an in-source configure and still named rc.148. Neither was ever meant to be committed - both arrived through a git add -A. Both are now gitignored, and CLAUDE.md says to stage by explicit path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VfYvJT5Nb71suJCowRBn5z	2026-08-23 10:39:23 +02:00
leonarski_fandClaude Opus 5	e4edcd6fa9	Carry the sample transformation chain as DetectorTransformation ... Replaces the start-message TransformationAxis of the previous commit, which was the wrong shape in two ways. DetectorTransformation (common/) mirrors a NeXus NXtransformations axis and holds nothing else: name, type, units, vector, offset, depends_on and the positions themselves. Deliberately without cleverness - the values are either a single number for an axis that does not move or one per image, and nothing derives a position from a start and an increment. That is the point: a producer will later want to report where a stage actually WENT rather than where it was told to go, and a structure that stores start+increment cannot express that. A million images cost 4 MB per axis, which is not a reason to be clever. Hence also the move to the END message: measured positions are only known once the run is over. And hence no metadata version bump, which the previous commit did make. The chain is optional; when it is absent the writer builds the identical chain from the start message, exactly as before. Nothing on the wire changes for a producer that does not send it, so a broker and a writer of different releases still interwork - the constraint the previous version stated is withdrawn. The writer transcribes a chain it is given, without recomputing an angle, which is what makes measured positions possible end to end. JFJochReader_TransformationChain_SentAndBuilt writes the same run both ways and checks the two files read back the same, chi/phi included. CBORSerialize_End_Transformations covers the wire 1:1, asserting the order survives and that a moving axis keeps one value per image while a stationary one keeps a single value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 23:36:50 +02:00
leonarski_fandClaude Opus 5	7efcf631de	Write module_offset as a float, and declare offset_units ... Two latent traps in the NXtransformations attributes, both inert today and both wrong the moment they are not. module_offset was an int32 with @vector = (0,0,0). NXmx types the field NX_FLOAT, and a translation needs a unit vector to be well formed - the direction of a zero-magnitude translation is arbitrary, not absent. Now a float with (0,0,1); the transformation it describes is unchanged, since the magnitude is still zero. Every transformation that carries an @offset wrote it without @offset_units, and every caller passed an empty string. A reader then falls back to the axis's own `units` - which on a rotation axis is degrees - and converts a length from degrees to millimetres. nxmx only performs that conversion when the offset is non-zero, so ours have never triggered it, but the first non-zero offset on a goniometer axis would. The helper now always declares it, rather than leaving it to a caller to remember. Measured after the change: module_offset is H5T_IEEE_F32LE, a rotation axis carries offset_units "m" beside units "deg", dials.import still reads the file, and the tilted-geometry cross-check is unchanged at 1.6e-6 mm. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 23:14:40 +02:00
leonarski_fandClaude Opus 5	842c43a86e	Send the sample transformation chain in mounting order ... The chain could not be expressed. A goniometer axis, the Smargon chi/phi and a grid stage each travelled by a different route - the `goniometer` map, a private JSON key inside user_data, and `grid_scan` - and nothing said what order they are mounted in. The order cannot go in the `goniometer` map either: that is a DECTRIS stream2 field, and RFC 8949 requires deterministic encoders to SORT map keys, so a map's order is not something a consumer may rely on. So `transformations` is sent as an ordered ARRAY, base first, each element carrying name, type, axis vector and angles. The `goniometer` map and `grid_scan` are still emitted beside it, unchanged, for consumers that only know stream2 - nothing vendor-defined is mutated, and a stream2 consumer sees exactly what it saw before. Smargon chi and phi are ordinary stationary axes that were only ever separate for historical reasons, and they now appear in the chain like any other. They are also read back: reader/ had no smargon support at all, so re-opening a file lost the head position silently. Combined with the earlier change that writes them for a still rather than only alongside a rotation or a grid scan, the round trip is now closed. Metadata version 7. A broker and a writer from different releases must not be mixed across this: an older writer ignores the chain and reads the unordered map, so anything whose order matters - a Smargon position, or a grid scan combined with a rotation - is not reproduced. Said so in docs/CBOR.md and the changelog. CBORSerialize_Start_Transformations asserts the ORDER survives, not just the contents, which is the whole point of the array. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 23:07:32 +02:00
leonarski_fandClaude Opus 5	d2f57975e8	Record whether the image is mirrored in Y, and label the .poni orientation ... Which way the detector's rows run was decided once, in the module assembly, and never stated again: not on the wire, not in the file, nowhere a consumer could read it. mirror_y was consumed inside the DetectorGeometryModular constructor and discarded. It is now a declared property of the detector setup, carried into the start message, written to HDF5 under detectorSpecific, and read back. Absence means true, which is the MX convention and the only thing Jungfraujoch has ever produced. Deliberately a boolean and not a corner enum: the assembled image can only be flipped in Y, so a four-corner value would encode states that cannot occur. DECTRIS stream2 has no field for this - checked against the specification - so the key is new rather than an extension of theirs, and a consumer that does not know it skips it and behaves exactly as before. The .poni file gains pyFAI's orientation. Without it pyFAI applies its own default, 3 (bottom left), and believes increasing row means physically upwards. The numbers still agreed - a mirror preserves 2theta, so radial integration was never affected - but the azimuth came out with the opposite sense, which matters for cake and sector integration. Declaring orientation 2 is not a one-line addition: it re-anchors Poni1 to the top edge and reverses rot2 and rot3, a row flip being improper. Measured against pyFAI 2026.5.0 by searching all four orientations, both Poni1 anchorings and all eight sign combinations: exactly two combinations reproduce the lab position DiffractionGeometry computes to 1.4e-17 m - the unlabelled form written before, and (orientation 2, Poni1 = height-1-beam_y, +rot1/+rot2/-rot3), which is now written. Calibration_PoniFileAxisConvention pins it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:54:50 +02:00
leonarski_fandClaude Opus 5	5e05515165	Write the grid stage as a base stage, not head-mounted ... The grid translations went innermost, i.e. mounted on the head, so a grid position turned with the spindle. At SLS the grid is an Aerotech xyz that the spindle is mounted ON, so the mounting order is base -> grid -> omega -> chi -> phi -> sample and a grid position is independent of omega. Identical to the previous chain at omega = 0, which is every grid scan collected so far, and correct rather than incorrect when it is not. A head-mounted stage exists too - the Smargon translates, and that is what helical uses - and would sit on the other side of omega. Only the base stage is modelled for now, which is the one actually used; the comment says so. Measured: dials.import reads a master with the new chain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:41:51 +02:00
leonarski_fandClaude Opus 5	cc334c5c55	Name the right rotation in the NXmx frame comment ... The comment said internal-to-McStas is 180 degrees about x. It is 180 degrees about z: internal (a,b,c) maps to McStas (-a,-b,+c), which is what makes the written vectors correct - internal +y becomes (0,-1,0), the -x of Rx(-rot2) becomes (1,0,0), and the -z of Rz(-rot3) is unchanged. 180 degrees about x is the internal-to-imgCIF relation, one step further on, and it is the frame the verification was done in - which is why the vectors are right and only the prose was wrong. Re-measured after the change: still 1.6e-6 mm over nine tilt settings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:33:30 +02:00
leonarski_fandClaude Opus 5	7039aa4e45	Stop treating a goniometer axis and a grid scan as alternatives ... They are not alternatives: a grid is usually collected at a particular head position, so an axis and a grid describe different parts of the same setup. The exclusion was enforced independently in four places - the API converter, the CBOR serializer, the writer and the reader - and each silently dropped the grid scan when an axis was present. Nothing warned. The writer now builds one chain from the base outwards, spindle -> chi -> phi -> helical -> grid translations, instead of two branches. NXmx applies the deepest dependency first, so the sample ends up innermost, which is what it physically is: the grid stage rides on the head and the head rides on the spindle. The grid translations consequently move inside the rotation - identical to before at omega = 0, and right rather than wrong when it is not. A grid scan with no axis at all now writes a stationary omega. NXmx has no way to say "there is no rotation", and a sample chain of translations alone is not something readers accept: dxtbx raises outright on it, so every grid-scan master we have written so far cannot be opened by DIALS. Measured on a file matching the new chain: dials.import reads it. At 0 degrees the rotation is the identity whatever the axis points along, so the conventional vector carries no geometric claim - it only has to be well formed. The API change is deliberately not breaking: no field changes type or cardinality, only the prose saying the two were exclusive, and a request that set both used to lose one silently and now does not. JFJochReader_GridScan asserted the absence of a goniometer; it now asserts the axis is present and stationary, which is the contract that matters - a grid scan must not read back as a sweep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:25:08 +02:00
leonarski_fandClaude Opus 5	b35672a0c3	Read any rotation axis by name, and tell a stationary axis from a sweep ... Two things the goniometer handling conflated. The axis name is free-form everywhere that writes it - the API imposes only minLength, the CBOR map uses the name as its key, and tests/CBORTest.cpp round trips one literally called "z" - but the reader looked for exactly "/entry/sample/transformations/omega". A sweep recorded as "phi" therefore came back as stills, in the viewer and in rugnux, with nothing to indicate it. The reader now walks the transformations group and takes whichever axis is a rotation, preferring one that turns; the grid scan is read independently rather than as the else-branch of the same test, since a grid scan can be taken at a given head position. Second: "an axis is defined" and "the axis is turning" were the same question, answered inconsistently - GetImagesPerFile checked the increment, IsRotationIndexing did not, and the CBOR decoder deleted zero-increment axes outright so the ambiguity could never surface. GoniometerAxis::IsScanning now asks it explicitly and the call sites go through it, so a stationary axis can be carried without being mistaken for rotation data. That mistake is not hypothetical: RotationIndexerCounter leaves its stride at zero for a zero increment, and Process() then never fires, so indexing would silently never run. Keeping stationary axes is also what lets the writer state where the head was for a still or a grid scan, which is the next step. JFJochReader_Goniometer_NonOmegaName covers the naming case through the writer and back; nothing did before, because both existing round trips use "omega". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 22:20:41 +02:00
leonarski_fandClaude Opus 5	1f4f77fe42	Choose images_per_file from the acquisition when it is not given ... The value means three different things at once. It is the unit of writer parallelism - whole files go round-robin to the writers, (image_number / images_per_file) % socket.size() in ZMQStream2Pusher::SendImage and TCPStreamPusher - it multiplies writer memory linearly, since every data-file plugin reserves per file, and it decides whether a legacy master is readable at all, because dxtbx follows only the first data file of one. That last point is what makes a flat default wrong. Measured with DIALS on a 2500-image rotation sweep written as legacy: split into five files it reports 2500 images and then raises IndexError beyond image 499, so it half-works silently; in one file all 2500 read. AutoPROC does not read VDS, so legacy has to stay the default, which leaves the file count as the only lever. So make it optional and resolve it from the acquisition. A rotation sweep of at most 20000 images goes into one data file - rotation datasets are small enough, and one writer keeps up with them. A grid scan splits on whole fast-axis rows, so a file is a meaningful piece of the grid. Stills and serial keep 1000, where the image count far exceeds it and the parallelism and the bounded writer memory are what matter. An explicit value is always taken literally. GetImagesPerFile is the single place this is resolved, and it must always return a fixed non-zero number, because everything downstream - receiver, pusher, puller, writer - requires one. That was already true of the old 0 = "one file" spelling; 0 is now gone from the API and omitting the field says the same thing better. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 21:57:12 +02:00
leonarski_fandClaude Opus 5	259b43154e	Writer: refuse a mistyped stream, and mark unreadable VDS frames ... Two ways the written files could misdescribe themselves without anyone noticing. The pixel format is stated twice and the two were never compared: each image carries its own type as a CBOR tag, which is what the data files are written with, while the master is typed from the start message. A stream whose header contradicts its images produced data files of one type under a master declaring another, and with NXmxVDS, HDF5 then converts silently on every read. HDF5DataFile::CreateFile now checks the two agree and refuses the run otherwise - the point where the values first meet, so it covers every path into the writer. Two test fixtures were relying on exactly that inconsistency. The HDF5 writer tests wrote uint16 buffers under a JUNGFRAU experiment, which converts to photon counts by default and so declares int16; they never read the pixels back, so it went unnoticed. The receiver-lite tests feed frames from compression_benchmark.h5, which really are signed int16, through a DECTRIS experiment, which declares unsigned by default - the same class of bug the pixel_signed propagation fixed on the live path. Both now declare what they send. Second: a virtual dataset whose source file is absent reads as the fill value, and HDF5 defaults that to zero, so a data file that was not copied alongside the master is indistinguishable from frames of genuine zero counts. Measured with DIALS on a four-file set with one file removed: 25 frames of pure zeros, no error and no warning. The image VDS is now filled with the error marker instead, which sits outside underload_value..saturation_value, so a reader masks those frames. Same measurement after the change: -32768 throughout, which DIALS excludes. Only the images ask for a fill value; the per-image metadata datasets keep the default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 21:44:04 +02:00
leonarski_fandClaude Opus 5	28cb185325	rugnux: log the detector geometry in XDS's convention ... XDS is never handed this geometry - the durin plugin gives it image data only (plugin_get_header returns dimensions, bytes per pixel, pixel size and frame count, nothing more) and XDS refines its own from XDS.INP. That is exactly what makes printing ours in the same convention useful: it turns "does our geometry agree with XDS's refinement" into reading two logs side by side. The two laboratory frames already coincide - x along increasing detector column, y along increasing row, z along the beam - so nothing is converted. XDS places a pixel at x_lab(i,j) = (i-ORGX)*QX*X_axis + (j-ORGY)*QY*Y_axis + DISTANCE*(X_axis x Y_axis) which is DiffractionGeometry::LabCoord with X_axis = poni_rot*(1,0,0) and Y_axis = poni_rot*(0,1,0). The axes are taken as differences of LabCoord so they track whatever the geometry currently is, tilt included, and a tilt goes out as the two axis vectors rather than as angles - the form XDS itself reports after refinement. Two traps are handled and documented: ORGX/ORGY are 1-based, XDS counting pixels from 1 where we count from 0; and they are the PONI, the foot of the perpendicular from the crystal, which is what our beam centre is too but is not the direct beam once the detector is tilted. ROTATION_AXIS is printed as stored. The direction is right, the frames being shared, but its sign has not been cross-checked against an XDS refinement, so a flip there should be read as unconfirmed rather than as a real disagreement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 20:54:26 +02:00
leonarski_fandClaude Opus 5	659ba0d5b9	Cross-check a tilted detector against pyFAI and DIALS ... Nothing constrained the tilted geometry. Every existing test is either self-consistent or moves one angle at a time, and every file CI writes has zero tilt - where a swapped axis, a reversed composition order and a wrong pivot all give exactly the same answer. Two real bugs lived in that gap. Two checks, because there are two things to guard. DiffractionGeometry_Tilted_vs_PyFAI_and_DIALS pins the model itself: all three PONI angles non-zero and of mixed sign, compared per pixel against reference positions from pyFAI (an independent implementation of the convention) and from DIALS, to 2 um. Because the references are quoted in their own frames and the test applies the documented mappings - pyFAI (t1,t2,t3) -> our (x,y,z), and imgCIF = ours turned 180 degrees about x - it pins those relations too, not just the arithmetic. The comment gives the snippets to regenerate both sets. tests/nxmx_geometry_dials_test.py guards the writer, which is where the bugs actually were and which the unit test cannot reach. CI writes a master, patches the geometry in and asks DIALS where the panel is. Only the angle VALUES are patched; the axis vectors, the depends_on chain and the pivot stay as the writer emitted them, so they remain under test. Verified to fail on the pre-fix encoding: 7.3 mm, exit 1, naming the chain as the thing to look at. Recorded in both, because it cost an hour: compare via get_origin() and the fast/slow axes, NOT get_pixel_lab_coord(), which applies a parallax correction from the sensor thickness that Jungfraujoch does not model - about 0.1 mm at the detector edge, easily mistaken for a geometry error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 20:50:38 +02:00
leonarski_fandClaude Opus 5	1eefd035c2	rugnux: stop negating Rot3 in the .poni file ... The PONI export mapped the internal angles to pyFAI as (+rot1, -rot2, -rot3). Checked against pyFAI 2026.5.0 directly - building a Geometry from the exported values and comparing calc_pos_zyx against the lab position DiffractionGeometry computes, per pixel over the whole detector and with each angle exercised on its own - the correct mapping is (+rot1, -rot2, +rot3): it agrees to 1.4e-17 m, while negating rot3 puts a pixel 25 mm out on a rot3-only geometry. rot1 and rot2 were already right, which is consistent with how this was originally validated: a LaB6 powder image, where rings sharpened once the rot2 flip was applied. That check could not have caught rot3, because a rotation about the beam leaves q and 2theta invariant and moves only the azimuth - so the error only ever showed up in cake/sector integration, and only for a detector actually rotated about the beam. rot3 is never refined and has no CLI flag, so in practice it is almost always zero. The old comment derived the signs from "a reflection in y between the MX and pyFAI frames". That gives the right answer for rot1 and rot2 and the wrong one for rot3, and pyFAI's own documentation contradicts itself on the direction of its axis 2, so the comment now records the empirical pin instead of a derivation. Calibration_PoniFileAxisConvention previously set rot3 to zero and asserted Rot3 == 0.0 - the one cell that could not fail. It now uses a non-zero rot3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-22 20:14:23 +02:00