1136 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
jungfrauandClaude Opus 5	f36cd88795	Give a 16-bit image a saturation code when its limit is 65534 ... The narrow encoding picked NARROW_BAD as the saturation code whenever the saturation limit was above UINT16_MAX - 2. That is one too strict. A real value is STRICTLY below the limit, so a limit of UINT16_MAX - 1 still leaves UINT16_MAX - 1 free to be the code; only a limit of the whole range has nothing to spare, and that is exactly the case where no pixel can be saturated, because the "is error" test claims UINT16_MAX first. At a limit of exactly 65534 the old condition therefore stored a saturated pixel as the masked code, and it widened back to INT32_MIN instead of INT32_MAX. Masked and saturated are not interchangeable: the strong-pixel search flags a saturated pixel unconditionally and a masked one never, so the overloaded core of the strongest spots dropped out of the strong-pixel mask. Bragg integration treats the two alike and the image statistics are taken from the raw value, so nothing downstream of those moved - which is why a battery over 24 crystals showed nothing. That value is not a corner case. GetByteDepthImage()-driven writing stores saturation_value = GetSaturationLimit() - 1 and the readers take it back as-is, so a 16-bit acquisition whose detector cutoff is at or above the full range comes back with a limit of exactly 65534. The one 16-bit dataset in the rotation test set declares 11963, which is safe, so it could not have caught this. Found by review, not by testing, because nothing tested the narrow path at all: every GPU test writes into the wide buffer directly and never asks for the narrow one, and every preprocessor test copies the image back to the host, which forces the wide path. So the test comes with the fix. It runs the wide and the narrow preprocessor over the same synthetic frame - values around each boundary, masked on a stride coprime with the value cycle so every value appears both masked and unmasked - across saturation limits of none, 5000, 0xFFFD, 0xFFFE and 0xFFFF, on both the host-upload and the device-decode entry point, and compares the statistics and every pixel. Against the old condition it fails with 625860 differing pixels; against this one it passes. Also makes the header self-contained: it uses __host__/__device__ and the CUDA vector types and only compiled because every includer happened to pull in cuda_runtime.h first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-18 18:41:53 -04:00
jungfrauandClaude Opus 5	a97d763098	Do not assume a DECTRIS detector's bit depths before asking it ... DetectorSetup gave a DECTRIS detector bit_depth_image = 16 and bit_depth_readout = 16 the moment it was constructed. Both mean "the detector told us", and at construction nothing has asked it: the SIMPLON client learns them at configure time, and the lite receiver reads them off the start message. Until then they were an assumption wearing the clothes of a measurement. The assumption was load-bearing in the wrong direction. GetByteDepthImage() gives the detector's image depth absolute priority - correctly, since the DECTRIS path forwards images verbatim and the depth has to be the one the pixels actually have - so a value that was always set meant everything below it was unreachable on DECTRIS: * image_format_settings.bit_depth_image, which the API documents as "bit depth of resulting image ... if not provided value is adjusted automatically", was silently discarded. It works on JUNGFRAU, where the detector's value is not set, and did nothing at all on an EIGER. No error, no warning. * so was the promotion to four bytes that summation needs. Both are now unset for DECTRIS. JUNGFRAU keeps its readout depth of 16, because there that is a property of the hardware rather than something configured. Where nothing supplies a depth the answer does not change: GetByteDepthImage() falls through to the same two bytes it produced before, and the fallback no longer throws merely because the readout depth is not known yet - GetBitDepthReadoutIfKnown() reports it as unknown instead of inventing one. The start message carries it as an optional already, and the receiver already guards on that, so "not known" travels end to end rather than being papered over. GetBitDepthReadout() still throws for the callers that genuinely require a value; all of them are on the FPGA path, where it is always set. Also fixes the jfjoch_test build, which the previous commit broke: SpotExtractorGPU's Extract() takes a PixelView now, and the parity test still passed a raw pointer. That should have been caught before it was pushed. Tests: DetectorSetup (with a new case pinning the contract this restores), plus DiffractionExperiment, CBOR, writer, HDF5, preprocessing, azimuthal integration and the GPU spot-finding / integration / decoder suites. rugnux unchanged end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-18 17:48:19 -04:00
jungfrauandClaude Opus 5	9f49e5abb7	Keep 16-bit images 16-bit through the GPU pipeline ... A detector reading out 16 bits had its frame widened to int32 the moment it was decoded, and every per-pixel pass over that frame then moved four bytes a pixel to carry two. Those passes - the ring statistics three times over, the strong-pixel search, spot extraction, the azimuthal and ROI integrators, Bragg integration - are the bulk of the image loop's device traffic, and 16 bits is the mode a fast acquisition runs in, which is exactly where throughput matters. The preprocessed image now keeps the width of its source. Two codes at the top of the 16-bit range carry the two special states, and they cannot collide with a real value: 0xFFFF masked, or the source's own bad-pixel marker. saturation a pixel at or above the saturation limit. 0xFFFE where the limit code leaves room - a 16-bit EIGER declares a count-rate limit of a few thousand, so there is room to spare - and 0xFFFF where the limit is the whole range, in which case the "is error" test has already claimed 0xFFFF, nothing can be saturated, and 0xFFFE stays a real value. Either way a real value is strictly below the saturation limit and so below both codes. Nothing is clipped and nothing is lost, and which code is in force is carried with the image rather than assumed. No engine learns a second convention. PixelView widens on load, so a masked pixel still reads as INT32_MIN and a saturated one as INT32_MAX, and every existing `v != INT32_MIN && v != INT32_MAX` test keeps its meaning. One code path, not two instantiations that can drift apart; the branch is on a pointer that is the same for every thread of every block, on kernels whose time is the loads it selects between. The vector loads are kept - four pixels still arrive in one transaction, 16 bytes wide or 8, whichever the image is. The wide path is unchanged, and is still taken for anything that is not a 16-bit source, and for any caller that wants the preprocessed image copied back to the host - that mirror is int32 and the CPU engines know only that convention. Measured on the one 16-bit dataset in the rotation test set, which is also the smallest detector in it (2.5M pixels, where per-pixel work is a small part of the loop): image loop 1.025 s -> 1.005 s at one GPU, whole run 5.64 s -> 5.52 s. The gain scales with the frame, so a 16M-pixel detector - where six full-frame passes are 86 % of the loop's GPU time - has much more to gain, and nothing here can measure that: every other dataset in the test set is stored 32-bit. Correctness on that dataset is exact where it can be: indexing rate, first-pass validation score and the integrated partial count are identical to the wide path, and its whole battery row - reflections, observations, space group, R_meas, CC1/2, ISa, mosaicity - is unchanged. Battery 6m17s -> 6m16s, 21/24 space groups, no failures. Also logs, once per run, the width the images are stored in, since it decides how much of the frame moves through every pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-18 01:52:27 -04:00
jungfrauandClaude Opus 5	642cbc7271	Stop the rotation pre-pass once it has what the second pass needs, and say where time went ... Three changes to the rotation two-pass and what it reports. THE PRE-PASS STOPS EARLY. The first pass exists to measure five things for the second: the post-refined detector distance and beam, the goniometer rotation scale, the frame-order-smoothed mosaicity, the space group, and its own indexing result for the supercell guard. All five are settled by the time the group is adopted. It then carried on and did a full production run anyway - a fourth scale/combine/merge in the adopted group, the correction surfaces, the error model, the resolution cutoff, the twinning analysis, the Wilson B, the statistics table - and wrote MTZ, CIF, HKL and a report to <prefix>_01_*. The second pass remakes every bit of that at the refined geometry a few seconds later, which is the result anybody reads. Measured over the rotation test set, running it anyway was 10 % of the battery, and up to 21 % of a single merge-dominated crystal. It now stops at the space-group decision and writes nothing; the _01_* files go with it, since they existed to compare the two passes and nothing in the pipeline or the test harness reads them. One value had to move for that. Pass 2 asks whether pass 1 reached its space group by PROMOTING the point group, and read that off result.twinning - a struct only filled once the final merge has run. It is now recorded where it is decided. EVERY RUN NOW REPORTS WHERE ITS TIME WENT. The phase marks that drove the GUI progress label are now also timed, and the run ends with a table of wall time per phase summed over both passes, with the mean number of cores each phase kept busy beside it (CPU time over wall time). That second column is the one that matters: a phase with a large share and one core is a single thread doing all the work, which is a different problem from a phase that is simply large. It costs two clock reads per phase, and it means a slow dataset on a machine with no profiler on it can still be diagnosed from its log. The cores column needs process CPU time and is zero where the platform has no getrusage; the wall column works everywhere. AND THE PER-IMAGE COST LINE WAS WRONG. It divided the per-stage means, which are per worker, by the thread count - but the image loop is capped at four workers per GPU, so with 2 GPUs at -N 48 those are 8 and 48 and every stage was reported 6x too small. It divides by the loop's own worker count now, and names it, so the line says what it means. This was the one output anybody tuning the program would read first. Battery 6m52s -> 6m17s, space group 21/24 with the same three disagreements, no failures, and no merged result outside the spread the binary already has against itself between two runs. (One weak, 82%-complete crystal moves several points of R_meas between any two runs of any binary, this one included - measured at 39.7 / 31.7 / 35.5 / 35.8 / 40.1 across five battery runs of four different builds.) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-18 01:12:21 -04:00
jungfrauandClaude Opus 5	debd73c0bd	Find the first pass's spots on every worker ... The first pass of rotation indexing feeds two sampling schemes and a validation set, and it found their spots one frame at a time on one CUDA stream. It reads 271 to 402 frames depending on the crystal, once per pass, and there are two passes. On a 16M-pixel dataset that is 6.7 s of a 30 s run at a mean occupancy of 1.9 threads out of 48. The comment said this was to keep the spot cache from depending on scheduling. The guarantee is stronger than that and survives: a frame's spots are a pure function of that frame, so whatever order the workers finish in, the cache ends up holding exactly what the serial loop put there. The feed order and the point at which the consecutive scheme stops accumulating are untouched, so the schemes see the same frames as before. The phase was never CPU-bound, which is why this works: its one thread was spin-waiting in the driver on the same six full-image passes, about 5.5 ms a frame. One engine per worker for the whole pass, not per batch. An engine on a 16M-pixel detector costs ~0.13 s to build - a first attempt rebuilt them per batch and came out SLOWER than the serial loop, four batches of eight engines against ~1.5 s of frame work. Two smaller things, in the same per-image path: The strong-pixel flagging kernel reads four pixels at a time and issues one atomicOr per four instead of per pixel. Same values, confirmed end to end. The beam-stop pre-scan sized its per-worker accumulators for the worker count rather than for the workers that use them. Two int64 arrays per pixel per shard, eight shards: 2.3 GB allocated and cleared on a 16M-pixel detector for a path the GPU never reads. They are allocated on use now, which is most of what made that phase look like eight workers with two of them running. Measured, three A/B pairs with the order alternated, whole run: 16M-pixel 3600-frame 42.8 s -> 38.8 s, 16M-pixel 1800-frame 38.7 s -> 36.1 s, both at two GPUs; a 4M-pixel control 23.3 s -> 23.1 s. First-pass indexing itself -44 % and -36 %. The two smaller datasets merge byte-identically over six runs each; the largest is not bit-reproducible in the unmodified binary either, and stayed inside that spread. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 18:06:03 -04:00
jungfrauandClaude Opus 5	a22a51372e	Run the parallel helpers on a pool instead of starting threads per chunk ... ParallelFor and ParallelChunks started fresh OS threads on every call, one per chunk, through std::async. There are 57 call sites and several sit inside iterative fits, so one run of the heaviest crystal created 13 497 threads and a 900-frame dataset 7 320. Both now run on a persistent pool. The contracts are unchanged: ParallelChunks keeps the same worker count and the same fixed split, so a reduction sums term for term as before, and ParallelFor keeps stealing per item. Two things in the pool are worth knowing. The caller is one of the hands - it claims its own region's tasks and then waits only on tasks already running - so a region entered from inside another region cannot deadlock at any depth, which a shared-queue pool would. And a task wakes one worker rather than the whole pool: on a large machine notify_all wakes every idle thread to find nothing, once per region, tens of thousands of times a run. Two hand-rolled copies of the same pattern now use it, in FrenchWilson and in the two histogram passes of ComputeAsuGroups. Be clear about what this buys today: nothing measurable. Thread creations drop 13 497 -> 770 and entering a parallel region goes from 1.2-2.2 ms to 112 us, an 11-20x cut, but wall clock on 48 threads is level with before, inside the +-5 % this machine's run-to-run placement is worth. What it removes is a cost that grows with the thread count - measured, entry is linear in it - and the machine this is heading for has four times the threads of the one it was measured on, where the same 335 regions a run would cost about 1.8 s of pure thread creation. ComputeAsuGroups' histogram also changes. It is an nthreads x n_groups table, 936 MB at -N 48 on the heaviest crystal and allocated five times a run, and the prefix over it walked DOWN a column - a 19.5 MB stride, so a cache and TLB miss per step, 234 M of them, serially. Both passes now walk rows and split over group ranges. The counts are integers, so the result is bit-identical. This one is reasoning, not measurement: at 48 threads it sits under this machine's noise and could not be shown either way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 18:05:35 -04:00
jungfrauandClaude Opus 5	65eb86ab66	Size the post-refine gather by its reflections, not by its frames ... Both passes of the gather are chunked over OUTCOMES, one per frame, but their cost is the tens of thousands of reflections inside each. ThreadsForWork was being asked about the frames: a rotation dataset has on the order of a thousand of them against a floor of 32768 items per thread, so it answered 1, and the pass that walks every reflection in the run ran on a single thread on every dataset in the test set. RotationScaleMerge::Ingest does the identical thing and asks with the observation count, which is what makes this an inconsistency rather than a tuning choice. The reflection total is already to hand a few lines up. The split is unchanged - only the worker count moves - so the output is identical. Measured on the heaviest crystal: the phase drops 33 %, from 8.6 s to 5.5 s, and its mean occupancy goes from 4.1 threads to 12.3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 18:05:19 -04:00
jungfrauandClaude Opus 5	2308bbad8c	Stop scaling and merging what the resolution range excludes ... A crystal integrated to the detector corner but merged well short of it carries observations through the whole merge that the merge then discards. On the heaviest dataset in the rotation test set that is 63.3 M partials of which 6.4 M are ever used: the other nine tenths are sorted, uploaded, scaled, combined, error-modelled and post-refined before anything looks at their resolution. Ingest copied every one of them unconditionally, and the d_min limit was first applied far downstream, in the ASU grouping. They are now dropped at ingest, immediately after the one big sort: - WHOLE raw-hkl runs are dropped, on the same rawrun_d the ASU grouping already tests. A per-observation test is not equivalent - a run is in or out today by one member's d - and using a different rule here would put the two out of step. - The drop happens AFTER the flux meter, which takes each frame's mean background over every reflection on it, and after the sort, so neither changes. - The compaction runs in index order, so a frame's observations stay contiguous and keep their order, and every per-frame sum keeps its sequence of roundings. Post-refinement reads the integration outcomes rather than the merge arrays, so it still sees every reflection. That is the point: --integration-high-resolution buys the same time by never integrating the reflections, and pays for it in the per-frame geometry, which wants them. Two more passes over the observation array go with it. The incident-flux divide was 15.8 % of all user cycles to read one int and divide one float across 5 GB; the per-frame mean it needs is now accumulated by the ingest fill loop - one frame, one thread, same order, so bit-exact - and the divide rides on the finiteness pass that already touches that field. And the geometry post-refinement is fitted on a bounded sample of partials, selected by a hash of the raw hkl so whole rocking events are kept or dropped together and the sweep and the detector are thinned uniformly. The sample size is 8 M and the reason it is not smaller is measured. Over a 126x thinning the fitted rotation scale is flat to 2e-5 and the beam centre moves 0.03 px, but the CELL scale breaks between 8 M and 4 M: the axis step keeps the 20 000 strongest events, so once the pool approaches that size it starts fitting weaker ones and the second pass's cell shifts by ~0.1 %. Measured on the heaviest crystal, three A/B pairs with the order alternated: 68.4 s -> 37.3 s wall, 530 s -> 221 s of CPU. Whole battery 8m07s -> 7m15s, space group 21/24 with the same three disagreements as before, no failures. Bit-identical is not available on the GPU path - the resident reductions and the fulls emit order depend on array length - so what is shown is that every difference sits inside the spread the unmodified binary has against itself between two runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 18:04:55 -04:00
jungfrauandClaude Opus 5	20f1827550	Fit the modulation surface on a grid that spans the detector ... The detector-frame modulation correction takes its 16x16 grid extent from a pass over every full, but the surface is fitted only on the fulls that belong to an ASU group. Those are two different populations, and the gap between them is whatever was integrated past the resolution the merge uses. That made the correction's fate depend on how far integration reached. Cut it back and the grid contracts onto the merged disc while the cell count stays the same, so each cell holds too few reflections, the surface over-fits, and cross-validation throws it away - correctly, on a surface that should never have been fitted at that scale. Varying only the integration limit on one rotation dataset, merged R_meas came out 28.4 / 33.1 / 29.0 / 32.8 / 31.7 %, and the four-point spread is entirely the correction switching on and off: every low value is a run where it was applied, every high value one where it was refused, with no exceptions. Nothing else moved. The grid now spans the detector. Cells with no observations in them keep a factor of 1 and cost nothing, and with integration running to the detector corner - the default - the grid is the one it always was, so the common case is unchanged. On a crystal carrying no resolution limit at all it takes merged R_meas from 39.7 % to 35.5 %. This is a correctness fix in its own right. It also has to come first: without it, any change that narrows the integrated resolution range trips the same over-fit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 18:04:23 -04:00
jungfrauandClaude Opus 5	088ba1cff8	Stop paying for workers and threads that do no work ... Four changes, all from measuring why this 48-core machine was SLOWER than a quarter of itself on a single job. The image loop now takes a worker count of its own. It is GPU-bound, every worker builds a private analysis engine of tens of megabytes of device and pinned memory - and every one of those allocations implicitly synchronises the device - so past a handful per card another worker adds setup and contention and no throughput. Measured at about 23 ms of pure setup per extra worker, which is why the penalty is WORSE on short runs: 200 images cost 0.86 s of loop at 12 workers and 2.25 s at 48. Capped at four per GPU, floor of eight. Every other phase still gets the full thread count, because each one starts its own workers. Ingest built its array with a serial push_back over every observation of every frame - 63 million of them on the largest crystal here. Each frame's block offset is known before anything is written, so the frames convert together, each still written by one thread in its own order. The pass that buckets observations by h was the single most expensive thing in a large run - 24% of all cycles, in five instructions. It strided an array of 80-byte observations to read one 4-byte field, and its store address depended on the loaded value, so the store buffer could not retire and the misses stopped overlapping. The sweep that already reads every observation now copies h out as it goes, and the bucketing walks that instead. And the post-refine passes took the raw thread count. One of them runs 134 times inside the rotation-scale fit, starting 48 threads each time to divide 390k terms among them; it is gated on the work now, like everything else. Measured on one crystal, N=48: 12.80 s -> 11.45 s, which is what 12 threads used to cost, and on the best-matched pair the two are now level. On the heaviest crystal ingest goes 13.5 s -> 9.9 s and the run 77.8 s -> 69.4 s. Battery 9m01s -> 8m24s, space group 21/24, no failures. Also restores get_gpu_numa_node() - the sysfs lookup deleted with NUMAHWPolicy - and an opt-in CPU pin to that node behind JFJOCH_PIN_CPU_TO_GPU_NODE. It is off because it measured neutral here: all four GPUs hang off two of the four nodes, so pinning to them costs a worker the other half of the machine. It is kept for boxes whose GPUs are spread over every socket. The old lookup had a latent bug - CUDA reports the PCI id with upper-case hex and sysfs paths are lower case, so on three of this machine's four GPUs it would have silently returned "unknown". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 10:53:47 -04:00
jungfrauandClaude Opus 5	f8b449a944	Score the first-pass validation frames together ... Two-pass rotation indexing picks between candidate lattices by forcing each one and counting how many of 60 validation frames it indexes. That count was a serial loop, and it is the single largest serial stretch outside scale/merge: 57% of the 4.9 s first-pass phase, a few Ceres solves per frame on one core while the other 47 and all four GPUs sit idle. The frames are scored together now. Each one's verdict is its own - the score is only how many index - and this is the same call the main image loop already makes from every one of its workers on this same IndexAndRefine, which writes nothing but unit_cells[] under its own mutex. With the candidate forced, GetLattice() returns it and the branch that would advance the indexer's own state is never reached. The offline solver stops on an iteration count rather than a clock, so a loaded machine cannot change a frame's verdict. The spot cache had to be filled first: it is a plain map filled on demand, and a lookup racing an insert is not something a map survives. Filling it stays serial and in frame order, so its contents do not depend on scheduling. First-pass scheme gaps on the heaviest crystal 4.93 s -> 2.06 s, with both schemes returning the same 60/60 they did before. Battery 9m23s -> 9m01s, space group 21/24, no failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 09:28:21 -04:00
jungfrauandClaude Opus 5	057fff98b2	Give each FFT direction a block and its histogram shared memory ... The de-novo indexer projects every spot onto each of 16384 search directions and bins the projections; the peak of each direction's spectrum is a reciprocal lattice row spacing. One thread owned a whole direction, so neighbouring lanes wrote 12.6 kB apart and every warp instruction touched 32 separate sectors of a 206 MB buffer with no chance of staying in a 4 MB L2. 160 million scattered global read-modify-writes, at about 14% of the card's bandwidth. One block per direction now, with the bins in shared memory. They are counts, so they are held as integers: an integer atomicAdd is a real shared-memory instruction where the float one compiles to a compare-and-swap retry loop, and a count below 2^24 converts to float exactly, so the output is bit for bit what the repeated += 1.0 produced. Above 48 kB of bins the old kernel still runs. 245.75 ms per launch -> 4.08 ms, so 0.98 s of the run -> 0.016 s. This machine runs two of them at once on two cards, so it is worth about half a second here and about a second on the single-GPU machines the viewer and the broker run on. The FFT it feeds takes 3.3 ms; preparing its input took 70x longer than transforming it. Alongside it, the per-frame scale fit divided by k^2 once per observation per IRLS iteration, and a loop-invariant divisor does not get hoisted out of a double division - ptxas emits the whole Newton refinement of the reciprocal every time. Hoisted, as 1/sigma already is a few lines above; the same expression in the three CPU scale paths went with it so the two stay algebraically identical. 54.92 ms per launch -> 44.56 ms, 1.65 s -> 1.34 s. That one is not bit-identical - a multiply by a rounded reciprocal differs from a correctly rounded quotient in the last place - so it can move a frame that sits on the convergence tolerance. Battery: 21/24 space groups, no failures, and 17 of 24 crystals identical to the previous run, against a floor of 13 of 24 for the same binary run twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 06:29:48 -04:00
jungfrauandClaude Opus 5	8ea3076f96	Take the serial passes out of the combine phase ... Four passes over all ten million partials and one over all four million fulls, each of them a single thread walking a fat struct: the snapshot of corr taken before the pass filters, the zeta filter, the frame rejection, the extraction of corr for the upload, and the scatter of the downloaded fulls arrays back into Obs. All are independent per element, so all are chunked now, and the split is gated on the amount of data rather than the core count. Measured on the heaviest crystal, summed over the run: corr snapshot 0.14 s -> 0.05, zeta filter 0.20 -> 0.04, corr upload 0.16 -> 0.07, fulls scatter 0.27 -> 0.06. Two things were tried in the same phase and are NOT here, both measured on the way past. Spreading the combine over the other three GPUs: the whole gpu_->Combine() call is 1.27 s over the entire run, so four cards could save about a second at best, but the partials would have to live on every device - 1.7 GB of transfer at the 1.5 GB/s this machine gets - and the fulls would have to be gathered back for the merge, which reads them where they are. The replication alone costs more than the best case saves. And staging the fulls download through a page-locked block, which reads like the textbook fix for copies running at 1.5 GB/s, measured 0.76 s -> 0.80 s: whatever limits them here, it is not the driver's bounce buffer. Battery 9m23s, space group 21/24, no failures; the crystals that moved are the two already known to sit on knife edges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 05:59:16 -04:00
jungfrauandClaude Opus 5	a1325637d2	Spread the scaling corrections and the space-group search over the cores ... Two thirds of a rotation run is one thread. The image loop is not the problem - on the heaviest crystal of the battery it is 1.8 s of 40 - and neither GPU nor CPU is saturated, because while the corrections and the space-group search run there is one core working and 47 idle. Mean occupancy over the whole run: 3.9 of 48. In the correction surfaces (absorption in the goniometer frame, detector-plane modulation, absorption against time and detector position - all one function): the per-cell accumulation, the score reduction and the final apply are now chunked, as are the three loops that assign a full to its cell, one of which spends a sine and a cosine per full de-rotating it into the crystal frame. Two full sorts of four million floats went with them: only the nine bin edges are wanted, so they are selected instead, each selection starting where the last one left off. The per-group pass is deliberately left serial. The terms of one group are spread all over the list, so the only way to give a thread groups of its own is to walk in group order, and that trades a near-sequential read of the fulls for a random one over a few hundred megabytes - the trade that already lost once in the combine kernel. The space-group search scores each candidate rotation by correlating I(h) against I(Rh) over the whole merge. Every operator it can ask about comes from a fixed list and none of them depend on each other, so they are scored up front, in parallel, and the search reads the cache. The scratch that stops a pair being counted twice is now per worker rather than shared. Worker counts are gated on how much work there is, not on how many cores the machine has (ThreadsForWork). Both parallel helpers start a thread per chunk, so a small dataset on a large node would otherwise pay for 48 thread starts to sum a few thousand terms - and this runs on 8-core laptops as well as on this node. Measured on the heaviest crystal, idle machine, two runs each, summed over both passes: those phases go 7.88 s -> 5.19 s. Whole-run wall time is the wrong ruler for it - it moves +-4 s between identical runs. Battery 9m45s -> 9m23s, space group 21/24, no failures; 16 of 24 crystals bit-identical to the previous run and the rest inside the noise floor of running one binary twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 02:38:20 -04:00
jungfrauandClaude Opus 5	f1dcfbe525	Give the process file its own thread ... Writing an image to the process file takes the global HDF5 mutex, which is the same one every worker needs to find its next image. The write is short - the file holds the per-image analysis, not the pixels - but with a worker per hardware thread they were still taking turns at it. The workers now post to a bounded queue and one thread owns the file. A DataMessage does not own its pixels, it points into the reader's buffer, so the raw image is parked in the queue beside its message; without that the worker frees the pixels on its next iteration and the writer reads whatever landed there. The queue is bounded at four per worker so a run whose analysis outpaces its writer cannot accumulate every image it has ever processed, and a write that throws - out of space, above all - is held and rethrown when the loop drains it, before the end message is written and the file finalized. Worth 6.8 s -> 6.5 s on a 16 Mpx rotation dataset at 48 workers, on top of the much larger gain from taking the read out of the same lock. Both process files, written with and without the writer thread, re-scale to the same 101215 unique reflections at the same ISa. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 01:09:02 -04:00
jungfrauandClaude Opus 5	f5b3193253	Ask HDF5 where an image is, then read it without the lock ... Two things every worker thread of an offline run did inside the global HDF5 mutex, per image. It opened /entry/data/data and asked it for its dataspace, its datatype and its creation plist, then asked those for the rank, the dimensions, the chunking and the compression. All of that is a property of the file and identical for all of its images, so it is now resolved once when the file is first touched. And it read the pixels - megabytes of them, with the lock held, which is what turned a worker per hardware thread into a queue. HDF5 can say where a chunk lives instead - address and byte count, a lookup in the chunk index with no read attached - so that is all it is asked for now, and the bytes are fetched after the lock is dropped, with a positional read that any number of threads can make through one handle at once. Chunk addresses count from the end of the user block, so its size is added; zero for anything this project writes, not for every file. A file that is not one chunk per image, or a chunk that was never written and exists only as a fill value, still goes the old way - only HDF5 knows what those read as. On a 16 Mpx rotation dataset with the process file being written, the per-image loop at 48 workers goes 12.4 s -> 6.8 s, and stops getting slower as workers are added: 8 workers were faster than 48 before, and are not now. Where no process file is written the same loop only improves ~1%, because this machine has 1.5 TB of RAM and held the whole 7 GB test set in page cache - the read was never the expensive part here. It is where the cache is cold or the filesystem is remote. Battery 9m45s, space group 21/24, no failures, unchanged. The Windows path uses ReadFile with an OVERLAPPED offset for the same reason pread is used elsewhere: it takes the offset as an argument rather than moving a shared file position, so the viewer keeps building under MSVC and gets the same concurrency. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-16 01:08:53 -04:00
jungfrauandClaude Opus 5	09cb9e2be9	Take the double precision out of the box integrator's inner loops ... boxsum summed its ring background in double and compared each ring pixel against a double threshold. The pixels are integers: a sum of at most a thousand int32 values is exact in a 64-bit integer AND exact in a double, so the two agree bit for bit, and comparing an integer against the floor of the threshold accepts exactly the same pixels as comparing it against the threshold itself. Both loops now do integer arithmetic. That was 39% of the card's double-precision pipe on the development machine and about three quarters of it on the production one, where the double rate is unchanged from Turing while the single rate has doubled - so this is worth more there than here. Alongside it, three things in the combine kernel. rr_nusable was computed by a whole extra walk over every observation and then never downloaded or read by anything. sum_wb and sum_cwb have no F in them, so they are the same in all three reweights and only the last round's values are ever used - two thirds of them were two divisions each, discarded. And CombineParams was the one parameter struct in the file without __restrict__, so the compiler could not assume the observation arrays and the freshly allocated fulls arrays were distinct. Measured on a crystal with 66 million partial observations: boxsum 12.2 s -> 8.3 s, the combine kernel 8.0 s -> 7.6 s, whole crystal 1m17s -> 1m12s. Battery 15m32s -> 9m59s. Same space group on all 24 crystals, none failed. Two things measured and NOT kept, recorded so they are not tried again: sorting the raw-hkl runs by length so a warp holds runs of similar length - it trades away the locality of neighbouring runs in the permutation and came out slower (7.6 s -> 8.8 s); and page-locking the integrator's host staging arrays individually - eleven separate registrations of small heap allocations overlap on shared pages and the driver refuses them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 23:22:29 -04:00
jungfrauandClaude Opus 5	3aa239fce8	Parallelise the incident-flux divide, drop a redundant sync ... DivideOutIncidentFlux was still the last fully serial pass in Ingest: a sweep over every observation to take each frame's mean background, and another to divide every rlp by its frame's flux. Ten gigabytes of traffic on one thread. The per-frame means go a frame at a time rather than an observation at a time, so each frame's running sum stays in one thread and in the order it had - splitting by observation would cut a frame across two threads and the partial sums would have to be recombined, which is a different sequence of roundings. The divide is per-element and splits anywhere. The adaptive spot finder synchronised after flagging strong pixels. The extractor that reads those pixels runs on the same stream, so the ordering already guaranteed the flagging had finished; the wait only idled the host, once per image. Measured on a crystal with 66 million partial observations: Ingest 8.5 s and 7.7 s -> 7.1 s and 6.6 s, whole crystal 1m24s -> 1m17s. Merged statistics unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 22:57:24 -04:00
jungfrauandClaude Opus 5	f36349060a	Sort the partials and the post-refine events in buckets, in parallel ... Both were one std::sort on one thread over tens of millions of elements, and together they were a third of a crowded crystal's run. Bucketing by h first makes them parallel. h is the comparator's leading key, so the sorted array is exactly the buckets laid end to end, and each bucket sorts on its own thread. In Ingest the keys are built straight into their bucket slot, so this replaces the build pass rather than adding one and the packed-key array is never duplicated; the extra memory is a few hundred kilobytes of histograms. Buckets are taken largest first, because the tail of the phase is whichever bucket finishes last. The run split falls out of the same structure for free: a run of equal (h,k,l) never crosses an h boundary, so each bucket counts its own runs, a scan over the buckets gives the offsets, and the arrays are sized exactly - which also removes the repeated growth the push_backs were paying for. The h range comes from the finiteness pass, which already reads every observation. The partials order became total in an earlier commit, when the observation index was added as the last key. That is what makes this safe rather than merely fast: the permutation is uniquely determined, so a bucket sort produces the same one a single sort would. Measured on a crystal with 66 million partial observations: Ingest 15.2 s and 14.3 s -> 8.3 s and 7.4 s, the post-refine event sort out of the top ten gaps entirely, the whole crystal 2m22s -> 1m24s. Battery 15m32s -> 10m05s. Same space group on all 24 crystals, none failed, and no crystal's R_meas moved by more than 0.3 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 22:31:05 -04:00
jungfrauandClaude Opus 5	621019140e	Reduce within the warp before the Bragg integration atomics ... fit and boxsum were 80% of GPU time on a crowded crystal - 78 s of it. Neither was bandwidth- or occupancy-bound: both sat at about an eighth of the issue rate the card can sustain, stalled. What stalls them is the block-wide accumulations. Every one has all 128 lanes of the block adding into one shared address, and a shared-memory atomicAdd on a float or a 64-bit integer has no instruction on either Turing or Ada - it compiles to a compare-and-swap retry loop. So those 128 lanes serialise into 128 retries, eighteen times per thread in fit. Summing across the warp first and letting one lane do the atomic leaves four per block instead of 128. That is the whole story: the arithmetic below was worth 2%, the atomics 5.6x. The arithmetic is still worth having, and is what was expected to matter: - compute_shell ran on all 128 threads of a block for a value that belongs to the reflection. It is two software double-precision divisions, on a card whose double throughput is a thirty-second (a sixty-fourth on the production one) of its single. One thread does it now. - The Kabsch inner loop divided by the same weight three times; the compiler emits the whole correctly-rounded sequence each time. One reciprocal now. Likewise the two Gaussian widths and the profile normalisation, which are constant over a reflection's cells and were divided per cell. - boxsum read the pixel before deciding whether it wanted it. The window is the bounding box of an ellipse, so nearly half of it is neither the signal disk nor the background ring, and those slots were fetching a cache line for nothing. Measured: fit 50.8 s -> 9.1 s, boxsum 27.5 s -> 12.1 s. A crowded crystal 2m22s -> 1m58s, a 16M-pixel one 39.5 s -> 37.2 s, the whole battery 12m30s -> 11m35s. Same space group on all 24 crystals, none failed. The integer sums are unchanged - addition is associative. The float ones move in their last bits and become more reproducible, since a fixed shuffle tree replaces whatever order the atomics arrived in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 22:15:00 -04:00
jungfrauandClaude Opus 5	2cbb3fc8b4	Build the GPU engines a worker never uses on first use, not always ... Every worker thread built a full set of analysis engines. Two of them are never asked for on the offline path: the fixed-threshold spot finder, because detection is adaptive by default, and the azimuthal integrator, because the fused adaptive finder produces the profile as a by-product. They are still needed elsewhere - the broker defaults to non-adaptive detection, and --no-adaptive-spots asks for the finder - so they are built on first use rather than removed. A lazily built finder takes the current resolution mask on construction; without that it would find spots outside the limits it was never told about. The bitshuffle decoder sized its output buffer for the widest pixel type there is rather than the one the images actually have, holding a second full frame per worker on 16-bit data. It is sized from the image now and grows if a later frame needs more. The shared-table checksum runs over eight interleaved lanes. FNV's multiply is a loop-carried dependency, so one chain retires a byte every few cycles whatever memory bandwidth is spare, and every worker hashes tens of megabytes of geometry tables as it builds its engines - about 5% of all CPU samples on a 16M-pixel detector. Measured on a 16M-pixel rotation dataset: cudaMalloc 11314 -> 9474 calls and, with cudaFree, 117 s -> 78 s of aggregate thread time; both synchronise the whole device, so that time is spent blocking every other worker. Whole battery 15m32s -> 12m30s. Data quality against main, over 24 crystals and eight statistics each: the same space group on all 24, and every difference smaller than what two runs of an IDENTICAL binary produce (measured: 13 of 24 crystals reproduce exactly run to run, worst R_meas swing 5.5 points, against 4.6 points for main vs this branch). The float atomics in the reductions have always made this so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 21:42:41 -04:00
jungfrauandClaude Opus 5	eb634400ea	Read four pixels at a time in the ring reduction ... reduce_rings_shared was 69% of all GPU kernel time - 116.8 s of a 70 s run across four cards. It is not bandwidth bound: flag_strong streams the same two arrays through the same grid-stride loop and reaches 196 GB/s, while this reached 30. The difference is the shared-memory atomics. Lanes in a warp read consecutive pixels along a detector row, a ring is a few pixels wide, so most of a warp lands in a handful of rings and the atomics to each one serialise. Two changes. The block reads four pixels per thread as one 16-byte and one 8-byte transaction, and merges the ones that fall in the same ring in registers before touching shared memory. Consecutive pixels usually DO share a ring, so this is where the win is: a run costs one set of atomics instead of one per pixel. npix is not guaranteed to be a multiple of four - it is width x height on the converted path, and detectors are not obliged to be even - so the vector loop stops short and a scalar loop finishes the remainder. Reading past the end would not fault, which is worse than if it did: it would fold uninitialised device memory into the accumulators and move the detection threshold in a way that does not reproduce. And the grid is sized from the occupancy the device reports, per pass. The two passes have different shared footprints - the first carries the corrected rings as well - so they do not fit the same number of blocks, and a grid sized for one left the other running a second wave at a quarter occupancy. The comment that justified the old grid reasoned from 1536 threads per SM, which is an Ada number; the card it ran on holds 1024. The run totals are still exactly what they were. The accumulators are unsigned 64-bit, so summing a run in a register and adding it once is the same value as adding each pixel separately - addition mod 2^64 is associative, overflow included - which is what keeps the ring statistics, and therefore the detection threshold, independent of how the work was grouped. That is the property the integer accumulators exist for. (The run accumulators are unsigned for the same reason: signed overflow would be undefined, and four squares of a large pixel value reach 2^64.) The corrected float sums, which feed the reported profile rather than any decision, change in their last bits as they already did between runs. Measured on a 16M-pixel rotation dataset: the kernel 116.8 s -> 19.1 s (6.1x), no longer the largest; the whole run 70 s -> 39.5 s. Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 12m47s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 19:21:24 -04:00
jungfrauandClaude Opus 5	56512414aa	Make the partials order total, and hoist 1/sigma out of the IRLS loop ... The sort that orders every observation by (h,k,l,image_number) was not a total order: two observations can genuinely share all four. The predictor emits BOTH intersections of a reflection's rotation circle with the Ewald sphere, and near the blind region - where zeta is smallest - the two are close enough in angle that both are accepted on the same frame. Which of them came first was then whatever the sort happened to produce. That was observable. The combine takes on_ice from the FIRST member of a rocking event, so the order decided whether a full was flagged as ice at all, and its per-event sums are floating point, so it moved intensities in their last bits. The observation's own index is now the final key, which orders them by arrival - and, more usefully, makes the order unique, so it no longer depends on which algorithm sorted it. sigma never changes once it is uploaded, so 1/sigma is the same in all thirty IRLS iterations of all three scaling iterations of all five scaling passes. It was being recomputed every time: a 64-bit reciprocal is a hardware estimate plus five refinement steps, and the profile put the three divisions in that loop at 21 of its 31 double-precision instructions. It is computed once now, in the pass that already streams every observation. The CPU has always hoisted it; this is the GPU catching up. Same expression on the same operand, so the value is what the loop used to compute, bit for bit. Also: PrepScaleObsKernel is not a grid-stride loop, but the scale-fulls path capped its grid at 65535 blocks like the grid-stride kernels around it. Above 16.8 million fulls that silently left the tail of sco_coeff/sco_ok stale. No dataset here reaches it; the cap is simply wrong for that kernel. And the AoS-to-SoA staging that feeds the GPU - the widest pass in Ingest, reading an 80-byte struct and writing fourteen arrays out of it - ran on one thread. Full 24-crystal battery: same space group on all 24, none failed, one crystal moved R_meas by 0.8 points with CC unchanged (it moves by that much between runs of an identical binary). 15m32s -> 13m35s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 18:56:26 -04:00
jungfrauandClaude Opus 5	e11c2a2b20	Fit the goniometer rotation scale in closed form ... The fit has ONE parameter, and it was handed to Ceres as one residual block per rocking event - 8 million of them on a large crystal. Each block is a functor, an auto-diff cost function and a loss object on the heap, and the solver then factorises an 8-million-by-one Jacobian on every iteration. It cost 13.7 s. The residual is closed-form in k. A rotation preserves length, so |p_lab| is |e_mid| whatever k is and only the z component moves; Rodrigues gives it exactly: r(k) = C + A cos(a k) - B sin(a k) = C + R cos(a k + psi) C = lambda |e|^2 / 2 + u_z (u.e), A = e_z - u_z (u.e), B = (u x e)_z with a the event's angle from the sweep centre. That is the same function the functor computes - Ceres uses the exact Rodrigues form here, so there is no small-angle branch to disagree with - and it reduces the fit to minimising a smooth function of one variable over the interval the solver was bounded to. It is scanned on a grid and then closed in by golden section; the objective's curvature jumps wherever an event crosses the Huber knee, which is why this is not a Newton iteration. The coefficients are computed in double and stored narrowed. Their rounding moves the minimiser by ~1e-10, and k is carried downstream as a float, so the committed value is the same to far more digits than anything reads. One pass over the events yields the five per-fifth partial sums, so the all-data fit and the five leave-a-fifth-out folds share it. That matters because the jackknife only runs when the fit is big enough to act on, and on a crystal that trips it the old code paid for six full solves. The partials gather ahead of it counted first and then filled instead of growing one vector by push_back tens of millions of times, which copied the whole thing on every doubling. Measured: unchanged verdict and k to five decimals on the regression crystals. Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 13m35s together with the scale/merge changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 18:56:09 -04:00
jungfrauandClaude Opus 5	6368c00173	Decode and accumulate the beam-stop projection on the GPU ... The pre-scan decompressed its frames on the host and folded them into a per-pixel projection there. On a 16M-pixel detector that is 60 frames of 72 MB to decompress and 20 bytes per pixel to read and write back per frame - about 40 GB of memory traffic - and it was the whole cost of the phase once the mask was no longer the bottleneck. Only the compressed chunk crosses PCIe now. BSLZ4DecoderGPU already exposes the raw decoded bytes (Decode(), the path its own tests use), which is what this needs: the projection is defined on the RAW STORED COUNTS with the pixel type's sentinel skipped, not on the preprocessed image, so nothing here goes through the preprocessor. Sums, maxima and counts are integers, so the device result is identical to the host's rather than merely close. Frames are folded in batches of four. The fold reads and writes the whole accumulator whatever the batch holds, so per frame it was spending most of the bandwidth on the accumulator rather than on the data; four is where that stops mattering, and every frame beyond it is another full frame of device memory, which costs more in cudaMalloc - device-synchronizing - than it saves. The accumulator is built on a thread of its own. It allocates and clears several hundred megabytes, and doing that in the constructor stalled the caller before it had read its first frame. Frames the device cannot take - anything but bitshuffle+LZ4 - still go to a host shard, so a run mixing compressions needs no second code path, and a build without CUDA is unchanged. RotationScaleMergeGPU set the CUDA device in its constructor and never put it back. CUDA's current device is per-thread, so that silently re-pinned the calling thread for the rest of its life, and the destructor freed several gigabytes against whatever device happened to be current by then - CudaDevicePtr records no device of its own. Every entry point now sets the device on entry and restores it on exit. ParallelFor/ParallelChunks moved to common/ParallelFor.h; two files had copies and a third wants them. Measured on a 16M-pixel rotation dataset: pre-scan 4.78 s -> 2.37 s -> ~2.0 s, shadow unchanged at 139126 pixels (22143 on a 2M-pixel dataset). Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 14m49s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 18:38:26 -04:00
jungfrauandClaude Opus 5	2f586d2267	Stop allocating GPU and pinned memory nothing reads ... Three resource fixes and two latent bugs, none of which changes a computed number. The preprocessed image has a host copy that only a CPU engine ever reads. On the GPU path every engine reads the device buffer instead, and rugnux always runs the fused adaptive finder, so that host copy is allocated, zeroed and PAGE-LOCKED for nothing - 72 MB per worker, 3.5 GB over 48 of them, and a cudaHostRegister each, which the driver serializes. It is now skipped by the same condition that already decides whether the device copies the image back. ImagePreprocessorBuffer keeps the pixel count separately so size() still answers when the mirror was not allocated. ROIIntegrationGPU asked device 0 for the SM count it sizes its grid from, while workers are pinned round-robin across the GPUs - so on a multi-GPU node it could size a grid from a card it never launches on. It asks the current device now, like every other engine. ~CudaRegisteredVector called a function that throws out of a destructor, and the move-assignment did the same from a noexcept function. Either would abort the process rather than report the failure, and teardown - after a device reset, or while another exception unwinds - is exactly where cudaHostUnregister fails. Both now use an unchecked unregister, as every other destructor in that header already does for its own teardown call. The throwing form stays for rebind()/unregister(), which are called from live code. Measured on a 16M-pixel rotation dataset: unchanged space group, merged reflection count and merging statistics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 17:50:19 -04:00
jungfrauandClaude Opus 5	50941f8df0	Make the beam-stop mask O(pixels) and parallel ... GetMask() was 3.28 s of the 4.78 s pre-scan on a 16M-pixel detector, all on one thread. Four changes, none of which alters the mask: dilate() was a multi-source BFS. On a full rectangle with no obstacles the 8-connected graph distance IS the Chebyshev distance - a path stepping towards the target never has to leave the frame - so the result is a dilation by the (2r+1) square clipped to the frame, which separates into a pass along x and a pass along y. That is O(1) per pixel whatever r is, with no queue and no 4-bytes-per-pixel distance array (72 MB, allocated and filled five times per call). The erode() case is the one that hurt: it dilates the COMPLEMENT, so on a detector whose shadow is under 1% of the pixels it seeded the BFS from essentially every pixel. fill_holes() floods the background from the border. It now floods the bounding box of the region grown by one: everything outside that box is background and the box's own ring is background, so the whole outside is one border-connected component and a background pixel inside the box is border-connected exactly when it reaches the ring. The three baseline iterations re-binned every pixel by radius and re-took a median each time. The iteration only ever excludes pixels whose background is below a cut, and dividing by a positive baseline is monotone, so a ring's excluded pixels are exactly its lowest ones and the next median is an order statistic of the same, unchanging ring. The rings are binned and sorted once; each iteration then picks a rank and counts a prefix. Nine full-image passes become one. box_sum's vertical pass walked one column at a time, striding a whole row per step and missing on every access; it now carries a strip of columns together. Each row's and each column's running sum keeps its terms in its order, so the floating-point rounding is unchanged - only the traversal differs. The pooled COUNT is a count of at most 25 pixels, so it is an exact integer box sum now rather than a floating-point one; the background itself stays in double, because its running sum adds and subtracts across a whole row and in float the two roundings would not cancel. The per-pixel passes then run on all threads, and GetMask takes a thread count. Measured on a 16M-pixel rotation dataset: GetMask 3.28 s -> 0.99 s, whole pre-scan 4.78 s -> 2.37 s, whole run 1m10s -> 1m03s. The mask is unchanged on both a 16M and a 2M-pixel dataset (139126 and 22143 shadow pixels), as are the space group, the merged reflection count and the merging statistics. Also corrected the comment on erode(): the dilation cannot seed outside the frame, so outside behaves as foreground, not as complement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 17:50:06 -04:00
jungfrauandClaude Opus 5	f12e3b6252	Parallelize the beam-stop pre-scan ... The pre-scan read its sample of frames in a plain serial loop: one thread did the HDF5 read, the decompression and the full-detector accumulation for every frame. The cost is fixed per frame rather than per dataset, so it grew straight with detector area - measured at 0.9 s on a 2M-pixel detector and 7.9 s on a 16M-pixel one, where it was 11% of the whole run with 47 of 48 cores idle. Frames are now read on several workers. ShadowFinder keeps one projection per worker so nothing is locked while an image is added, and the projections are summed when the mask is read; the sums and counts are integers, so the result does not depend on how the frames were spread over the workers. A shard that never counted a pixel is skipped when the maxima are merged - it holds 0, which would otherwise beat a genuinely negative maximum. Worker count is capped (PRESCAN_MAX_WORKERS): a shard costs 20 bytes per pixel, and the accumulation is memory-bound, so a handful of workers already saturates it. The beam-centre spot pool is stitched together in sample order after the workers join, so frame numbering and the spot list are what the serial read produced regardless of how the workers interleaved. A frame still joins the pool only if it could be read. ShadowFinder::AddImage took its decompression scratch buffer BY VALUE, so the caller's buffer stayed empty and every frame allocated and zero-filled a fresh full-size uncompressed image (72 MB on a 16M-pixel detector) and freed it again. It takes a reference now, and each worker reuses one buffer. Measured on a 16M-pixel rotation dataset: pre-scan 7.9 s -> 4.8 s, whole run 69.2 s -> 65.1 s. Results are unchanged - same shadow pixel count, same space group, same merged reflection count and merging statistics on both a 16M and a 2M-pixel dataset, and the beam-centre path still commits the same centre. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 17:13:53 -04:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.161	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.160	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.159	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.158	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.157	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.156	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65 1.0.0-rc.155	2026-06-25 22:01:48 +02:00
leonarski_f	6136f858af	v1.0.0-rc.154 (#64) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.154	2026-06-25 18:12:00 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63 1.0.0-rc.153	2026-06-23 20:29:49 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62 1.0.0-rc.152	2026-06-17 20:36:24 +02:00
leonarski_f	ef52dac2ee	v1.0.0-rc.151 (#61) ... * jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname Reviewed-on: #61 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.151	2026-06-16 14:13:29 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60 1.0.0-rc.150	2026-06-15 20:24:15 +02:00
leonarski_f	ea575f790a	v1.0.0-rc.149 (#59) ... * XDS plugin: Fix HDF5 mutex to run on multiple processors Reviewed-on: #59 1.0.0-rc.149	2026-06-13 21:27:41 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58 1.0.0-rc.148	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56 1.0.0-rc.146	2026-05-28 18:48:35 +02:00
leonarski_f	75f1c5f954	SHIM library improvements from the HDF Group	2026-05-08 11:39:51 +02:00
leonarski_f	caef26873e	v1.0.0-rc.145 (#55) ... This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144. * **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future. * XDS plugin: Improve performance of VDS reading. * jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD. * jfjoch_writer: Clean-up of HDF5 routines to better handle issues. Reviewed-on: #55 1.0.0-rc.145	2026-05-06 21:50:02 +02:00
leonarski_f	7d34e8a049	v1.0.0-rc.144 (#54) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Improve performance of preview JPEG image generator at receiver startup (saving about 150 ms on measurement start for 16M) Reviewed-on: #54 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.144	2026-05-01 17:06:36 +02:00
leonarski_f	6e0bb971ac	v1.0.0-rc.143 (#53) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Avoid copying gain calibration together with DiffractionExperiment Reviewed-on: #53 1.0.0-rc.143	2026-04-30 22:16:50 +02:00
leonarski_f	87fde1b32e	v1.0.0-rc.142 (#52) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8 * Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent Reviewed-on: #52 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 16:47:53 +02:00
leonarski_f	d760b12a18	v1.0.0-rc.141 (#51) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times * frontend: Fix selection of FFTW in indexing settings Reviewed-on: #51 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 13:04:54 +02:00