1120 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
jungfrauandClaude Opus 5	09cb9e2be9	Take the double precision out of the box integrator's inner loops ... boxsum summed its ring background in double and compared each ring pixel against a double threshold. The pixels are integers: a sum of at most a thousand int32 values is exact in a 64-bit integer AND exact in a double, so the two agree bit for bit, and comparing an integer against the floor of the threshold accepts exactly the same pixels as comparing it against the threshold itself. Both loops now do integer arithmetic. That was 39% of the card's double-precision pipe on the development machine and about three quarters of it on the production one, where the double rate is unchanged from Turing while the single rate has doubled - so this is worth more there than here. Alongside it, three things in the combine kernel. rr_nusable was computed by a whole extra walk over every observation and then never downloaded or read by anything. sum_wb and sum_cwb have no F in them, so they are the same in all three reweights and only the last round's values are ever used - two thirds of them were two divisions each, discarded. And CombineParams was the one parameter struct in the file without __restrict__, so the compiler could not assume the observation arrays and the freshly allocated fulls arrays were distinct. Measured on a crystal with 66 million partial observations: boxsum 12.2 s -> 8.3 s, the combine kernel 8.0 s -> 7.6 s, whole crystal 1m17s -> 1m12s. Battery 15m32s -> 9m59s. Same space group on all 24 crystals, none failed. Two things measured and NOT kept, recorded so they are not tried again: sorting the raw-hkl runs by length so a warp holds runs of similar length - it trades away the locality of neighbouring runs in the permutation and came out slower (7.6 s -> 8.8 s); and page-locking the integrator's host staging arrays individually - eleven separate registrations of small heap allocations overlap on shared pages and the driver refuses them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 23:22:29 -04:00
jungfrauandClaude Opus 5	3aa239fce8	Parallelise the incident-flux divide, drop a redundant sync ... DivideOutIncidentFlux was still the last fully serial pass in Ingest: a sweep over every observation to take each frame's mean background, and another to divide every rlp by its frame's flux. Ten gigabytes of traffic on one thread. The per-frame means go a frame at a time rather than an observation at a time, so each frame's running sum stays in one thread and in the order it had - splitting by observation would cut a frame across two threads and the partial sums would have to be recombined, which is a different sequence of roundings. The divide is per-element and splits anywhere. The adaptive spot finder synchronised after flagging strong pixels. The extractor that reads those pixels runs on the same stream, so the ordering already guaranteed the flagging had finished; the wait only idled the host, once per image. Measured on a crystal with 66 million partial observations: Ingest 8.5 s and 7.7 s -> 7.1 s and 6.6 s, whole crystal 1m24s -> 1m17s. Merged statistics unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 22:57:24 -04:00
jungfrauandClaude Opus 5	f36349060a	Sort the partials and the post-refine events in buckets, in parallel ... Both were one std::sort on one thread over tens of millions of elements, and together they were a third of a crowded crystal's run. Bucketing by h first makes them parallel. h is the comparator's leading key, so the sorted array is exactly the buckets laid end to end, and each bucket sorts on its own thread. In Ingest the keys are built straight into their bucket slot, so this replaces the build pass rather than adding one and the packed-key array is never duplicated; the extra memory is a few hundred kilobytes of histograms. Buckets are taken largest first, because the tail of the phase is whichever bucket finishes last. The run split falls out of the same structure for free: a run of equal (h,k,l) never crosses an h boundary, so each bucket counts its own runs, a scan over the buckets gives the offsets, and the arrays are sized exactly - which also removes the repeated growth the push_backs were paying for. The h range comes from the finiteness pass, which already reads every observation. The partials order became total in an earlier commit, when the observation index was added as the last key. That is what makes this safe rather than merely fast: the permutation is uniquely determined, so a bucket sort produces the same one a single sort would. Measured on a crystal with 66 million partial observations: Ingest 15.2 s and 14.3 s -> 8.3 s and 7.4 s, the post-refine event sort out of the top ten gaps entirely, the whole crystal 2m22s -> 1m24s. Battery 15m32s -> 10m05s. Same space group on all 24 crystals, none failed, and no crystal's R_meas moved by more than 0.3 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 22:31:05 -04:00
jungfrauandClaude Opus 5	621019140e	Reduce within the warp before the Bragg integration atomics ... fit and boxsum were 80% of GPU time on a crowded crystal - 78 s of it. Neither was bandwidth- or occupancy-bound: both sat at about an eighth of the issue rate the card can sustain, stalled. What stalls them is the block-wide accumulations. Every one has all 128 lanes of the block adding into one shared address, and a shared-memory atomicAdd on a float or a 64-bit integer has no instruction on either Turing or Ada - it compiles to a compare-and-swap retry loop. So those 128 lanes serialise into 128 retries, eighteen times per thread in fit. Summing across the warp first and letting one lane do the atomic leaves four per block instead of 128. That is the whole story: the arithmetic below was worth 2%, the atomics 5.6x. The arithmetic is still worth having, and is what was expected to matter: - compute_shell ran on all 128 threads of a block for a value that belongs to the reflection. It is two software double-precision divisions, on a card whose double throughput is a thirty-second (a sixty-fourth on the production one) of its single. One thread does it now. - The Kabsch inner loop divided by the same weight three times; the compiler emits the whole correctly-rounded sequence each time. One reciprocal now. Likewise the two Gaussian widths and the profile normalisation, which are constant over a reflection's cells and were divided per cell. - boxsum read the pixel before deciding whether it wanted it. The window is the bounding box of an ellipse, so nearly half of it is neither the signal disk nor the background ring, and those slots were fetching a cache line for nothing. Measured: fit 50.8 s -> 9.1 s, boxsum 27.5 s -> 12.1 s. A crowded crystal 2m22s -> 1m58s, a 16M-pixel one 39.5 s -> 37.2 s, the whole battery 12m30s -> 11m35s. Same space group on all 24 crystals, none failed. The integer sums are unchanged - addition is associative. The float ones move in their last bits and become more reproducible, since a fixed shuffle tree replaces whatever order the atomics arrived in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 22:15:00 -04:00
jungfrauandClaude Opus 5	2cbb3fc8b4	Build the GPU engines a worker never uses on first use, not always ... Every worker thread built a full set of analysis engines. Two of them are never asked for on the offline path: the fixed-threshold spot finder, because detection is adaptive by default, and the azimuthal integrator, because the fused adaptive finder produces the profile as a by-product. They are still needed elsewhere - the broker defaults to non-adaptive detection, and --no-adaptive-spots asks for the finder - so they are built on first use rather than removed. A lazily built finder takes the current resolution mask on construction; without that it would find spots outside the limits it was never told about. The bitshuffle decoder sized its output buffer for the widest pixel type there is rather than the one the images actually have, holding a second full frame per worker on 16-bit data. It is sized from the image now and grows if a later frame needs more. The shared-table checksum runs over eight interleaved lanes. FNV's multiply is a loop-carried dependency, so one chain retires a byte every few cycles whatever memory bandwidth is spare, and every worker hashes tens of megabytes of geometry tables as it builds its engines - about 5% of all CPU samples on a 16M-pixel detector. Measured on a 16M-pixel rotation dataset: cudaMalloc 11314 -> 9474 calls and, with cudaFree, 117 s -> 78 s of aggregate thread time; both synchronise the whole device, so that time is spent blocking every other worker. Whole battery 15m32s -> 12m30s. Data quality against main, over 24 crystals and eight statistics each: the same space group on all 24, and every difference smaller than what two runs of an IDENTICAL binary produce (measured: 13 of 24 crystals reproduce exactly run to run, worst R_meas swing 5.5 points, against 4.6 points for main vs this branch). The float atomics in the reductions have always made this so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 21:42:41 -04:00
jungfrauandClaude Opus 5	eb634400ea	Read four pixels at a time in the ring reduction ... reduce_rings_shared was 69% of all GPU kernel time - 116.8 s of a 70 s run across four cards. It is not bandwidth bound: flag_strong streams the same two arrays through the same grid-stride loop and reaches 196 GB/s, while this reached 30. The difference is the shared-memory atomics. Lanes in a warp read consecutive pixels along a detector row, a ring is a few pixels wide, so most of a warp lands in a handful of rings and the atomics to each one serialise. Two changes. The block reads four pixels per thread as one 16-byte and one 8-byte transaction, and merges the ones that fall in the same ring in registers before touching shared memory. Consecutive pixels usually DO share a ring, so this is where the win is: a run costs one set of atomics instead of one per pixel. npix is not guaranteed to be a multiple of four - it is width x height on the converted path, and detectors are not obliged to be even - so the vector loop stops short and a scalar loop finishes the remainder. Reading past the end would not fault, which is worse than if it did: it would fold uninitialised device memory into the accumulators and move the detection threshold in a way that does not reproduce. And the grid is sized from the occupancy the device reports, per pass. The two passes have different shared footprints - the first carries the corrected rings as well - so they do not fit the same number of blocks, and a grid sized for one left the other running a second wave at a quarter occupancy. The comment that justified the old grid reasoned from 1536 threads per SM, which is an Ada number; the card it ran on holds 1024. The run totals are still exactly what they were. The accumulators are unsigned 64-bit, so summing a run in a register and adding it once is the same value as adding each pixel separately - addition mod 2^64 is associative, overflow included - which is what keeps the ring statistics, and therefore the detection threshold, independent of how the work was grouped. That is the property the integer accumulators exist for. (The run accumulators are unsigned for the same reason: signed overflow would be undefined, and four squares of a large pixel value reach 2^64.) The corrected float sums, which feed the reported profile rather than any decision, change in their last bits as they already did between runs. Measured on a 16M-pixel rotation dataset: the kernel 116.8 s -> 19.1 s (6.1x), no longer the largest; the whole run 70 s -> 39.5 s. Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 12m47s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 19:21:24 -04:00
jungfrauandClaude Opus 5	56512414aa	Make the partials order total, and hoist 1/sigma out of the IRLS loop ... The sort that orders every observation by (h,k,l,image_number) was not a total order: two observations can genuinely share all four. The predictor emits BOTH intersections of a reflection's rotation circle with the Ewald sphere, and near the blind region - where zeta is smallest - the two are close enough in angle that both are accepted on the same frame. Which of them came first was then whatever the sort happened to produce. That was observable. The combine takes on_ice from the FIRST member of a rocking event, so the order decided whether a full was flagged as ice at all, and its per-event sums are floating point, so it moved intensities in their last bits. The observation's own index is now the final key, which orders them by arrival - and, more usefully, makes the order unique, so it no longer depends on which algorithm sorted it. sigma never changes once it is uploaded, so 1/sigma is the same in all thirty IRLS iterations of all three scaling iterations of all five scaling passes. It was being recomputed every time: a 64-bit reciprocal is a hardware estimate plus five refinement steps, and the profile put the three divisions in that loop at 21 of its 31 double-precision instructions. It is computed once now, in the pass that already streams every observation. The CPU has always hoisted it; this is the GPU catching up. Same expression on the same operand, so the value is what the loop used to compute, bit for bit. Also: PrepScaleObsKernel is not a grid-stride loop, but the scale-fulls path capped its grid at 65535 blocks like the grid-stride kernels around it. Above 16.8 million fulls that silently left the tail of sco_coeff/sco_ok stale. No dataset here reaches it; the cap is simply wrong for that kernel. And the AoS-to-SoA staging that feeds the GPU - the widest pass in Ingest, reading an 80-byte struct and writing fourteen arrays out of it - ran on one thread. Full 24-crystal battery: same space group on all 24, none failed, one crystal moved R_meas by 0.8 points with CC unchanged (it moves by that much between runs of an identical binary). 15m32s -> 13m35s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 18:56:26 -04:00
jungfrauandClaude Opus 5	e11c2a2b20	Fit the goniometer rotation scale in closed form ... The fit has ONE parameter, and it was handed to Ceres as one residual block per rocking event - 8 million of them on a large crystal. Each block is a functor, an auto-diff cost function and a loss object on the heap, and the solver then factorises an 8-million-by-one Jacobian on every iteration. It cost 13.7 s. The residual is closed-form in k. A rotation preserves length, so |p_lab| is |e_mid| whatever k is and only the z component moves; Rodrigues gives it exactly: r(k) = C + A cos(a k) - B sin(a k) = C + R cos(a k + psi) C = lambda |e|^2 / 2 + u_z (u.e), A = e_z - u_z (u.e), B = (u x e)_z with a the event's angle from the sweep centre. That is the same function the functor computes - Ceres uses the exact Rodrigues form here, so there is no small-angle branch to disagree with - and it reduces the fit to minimising a smooth function of one variable over the interval the solver was bounded to. It is scanned on a grid and then closed in by golden section; the objective's curvature jumps wherever an event crosses the Huber knee, which is why this is not a Newton iteration. The coefficients are computed in double and stored narrowed. Their rounding moves the minimiser by ~1e-10, and k is carried downstream as a float, so the committed value is the same to far more digits than anything reads. One pass over the events yields the five per-fifth partial sums, so the all-data fit and the five leave-a-fifth-out folds share it. That matters because the jackknife only runs when the fit is big enough to act on, and on a crystal that trips it the old code paid for six full solves. The partials gather ahead of it counted first and then filled instead of growing one vector by push_back tens of millions of times, which copied the whole thing on every doubling. Measured: unchanged verdict and k to five decimals on the regression crystals. Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 13m35s together with the scale/merge changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 18:56:09 -04:00
jungfrauandClaude Opus 5	6368c00173	Decode and accumulate the beam-stop projection on the GPU ... The pre-scan decompressed its frames on the host and folded them into a per-pixel projection there. On a 16M-pixel detector that is 60 frames of 72 MB to decompress and 20 bytes per pixel to read and write back per frame - about 40 GB of memory traffic - and it was the whole cost of the phase once the mask was no longer the bottleneck. Only the compressed chunk crosses PCIe now. BSLZ4DecoderGPU already exposes the raw decoded bytes (Decode(), the path its own tests use), which is what this needs: the projection is defined on the RAW STORED COUNTS with the pixel type's sentinel skipped, not on the preprocessed image, so nothing here goes through the preprocessor. Sums, maxima and counts are integers, so the device result is identical to the host's rather than merely close. Frames are folded in batches of four. The fold reads and writes the whole accumulator whatever the batch holds, so per frame it was spending most of the bandwidth on the accumulator rather than on the data; four is where that stops mattering, and every frame beyond it is another full frame of device memory, which costs more in cudaMalloc - device-synchronizing - than it saves. The accumulator is built on a thread of its own. It allocates and clears several hundred megabytes, and doing that in the constructor stalled the caller before it had read its first frame. Frames the device cannot take - anything but bitshuffle+LZ4 - still go to a host shard, so a run mixing compressions needs no second code path, and a build without CUDA is unchanged. RotationScaleMergeGPU set the CUDA device in its constructor and never put it back. CUDA's current device is per-thread, so that silently re-pinned the calling thread for the rest of its life, and the destructor freed several gigabytes against whatever device happened to be current by then - CudaDevicePtr records no device of its own. Every entry point now sets the device on entry and restores it on exit. ParallelFor/ParallelChunks moved to common/ParallelFor.h; two files had copies and a third wants them. Measured on a 16M-pixel rotation dataset: pre-scan 4.78 s -> 2.37 s -> ~2.0 s, shadow unchanged at 139126 pixels (22143 on a 2M-pixel dataset). Full 24-crystal battery: same space group on all 24, none failed, 15m32s -> 14m49s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 18:38:26 -04:00
jungfrauandClaude Opus 5	2f586d2267	Stop allocating GPU and pinned memory nothing reads ... Three resource fixes and two latent bugs, none of which changes a computed number. The preprocessed image has a host copy that only a CPU engine ever reads. On the GPU path every engine reads the device buffer instead, and rugnux always runs the fused adaptive finder, so that host copy is allocated, zeroed and PAGE-LOCKED for nothing - 72 MB per worker, 3.5 GB over 48 of them, and a cudaHostRegister each, which the driver serializes. It is now skipped by the same condition that already decides whether the device copies the image back. ImagePreprocessorBuffer keeps the pixel count separately so size() still answers when the mirror was not allocated. ROIIntegrationGPU asked device 0 for the SM count it sizes its grid from, while workers are pinned round-robin across the GPUs - so on a multi-GPU node it could size a grid from a card it never launches on. It asks the current device now, like every other engine. ~CudaRegisteredVector called a function that throws out of a destructor, and the move-assignment did the same from a noexcept function. Either would abort the process rather than report the failure, and teardown - after a device reset, or while another exception unwinds - is exactly where cudaHostUnregister fails. Both now use an unchecked unregister, as every other destructor in that header already does for its own teardown call. The throwing form stays for rebind()/unregister(), which are called from live code. Measured on a 16M-pixel rotation dataset: unchanged space group, merged reflection count and merging statistics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 17:50:19 -04:00
jungfrauandClaude Opus 5	50941f8df0	Make the beam-stop mask O(pixels) and parallel ... GetMask() was 3.28 s of the 4.78 s pre-scan on a 16M-pixel detector, all on one thread. Four changes, none of which alters the mask: dilate() was a multi-source BFS. On a full rectangle with no obstacles the 8-connected graph distance IS the Chebyshev distance - a path stepping towards the target never has to leave the frame - so the result is a dilation by the (2r+1) square clipped to the frame, which separates into a pass along x and a pass along y. That is O(1) per pixel whatever r is, with no queue and no 4-bytes-per-pixel distance array (72 MB, allocated and filled five times per call). The erode() case is the one that hurt: it dilates the COMPLEMENT, so on a detector whose shadow is under 1% of the pixels it seeded the BFS from essentially every pixel. fill_holes() floods the background from the border. It now floods the bounding box of the region grown by one: everything outside that box is background and the box's own ring is background, so the whole outside is one border-connected component and a background pixel inside the box is border-connected exactly when it reaches the ring. The three baseline iterations re-binned every pixel by radius and re-took a median each time. The iteration only ever excludes pixels whose background is below a cut, and dividing by a positive baseline is monotone, so a ring's excluded pixels are exactly its lowest ones and the next median is an order statistic of the same, unchanging ring. The rings are binned and sorted once; each iteration then picks a rank and counts a prefix. Nine full-image passes become one. box_sum's vertical pass walked one column at a time, striding a whole row per step and missing on every access; it now carries a strip of columns together. Each row's and each column's running sum keeps its terms in its order, so the floating-point rounding is unchanged - only the traversal differs. The pooled COUNT is a count of at most 25 pixels, so it is an exact integer box sum now rather than a floating-point one; the background itself stays in double, because its running sum adds and subtracts across a whole row and in float the two roundings would not cancel. The per-pixel passes then run on all threads, and GetMask takes a thread count. Measured on a 16M-pixel rotation dataset: GetMask 3.28 s -> 0.99 s, whole pre-scan 4.78 s -> 2.37 s, whole run 1m10s -> 1m03s. The mask is unchanged on both a 16M and a 2M-pixel dataset (139126 and 22143 shadow pixels), as are the space group, the merged reflection count and the merging statistics. Also corrected the comment on erode(): the dilation cannot seed outside the frame, so outside behaves as foreground, not as complement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 17:50:06 -04:00
jungfrauandClaude Opus 5	f12e3b6252	Parallelize the beam-stop pre-scan ... The pre-scan read its sample of frames in a plain serial loop: one thread did the HDF5 read, the decompression and the full-detector accumulation for every frame. The cost is fixed per frame rather than per dataset, so it grew straight with detector area - measured at 0.9 s on a 2M-pixel detector and 7.9 s on a 16M-pixel one, where it was 11% of the whole run with 47 of 48 cores idle. Frames are now read on several workers. ShadowFinder keeps one projection per worker so nothing is locked while an image is added, and the projections are summed when the mask is read; the sums and counts are integers, so the result does not depend on how the frames were spread over the workers. A shard that never counted a pixel is skipped when the maxima are merged - it holds 0, which would otherwise beat a genuinely negative maximum. Worker count is capped (PRESCAN_MAX_WORKERS): a shard costs 20 bytes per pixel, and the accumulation is memory-bound, so a handful of workers already saturates it. The beam-centre spot pool is stitched together in sample order after the workers join, so frame numbering and the spot list are what the serial read produced regardless of how the workers interleaved. A frame still joins the pool only if it could be read. ShadowFinder::AddImage took its decompression scratch buffer BY VALUE, so the caller's buffer stayed empty and every frame allocated and zero-filled a fresh full-size uncompressed image (72 MB on a 16M-pixel detector) and freed it again. It takes a reference now, and each worker reuses one buffer. Measured on a 16M-pixel rotation dataset: pre-scan 7.9 s -> 4.8 s, whole run 69.2 s -> 65.1 s. Results are unchanged - same shadow pixel count, same space group, same merged reflection count and merging statistics on both a 16M and a 2M-pixel dataset, and the beam-centre path still commits the same centre. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-15 17:13:53 -04:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.161	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.160	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.159	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.158	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.157	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.156	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65 1.0.0-rc.155	2026-06-25 22:01:48 +02:00
leonarski_f	6136f858af	v1.0.0-rc.154 (#64) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.154	2026-06-25 18:12:00 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63 1.0.0-rc.153	2026-06-23 20:29:49 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62 1.0.0-rc.152	2026-06-17 20:36:24 +02:00
leonarski_f	ef52dac2ee	v1.0.0-rc.151 (#61) ... * jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname Reviewed-on: #61 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.151	2026-06-16 14:13:29 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60 1.0.0-rc.150	2026-06-15 20:24:15 +02:00
leonarski_f	ea575f790a	v1.0.0-rc.149 (#59) ... * XDS plugin: Fix HDF5 mutex to run on multiple processors Reviewed-on: #59 1.0.0-rc.149	2026-06-13 21:27:41 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58 1.0.0-rc.148	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56 1.0.0-rc.146	2026-05-28 18:48:35 +02:00
leonarski_f	75f1c5f954	SHIM library improvements from the HDF Group	2026-05-08 11:39:51 +02:00
leonarski_f	caef26873e	v1.0.0-rc.145 (#55) ... This is an UNSTABLE release. The release has significant modifications for HDF5 writing logic - in case of troubles go back to 1.0.0-rc.144. * **Default HDF5 writing mode is with VDS, not soft-links** - this improves DIALS compatibility and makes format more future-proof, NXmx legacy format might be phased-out in the future. * XDS plugin: Improve performance of VDS reading. * jfjoch_writer: Significant improvement on how file systems I/O are handled through a dedicated pass-through VFD. * jfjoch_writer: Clean-up of HDF5 routines to better handle issues. Reviewed-on: #55 1.0.0-rc.145	2026-05-06 21:50:02 +02:00
leonarski_f	7d34e8a049	v1.0.0-rc.144 (#54) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Improve performance of preview JPEG image generator at receiver startup (saving about 150 ms on measurement start for 16M) Reviewed-on: #54 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.144	2026-05-01 17:06:36 +02:00
leonarski_f	6e0bb971ac	v1.0.0-rc.143 (#53) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Avoid copying gain calibration together with DiffractionExperiment Reviewed-on: #53 1.0.0-rc.143	2026-04-30 22:16:50 +02:00
leonarski_f	87fde1b32e	v1.0.0-rc.142 (#52) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Support for newer CUDA architectures (notably Blackwell); minimum CUDA version 12.8 * Minor changes to jfjoch_process, jfjoch_fpga_test and jfjoch_lite_perf_test to make them more consistent Reviewed-on: #52 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 16:47:53 +02:00
leonarski_f	d760b12a18	v1.0.0-rc.141 (#51) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Azimuthal integration mapping is generated with parallel computations, significantly reducing setup times * frontend: Fix selection of FFTW in indexing settings Reviewed-on: #51 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-30 13:04:54 +02:00
leonarski_f	239a441ee6	v1.0.0-rc.140 (#50) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: For DECTRIS detectors, ZeroMQ link is persistent, to save time for establishing new connection * jfjoch_broker: Minor bug fixes for rare conditions Reviewed-on: #50	2026-04-29 21:40:22 +02:00
leonarski_f	4878318c27	v1.0.0-rc.139 (#49) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Further reduce startup time for DECTRIS detectors by selectively modifying SIMPLON parameters on `/start` * jfjoch_broker: Further reduce startup time for DECTRIS detectors by not setting beam center and detector distance via SIMPLON API on '/start' * jfjoch_broker: Add an extra message to ZeroMQ puller ready to monitor Lite worklow preparation time * jfjoch_broker: Image buffer configuration is postponed for Lite receiver flow till start message is received * jfjoch_broker: Use nanoseconds internally for frame/image/readout time * jfjoch_broker: Extra messages added for receiver operation (to be removed after debugging finished) * jfojch_broker: Improve profiling of different data analysis steps * jfjoch_broker: Record integration reflection count * jfjoch_broker: Fix bug where ZeroMQ preview frequency was confusing time units (micro vs. milliseconds) * jfjoch_broker: Fix bug where '/wait_till_done' got deadlocked * jfjoch_writer: Fix confusion between NaN and zero in floating-point datasets **Breaking changes**: detector definition is now using nanoseconds to define minimum frame time, minimum count time and readout time. Reviewed-on: #49 1.0.0-rc.139	2026-04-29 09:50:50 +02:00
leonarski_fandtakaba_k	230480e390	v1.0.0-rc.138 (#48) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Cleanup DECTRIS start-up code to enable a shorter start time jfjoch_broker: Allow for asynchronous start to allow overlapping detector configuration with other beamline preparations jfjoch_broker: Goniometer axis name is converted to lowercase jfjoch_broker: Fix bug, where wrong HTTP error codes were returned jfjoch_broker: Improve sigma estimation during merging (K. Takaba) --------- Co-authored-by: takaba_k <kiyofumi.takaba@psi.ch> Reviewed-on: #48 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.138	2026-04-27 19:56:14 +02:00
leonarski_f	c981e1b91c	v1.0.0-rc.137 (#46) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Better track time for each operation in the processing stack * jfjoch_broker: Rewrite preprocessing of diffraction images in the non-FPGA workflow to better use GPUs (work in progress) * jfjoch_broker: Remove ROI calculation in the non-FPGA workflow (work in progress) * jfjoch_viewer: Toolbar displays image number starting from 1 (instead of 0) Reviewed-on: #46 1.0.0-rc.137	2026-04-25 19:59:21 +02:00
leonarski_f	c1c170112c	v1.0.0-rc.136 (#45) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Improve logic regarding indexing architecture and thread pools (work in progress). Reviewed-on: #45 1.0.0-rc.136	2026-04-20 11:54:33 +02:00
leonarski_f	bb9f5c715f	v1.0.0-rc.135 (#44) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4) * jfjoch_viewer: Improve image render performance Reviewed-on: #44 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.135	2026-04-16 11:59:59 +02:00
leonarski_f	4a852b4d6b	v1.0.0-rc.134 (#43) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long * jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata * XDS plugin: Add generation of Jungfraujoch plugin for XDS * CI: Add tests with XDS and DIALS (`xia2.ssx`) Reviewed-on: #43 1.0.0-rc.134	2026-04-09 13:30:47 +02:00
leonarski_f	81bd9a06a1	CI pipeline upgrade (#42) ... Updates to CI pipeline * New docker image for Ubuntu 22.04 with CMake 3.26 * New docker image for Rocky 9 with DIALS 3.27 * New automated test to check for DIALS processing with xia2.ssx Reviewed-on: #42	2026-03-28 20:06:23 +01:00
leonarski_f	6133da1377	v1.0.0-rc.133 ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * jfjoch_broker: Use httplib for HTTP server instead of Pistache * jfjoch_broker: Drop OpenSSL support * jfjoch_broker: Base work for multi-lattice support in the future * Update dependencies to more recent versions (spdlog, HDF5, Catch2, httplib) Reviewed-on: #41 1.0.0-rc.133	2026-03-26 20:50:33 +01:00
leonarski_f	2ab9deb20b	v1.0.0-rc.123 (#40) ... Documentation update Reviewed-on: #40 1.0.0-rc.132	2026-03-12 12:17:11 +01:00
leonarski_f	166fcdb68f	v1.0.0-rc.131 (#39) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Fix bug in saving JUNGFRAU calibration (pedestal/pedestalRMS) * jfjoch_viewer: Fix calibration (pedestal) images being open flipped * jfjoch_process: Add space group detection (EXPERIMENTAL) Reviewed-on: #39 1.0.0-rc.131	2026-03-07 11:34:04 +01:00
leonarski_f	928789a67c	v1.0.0-rc.130 (#37) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Rotation indexer has two retries if failes * jfjoch_broker: Rotation indexer handles small number of rotation images (like test shot) * jfjoch_broker: Integration calculates background mask based on R2 radius * jfjoch_process: HDF5 files are not saved by default Reviewed-on: #37 1.0.0-rc.130	2026-03-06 14:38:56 +01:00
leonarski_f	64002f1e29	v1.0.0-rc.129 (#36) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files) * jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction * jfjoch_broker: Image pusher statistics are accessible via the REST interface * jfjoch_writer: Supports TCP image socket and for these auto-forking option Reviewed-on: #36 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.129	2026-03-05 22:13:12 +01:00
leonarski_f	3036b1d37f	v1.0.0-rc.128 (#35) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Handle properly reuse of image buffer locations * jfjoch_broker: Fix bug in counting idle slots * jfjoch_broker: Force obtuse angle for monoclinic cells * jfjoch_process: Change scaling refinement tolerance Reviewed-on: #35 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.128	2026-03-03 22:24:44 +01:00
leonarski_f	f3e0a15d26	v1.0.0-rc.127 (#34) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Default EIGER readout time is 20 microseconds * jfjoch_broker: Multiple improvements regarding performance * jfjoch_broker: Image buffer allows to track frames in preparation and sending * jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer * jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets * jfjoch_writer: Fixes regarding properly closing files in long data collections * jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code Reviewed-on: #34 1.0.0-rc.127	2026-03-02 15:57:12 +01:00
leonarski_f	32b681e591	v1.0.0-rc.126 (#33) ... This is an UNSTABLE release. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Fix bug for monoclinic space groups being wrongly refined when beta is much different from 90 deg. Reviewed-on: #33 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.126	2026-02-19 07:01:38 +01:00