1141 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	e45a1577d6	jfjoch_process: optional absorption surface for rot3d scaling (--absorption) ... Adds an opt-in smooth absorption correction for rotation scaling. After the rot3d fulls are scaled, --absorption[=num] fits a multiplicative surface A(s1_crystal) - a degree<=4 monomial basis (real spherical harmonics up to l=4, as XDS/DIALS) of the diffracted-beam direction in the crystal/goniometer frame, by ridge-regularized log-linear least-squares of I_scaled/I_ref weighted by (I/sigma)^2, over num iterations (default 3); the surface divides image_scale_corr and the fulls are re-merged. Off by default and a no-op without rot3d. On the test panel (~13 keV, thin crystals) it is metric-neutral - fitted rms(log A) ~3-4%, ISa/CC1/2 unchanged - because absorption is negligible there and the per-frame scale G(phi) already absorbs the angular part. It is kept as a lever for low-energy data (e.g. 6 keV) where absorption becomes significant. Stored as ScalingSettings::absorption_iter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 15:23:01 +02:00
leonarski_fandClaude Opus 4.8	77f1ed2566	jfjoch_process: default scale-fulls for rot3d; skip _process.h5 when merging ... Two offline-processing ergonomics changes. scale-fulls is now ON by default for -P rot3d (it refits the per-frame scale on the combined fulls and lifts ISa substantially, e.g. HEWL rot3d 7.0 -> 16.4). --scale-fulls stays as the explicit opt-in for non-rot3d order; new --no-scale-fulls disables it for rot3d. (scale_fulls is now an optional<bool> defaulting to combine_3d.) Note: on low-completeness data the Unity-reference refit can cost a little CC1/2 (endothiapepsin ~70% complete: -5% in a mid shell); pair with --reject-outliers for the full low-symmetry benefit. When merging (-M), the merged reflections (.mtz/.cif) are the wanted output, so the large per-image _process.h5 is no longer written by default - it routinely ran to hundreds of MB. Pass --write-process-h5 to also emit it. Without merging the _process.h5 is the only output and is always written. Implemented with a ProcessConfig.write_process_h5 flag gating the FileWriter; reflection and image.dat writing are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 14:50:18 +02:00
leonarski_fandClaude Opus 4.8	21dd99524f	rotation_indexer: map FFT primitive cell to the SG conventional setting ... When a space group is supplied without a reference cell, de-novo two-pass rotation indexing fed the FFT's Niggli-reduced primitive cell straight into XtalOptimizer as if it were the conventional cell. For non-primitive lattices (centered I/F/R/C, or hexagonal where the primitive pair sits at gamma=60) the conventional-system model then refined to a wrong minimum and indexed 0% of frames: cytC (P3121) gave 103.9/103.9/78 instead of 83.7/83.7/88.6, insulin (I213) 66.7 instead of 77.65, insulin-R3 51/51/36 instead of 81.4/81.4/33.3. Run LatticeSearch on the FFT primitive cell (it already yields the correct conventional cell + reindex for I/F/R/C). For the one remaining gap - a metrically hexagonal lattice that the geometry-keyed search lands on the ortho-hexagonal C setting - re-express the reduced primitive cell in conventional hexagonal axes (b -> b - a opens gamma 60 -> 120). De-novo "-S" now indexes cytC/insu/Ins_H/lyso/MyoB/EP/lyso_ref at 100% with the correct cell; the "-C -S" path is unchanged. The helper stays in this .cpp (g++) rather than CrystalLattice.h to avoid recompiling CUDA units, which is broken under the box's CUDA-13 nvcc; promote it to a method once that is fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-30 14:29:50 +02:00
leonarski_fandClaude Opus 4.8	3aec235f67	integration: add --reciprocal-profile dial (global reciprocal-space width) ... The per-shell profile width is learned in pixels, so it varies ~4x with resolution (mostly the geometric projection of a near-constant reciprocal-space relrod) and must be binned per shell -> it starves at high resolution / on sparse data. The new --reciprocal-profile flag instead learns ONE global width in reciprocal space, sigma2_q,tan = A + B|q| + C|q|^2: the Jacobian g_tan=cos(2theta) removes the geometric projection, and C|q|^2 is the crystal mosaicity relrod (variance ~(eta|q|)^2). Applied per reflection as sigma2_tan,px = (A + B|q| + C|q|^2)/g_tan^2 (B,C clamped >=0; quadratic->linear->constant fallback). Off by default. On the sharp HEWL test crystal (mosaicity 0.091deg, so C fits to ~0 and it reduces to the validated linear form) it is metric-neutral: ISa 16.2->16.3, anomalous 0.92x unchanged, CCref band 90.0->89.9, CC1/2 a touch lower (per-shell isn't starved at 23k spots/shell, and a global fit is less flexible). So: simpler + more transferable at a small CC1/2 cost, ISa/anomalous held. Its payoff is on MOSAIC crystals (large C|q|^2), where per-shell starves on the wide weak high-res spots and 6 shells are too coarse; both lyso test crystals are sharp, so it ships as a dial to try on mosaic data elsewhere. A separate radial relrod fit was tried and dropped (no gain). See NEXTGEN_INTEGRATOR.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 14:26:08 +02:00
leonarski_fandClaude Opus 4.8	5ba5fe8ad1	integration: centre the profile on the predicted sub-pixel position ... The profile was learned and applied on the integer pixel round(predicted), so a shared profile sits up to 0.5 px off the true spot (and stacking spots with random sub-pixel offsets broadens the learned profile). Build the Gaussian per reflection instead, centred on the predicted sub-pixel offset -- noise-free geometry, unlike the observed centroid, which hurt -- and elongated radially as before. HEWL rotation @1.0A: ISa 15.7->16.2, CCref band 89.9->90.0, CCxds 94.8->95.0 (high-res 1.00A CCref 66.0->66.9); sharp serial stills 1.68A CC1/2 61.6->62.5; anomalous S peak 0.92x XDS (no accuracy traded). De-broadening the learned width by the 1/12 px^2/axis integer-binning floor was tested and rejected (it over-narrows). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 07:34:06 +02:00
leonarski_fandClaude Opus 4.8	26901da42f	integration: de-contaminate the profile width, add a radial parallax ellipse ... The profile-fit width came from the full 13x13 second moment, which runs 3-8x wider than the true spot: neighbour reflections leak into the (unmasked) learning grid -- catastrophic at low res where spots crowd the beam -- and the far corners (lever arm dx^2+dy^2 up to 72) add rectified background noise. Splitting the spot moment into radial vs tangential shows the tangential width is isotropic (mosaicity/divergence) while the radial excess is pure sensor parallax ~tan^2(2th). - Measure the width over the signal disk r1 on the monochromatic path (inherently excludes neighbours, caps the radial tail); keep the generous full-grid width on the broadband/stills path (sparse spots, the centroid-undersampling floor). - Extend the (was bandwidth-only) radial ellipse with an analytic, material-aware parallax term c_par*tan^2(2theta), c_par = Var(z)/pixel^2 from sensor thickness + material + energy (parallax_var_px2; Si and CdTe), plus a fixed weak-spot capture term on the monochromatic path only. HEWL rotation @1.0A: ISa 13.5->15.7, CC1/2 1.12A 91.3->95.9 / 1.05A 83.0->85.2, external CCref band 88.1->89.9, CCxds 93.4->94.8, R-meas 9.4->8.7; low/mid flat. Sharp serial stills gain slightly from parallax; broadband stills neutral. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 00:54:59 +02:00
leonarski_fandClaude Opus 4.8	b51036ad30	broker: don't clobber sensor thickness/material with the API default ... JFJochBrokerParser set SensorThickness_um / SensorMaterial unconditionally from the request's Detector model, but that model defaults them to 320 um / Si with IsSet=false. So any start request that didn't explicitly carry the sensor overwrote the detector-reported value (DECTRIS SIMPLON read) or the detector-specific default with 320 um -- the "PILATUS4 ends up 320 um" symptom. Guard both with the IsSet flag, mirroring highVoltageV just above. The receiver -> FillMessage -> CBOR -> writer chain was already correct; the value was simply wrong at the source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 23:00:06 +02:00
leonarski_fandClaude Opus 4.8	6b989b4213	reader: read sensor thickness/material from the HDF5 metadata ... HDF5MetadataSource ignored /entry/instrument/detector/sensor_thickness and sensor_material and left the DetectorSetup default (320 um, Si). Read them when present (NXmx stores thickness in metres), so reprocessing honours the recorded sensor -- which the parallax/absorption model needs. (The acquisition path still records the 320 um default because DetDECTRIS sets no per-model thickness; that is a separate, acquisition-side fix.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 23:00:06 +02:00
leonarski_fandClaude Opus 4.8	7a4fadf29b	bragg_prediction: reciprocal Lorentz divides by |S||S0|, not S.S0 ... S.S0 = |S||S0| cos(2theta), so dividing the rocking rate |m2 . (S x S0)| by it added a spurious 1/cos(2theta) factor (1.0 at low res, 1.8x at 1 A) to the reciprocal Lorentz, hence to the absolute/Wilson scale. Divide by |S||S0| to get the correct zeta*sin(2theta) (times a constant 1/lambda^2 the overall scale absorbs). The spurious factor depends on 2theta only, so it cancels within a resolution shell and between symmetry mates -- CC1/2, CCref and R-meas are metric-neutral -- but it corrupts the absolute scale and destabilises the per-image B-factor and cross-resolution error model. CPU and GPU kernels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 23:00:06 +02:00
leonarski_fandClaude Opus 4.8	40f650b9e7	scale_merge: pool the per-frame background across each rocking event in Combine3D ... Each partial subtracted its own independently-estimated per-frame background, so a weak full assembled from N frames accumulated one background-estimation variance per frame. The true background is flat over the few frames of one rocking event, so replace each partial's background by the event mean and correct its intensity by n_bkg*(bkg - pooled), where n_bkg = (sigma^2 - I)/bkg is the effective background-pixel count -- correct for weak AND strong reflections (sigma^2/bkg would over-count strong ones and over-correct them). Single-frame events are a no-op. Measured (rotation lyso @1.0 A): 1.05 A CC1/2 81.3 -> 83.0, R-meas 81.4 -> 77.2, CCref +0.2, CCxds +0.3; overall R-meas 10.0 -> 9.4%; ISa preserved (13.5). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 22:41:37 +02:00
leonarski_fandClaude Opus 4.8	eccd10a0a7	integration: neighbour-mask the profile-fit background ring, widen r3 8->10 ... ProfileIntegrate2D::BoxSum now excludes every predicted reflection's r2 disk from the r2..r3 background annulus (mirroring BraggIntegrate2D), so a neighbour Bragg peak can no longer bias a reflection's background high and over-subtract. With neighbours excluded the annulus can safely widen, so the default r_3 goes 8 -> 10 (more background pixels, lower-variance estimate). Measured (rotation lyso @1.0 A, external CCref/CCxds vs XDS): 1.05 A CCref +1.3 / CCxds +1.3, R-meas -5 pts, low-res R-meas unchanged. Serial (Jet8 @0.0002 bandwidth): 1.68 A CC1/2 +3.9 / CCref +1.1, 1.58 A +4.0 / +1.6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 22:41:14 +02:00
leonarski_fandClaude Opus 4.8	bbd888dcc3	integration: gate the background-ring sigma-clip to stills (bandwidth>0) ... The 76e88b0f sigma-clip (reject ring pixels above mean+3*sqrt(mean)) de-biases bandwidth-streaked high-resolution stills, but it ran on all data. On rotation (no streaks) it clips legitimate high background pixels and biases the mean low, slightly inflating weak intensities and hurting the anomalous signal. Gate it to bandwidth>0 (stills), matching how the 814dff34 radial-profile change is already gated. Rotation lysozyme (self-scaled, smooth-g, -A): anomalous S-peak 0.84x -> 0.86x of XDS (SD_MET 11.71 -> 11.94, CL_CL 1.28x -> 1.25x), ISa unchanged. Stills (bandwidth set) are byte-identical (clip still applies). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8	99b923e727	scaling: smooth per-frame scale G across frames before the rot3d combine ... ScaleOnTheFly fits each frame's scale G independently with no neighbour coupling, so the few partials of one rocking event are weight-summed in combine3D on inconsistent scales - jitter that never enters the full's counting sigma and instead surfaces as scatter between symmetry mates, inflating the error-model b (low ISa). A centered moving average of log(G) over a small frame window (default 9, on for rot3d) removes it, mirroring XDS's smooth scaling. Complementary to --scale-fulls (which rescales between fulls, after the combine): smoothing fixes within-event scale, scale-fulls fixes between-full. On the rotation lysozyme set (1.4A, merged, with --scale-fulls): ISa 11.7 -> 15.0, R_meas 10.0% -> 8.3%, CCref stable, chi2 ~0.97 (honest). Anomalous (full-res): ANODE S-peak 0.61x -> 0.80x of XDS. --smooth-g[=window] tunes/disables it (=0 off); --mosaicity <deg> is a diagnostic that fixes the scaling mosaicity for sweeps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-29 19:02:47 +02:00
leonarski_fandClaude Opus 4.8	02477f1ce4	space group: determine centering from absences; jfjoch_process re-indexes from scratch ... Fixes wrong space-group assignment on cubic insulin (true I2_13, a=77.6). SearchSpaceGroup: drop the lattice-centering gate (and the now-unused lattice_centering option). The indexer often returns the conventional cell, whose geometry hides I/F/C centering - that information lives only in the systematic absences. Stage B now tests every centering of the point group and confirms it from the data, so an indexer-reported 'P' no longer excludes I2_13/I23. (I23 and I2_13 remain indistinguishable by absences and are reported as such.) jfjoch_process: discard any unit cell stored in the input HDF5 by default so the cell is re-determined from scratch. A stale/wrong stored cell otherwise resolves the indexing algorithm to FFBIDX, which trusts that cell and locks onto the wrong lattice (Ins_I_3 went from 2.7% -> 76% indexing). A user-supplied -C cell still applies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 16:39:03 +02:00
leonarski_fandClaude Opus 4.8	107bcae0f0	merge: de-bias the error-model variance fit (chi^2 was ~2.2, now ~1) and report chi^2 ... RefineErrorModel fits the variance model a*sigma^2 + (b*<I>)^2 to the binned *median* of the squared symmetry-mate deviations, chosen for robustness. But a single deviation squared over its variance is chi-square(1)-distributed, whose median is only 0.4549 of its mean, so the fit was calibrating the variances to 0.4549x their true value: the merged sigmas came out ~1.48x too small and the achieved reduced chi^2 was 1/0.4549 = 2.2, not 1. The error model was internally well-behaved (flat chi^2 across resolution) but globally over-confident, which inflated ISa (=1/b) by ~1.48x and made the exported sigmas too optimistic for downstream weighting / French-Wilson. Divide the binned median by the chi-square(1) median (0.4549) to recover an unbiased estimate of the mean E[dev^2]=sigma^2, keeping the robustness of the median while targeting reduced chi^2 = 1. Also compute the achieved median reduced chi^2 (same normalization) and report it on the "Error model" line so mis-calibration can no longer drift silently. Verified: HEWL rotation a 0.588->1.292, b 0.052->0.077, ISa 19.1->12.9, chi^2 2.17->1.06; serial Jet8 ISa 1.0->0.7, chi^2 0.92. Relative ISa comparisons and all CC1/2/CCref/anomalous metrics are unchanged (sigma-independent or a common constant); only the absolute sigma calibration is corrected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 15:57:42 +02:00
leonarski_f	a0856e1042	CI: Build Windows non-CUDA installer	2026-06-28 15:39:34 +02:00
leonarski_fandClaude Opus 4.8	a58f1fb046	docs: note background outlier-reject and bandwidth radial profile in integration ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 12:03:10 +02:00
leonarski_fandClaude Opus 4.8	814dff34cb	integration: radially-elongated profile for bandwidth-streaked high-res stills ... With a finite energy bandwidth each reflection is smeared RADIALLY by sigma_bw = bandwidth_sigma * R_px (R_px = distance from the beam centre, so large at high resolution): high-resolution spots become radial streaks. The isotropic per-shell Gaussian both mis-weights them and clips the streak tail on the fixed profile grid, losing intensity (biased low, noisy). When a bandwidth is set, fit each reflection with a per-reflection Gaussian elongated only along its radial direction - sigma^2_radial = sigma^2_intrinsic + sigma_bw^2, sigma^2_tangential = sigma^2_intrinsic - on a grid grown to hold the streak. Unlike an isotropic widening this adds no tangential background. It only engages where the smear exceeds the intrinsic spot (high resolution); low/mid resolution and monochromatic data (bandwidth 0, e.g. rotation) are untouched. On the HEWL serial-stills jet (with the background sigma-clip) this lifts the overall CC-vs-reference 52 -> 55% and the high-resolution I/sig (1.7 A 0.5 -> 1.4), recovering the 2.0-2.5 A band, with CC1/2 preserved (the per-shot noise the wider region adds is averaged out by the high serial multiplicity). Rotation ISa 19.1 unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 12:01:56 +02:00
leonarski_fandClaude Opus 4.8	76e88b0fca	integration: sigma-clip the background ring to de-bias high-resolution stills ... Weak high-resolution still reflections were systematically over-subtracted: a bandwidth-streaked high-res spot (or a neighbour) leaks into the r2-r3 background annulus and biases its mean high, so the subtracted background is too large and the merged high-resolution intensities go negative (seen as reproducibly negative <I/sig> at 100% completeness and high multiplicity past ~1.9 A). Add one high-outlier sigma-clip pass to the box-sum background (reject ring pixels above mean + 3*sqrt(mean), recompute) so the contamination no longer inflates it. A clean Poisson background is essentially unchanged (~0.1% exceed the cut). On the HEWL serial-stills jet this de-biases the high-res band - <I/sig> 2.03 A 0.9 -> 1.6, 1.79 A -0.1 -> +0.7 - extends the usable resolution ~2.2 -> ~2.0 A and improves overall R-meas 130 -> 124%, with CC1/2 and CC-vs-reference neutral. The rotation crystal is unchanged (ISa 19.1), its clean backgrounds being barely clipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 11:50:31 +02:00
leonarski_fandClaude Opus 4.8	7bb25979e6	docs: correct CPU_DATA_ANALYSIS integration section (profile-fit, not summation) ... The default 2D Bragg integrator is ProfileIntegrate2D (Kabsch profile fit with a per-resolution-shell Gaussian profile and de-biased variance), with box summation as the seed/fallback (--integrator boxsum|gaussian|empirical). Section 9 and the section 13 note both still claimed integration was summation-only with no profile fitting; rewrite them to describe the profile-fit default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 11:16:00 +02:00
leonarski_fandClaude Opus 4.8	59c474e4b0	docs: big-picture rotation 3D integration + intrinsic widths (CPU_DATA_ANALYSIS) ... Document the rot3d path that was missing: a new section on combining a reflection's per-frame partials into one full (de-biased weighted combine, captured fraction, capture-aware systematic uncertainty, XDS-order full re-scaling) so the merge sees counting statistics instead of rocking-curve slicing. Recast the profile-radius and mosaicity sections as what the system does - profile radius as the intrinsic (bandwidth-deconvolved) width, mosaicity by ML with a search window wide enough to capture the rocking tail and held fixed during scaling - rather than the optimisation narrative. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 10:56:49 +02:00
leonarski_fandClaude Opus 4.8	7b464e4b3c	indexing: deconvolve energy bandwidth from the profile radius ... The profile radius (intrinsic excitation-error width = mosaicity + divergence) was the plain RMS of dist_ewald over indexed spots. With a finite energy bandwidth that spread is broadened by the bandwidth's radial smear sigma_bw = bandwidth_sigma*lambda/(2 d^2), which prediction then re-applies per reflection - so bandwidth was counted twice and the radius was inflated (most at high resolution, sigma_bw ~ 1/d^2). Subtract the bandwidth variance from the measured spread so the radius is the intrinsic width. bandwidth = 0 (monochromatic / rotation) is unchanged. Small for narrow bandwidths (~6% of the variance, ~4% radius on the 1% jet); matters for wide-bandwidth / pink beam. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 10:56:49 +02:00
leonarski_fandClaude Opus 4.8	043ff0e864	docs: mosaicity MLE search-window fix + capture-aware rotation precision (CPU_DATA_ANALYSIS) ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 23:58:45 +02:00
leonarski_fandClaude Opus 4.8	82b8a9752c	rotation: fix mosaicity underestimate + default-on capture-uncertainty (ISa 10.7->19.1) ... The rotation per-image mosaicity was ~3x too small (0.045 vs the true 0.13deg), which crippled the partiality model and capped per-observation precision: it predicted reflections on too few frames and over-peaked the rocking partiality, so the rot3d-combined fulls were ~1.7x noisier than XDS's, the integration bottleneck behind the jungfraujoch-vs-XDS ISa gap. Two root causes, both fixed: - CalcMosaicityXDS (Kabsch-2010 MLE) searched each spot's exact-Bragg phi only within +-wedge (the 0.2deg oscillation). Reflections recorded at larger rocking offset - the tail that defines the mosaic width - fell outside and were dropped, truncating the tau distribution so the MLE underestimated ~2x. Widen the search window to wedge+0.8deg; the MLE then converges to the true 0.13deg (and is insensitive to widening further, since it weights by the recorded fraction). - ScaleOnTheFly then re-refined the mosaicity from the intensity residual, which is degenerate with the per-image scale G and collapses it toward its floor. Trust the (now correct) indexing mosaicity and keep it fixed during scaling. With the correct mosaicity, --capture-uncertainty (which down-weights the over-extrapolated under-captured fulls) now pays off strongly, so default it ON (1.0) for the rot3d combine; it stays off for non-rot3d. Together on the HEWL rotation crystal: ISa 10.7 -> 19.1, and anomalous peak height vs XDS goes from 52% to ~78% (CL_CL 1.92x -> 1.29x). This reaches XDS's own published-correction ceiling (DECAY+ABSORP+MODPIX ~= 19.6); the remaining gap to its quoted ISa 28 is the I->inf extrapolation. No effect on the stills path (rotation-only code). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 23:57:04 +02:00
leonarski_fandClaude Opus 4.8	ed98775ba3	rot3d: capture-aware sigma for under-captured fulls + expose per-image scale ... The rot3d combine over-extrapolates fulls reconstructed from only a fraction f of their rocking curve (min_partiality admits f as low as 0.02). Against XDS on HEWL these low-capture fulls are systematically biased HIGH (+15% at f=0.8 to +100% at f=0.3), and the bias - not random scatter - is the strong-reflection floor that hurts anomalous accuracy. --capture-uncertainty <coeff> (default 0 = off, baseline bit-identical) adds a systematic uncertainty ~coeff*(1-f)*I to each full's sigma, so the merge down-weights the over-extrapolated fulls and the error model treats their scatter as expected. Unlike outlier rejection (which trades accuracy for CC1/2), this fixes a real bias, so accuracy improves: at coeff=1.0 the anomalous peak height vs XDS rises CL_CL +16%, SD_MET/SG_CYS +5-6%, ISa 10.7->11.0. Rotation- only (no-op for stills, which never combine). Also expose the per-image scale offline: Combine3D now carries the first-pass per-image scale metadata (G, B, mosaicity, wedge, CC) forward instead of dropping it, and jfjoch_process -M writes <prefix>_image.dat from it (the offline self-scaling result was otherwise unobservable - process.h5's per-image arrays are only filled on the online path). This enabled the XDS DECAY comparison (jfjoch G tracks XDS, r=0.93). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 19:41:20 +02:00
leonarski_fandClaude Opus 4.8	f261adf42a	tools: add jfjoch_recompress (bitshuffle/LZ4 -> bitshuffle/zstd) ... New offline tool that re-compresses /entry/data/data of a _data_NNNNNN.h5 file from bitshuffle/LZ4 to the standard bitshuffle/Zstd HDF5 filter. Every other object (groups, datasets, attributes, the dataset's own attributes, dims/dtype/chunking/block size) is reproduced unchanged. It writes a fresh file - only /entry/data/data is re-encoded, every other object is H5Ocopy'd verbatim - which then atomically replaces the original via rename(). This needs no h5repack (the new file has no leftover space) and is crash-safe (the original is opened read-only until the rename). Frames are streamed one at a time through the registered bitshuffle filter (decompress LZ4, compress Zstd), so it is dtype-agnostic and never holds the whole dataset in memory. Output is read by the standard bitshuffle+zstd HDF5 plugin (verified against the hdf5plugin/DIALS libh5bshuf.so, which links libzstd and supports the zstd mode). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 15:24:30 +02:00
leonarski_fandClaude Opus 4.8	809f6f92a3	Regenerate OpenAPI clients for bszstd_rlehuf compression value ... update_version.sh regenerated the TypeScript client (types.gen.ts, zod.gen.ts) and Redoc docs to include the new "bszstd_rlehuf" compression enum value added to jfjoch_api.yaml; the package-lock version field follows VERSION. The C++ server model treats compression as a plain string, so it needed no change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 14:52:12 +02:00
leonarski_fandClaude Opus 4.8	ac75177c5a	Test: prove Huffman-literals Compressed_Block with Number_of_Sequences=0 is valid ... Builds a single Compressed_Block (Huffman-coded Literals_Section, empty Sequences_Section) and checks: the block type is Compressed, its trailing Number_of_Sequences byte is 0, and stock ZSTD_decompress reconstructs the literals exactly. This is the format guarantee from zstd_compression_format.md ("if Number_of_Sequences == 0 ... Block's decompressed content is defined solely by the Literals Section content"), locked into the test suite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 14:47:16 +02:00
leonarski_fandClaude Opus 4.8	7e7a73062c	Compression: add BSHUF_ZSTD_RLE_HUFF (RLE runs + Huffman literals) ... New CompressionAlgorithm that emits a standard Zstandard frame: zero/0xFF runs become RLE_Blocks (like BSHUF_ZSTD_RLE) and literal regions become Compressed_Blocks with per-block adaptive Huffman literals and no sequences (Number_of_Sequences=0). Short runs are absorbed into the literal stream; incompressible literals fall back to Raw_Blocks so the worst case stays within ZSTD_compressBound. The Huffman tree + bitstream are produced by zstd's own HUF_compress{1,4}X_repeat (the same calls ZSTD_compressLiterals uses); only the frame/block/literals-section framing is hand-written, with comments citing zstd_compression_format.md so it can be checked clause by clause. Output decodes with stock ZSTD_decompress, so no reader changes are needed (decode routes like BSHUF_ZSTD). On sparse diffraction this gives ~12% smaller files than bitshuffle/LZ4 at about the same end-to-end speed, sitting between LZ4 and full ZSTD; for maximum ratio use BSHUF_ZSTD. Robust on any input: tests round-trip pure zeros, Poisson(10), Mersenne-Twister noise (checked against the size bound), an extreme-sparsity mask, and a real lyso image through stock ZSTD_decompress. API: exposed as "bszstd_rlehuf"; regenerate the Python/TS clients (update_version.sh) to surface the new value there. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 14:41:46 +02:00
leonarski_fandClaude Opus 4.8	02514bb6b7	Compressor: throw on overflow instead of returning a negative size ... Compress() and FrameTransformation::CompressImage() returned int64_t with a negative value meaning "did not fit". That is a footgun: the negative result silently converts to a huge size_t if a caller forgets to check it. Return size_t and instead throw a named CompressionBufferTooSmallException (deriving from JFJochException, Compression category) when the output would not fit the destination buffer. The receiver catches it explicitly and drops just that frame, as before; the offline/GetCompressedImage path uses a worst-case buffer so it never throws. Add a test that a too-small destination throws and a worst-case buffer does not. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 08:17:21 +02:00
leonarski_fandClaude Opus 4.8	74584c23ac	Compressor: scale block size to a per-algorithm byte target ... Replace the fixed-element DefaultBlockSize with a byte target divided by elem_size to get the block element count, so the per-block working set (and thus cache behaviour) stays constant across pixel bit depths instead of halving from 8- to 16- to 32-bit. The target is per-algorithm, following the measured sweet spots on sparse data: LZ4 wants a small, cache-resident block for throughput (16 kB), ZSTD/RLE want a large block for ratio (128 kB). The gap is widest on extreme-sparsity inputs such as the uint32 pixel_mask, where large-block ZSTD reaches 100-1800x vs ~160x for LZ4. The block size is read back per-dataset from the bitshuffle stream header (block_size = header_bytes / elem_size) and the HDF5 filter params, so the decompressor and external readers (XDS/Neggia/Durin/CrystFEL) need no change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 07:56:08 +02:00
leonarski_fandClaude Opus 4.8	aadba5b343	Compressor: bump bitshuffle block size 4096 -> 16384 elements ... On sparse lyso frames the larger block improves compression ratio across all bshuf algorithms (16-bit data): ZSTD 8.58 -> 9.30, LZ4 7.38 -> 7.58, RLE 6.82 -> 6.90. 16384 captures most of the gain available from even larger blocks (ZSTD tops out ~9.55 at 65536) while staying close to the cache sweet spot: the cheap codecs (LZ4, RLE) peak in throughput once a block's working set fits L1d (~4096 elem here), so very large blocks trade real throughput for diminishing ratio - and that penalty is worse on the Xeon Gold/Platinum production hosts (smaller private L2, shared-L3 contention under many parallel compression threads). The block size is stored per-dataset in the bitshuffle HDF5 filter params, so existing readers (XDS/Neggia/Durin/CrystFEL) stay compatible. Move the per-block bitshuffle scratch off the inline member array onto a lazily-sized heap vector, like tmp_space, so the block size no longer bloats every stack-allocated compressor (incl. the transient ones in CBORStream2Serializer). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-26 20:59:55 +02:00
leonarski_fandClaude Opus 4.8	c332e45a54	Compressor: make Compress size-aware, drop frames that don't fit ... JFJochBitShuffleCompressor::Compress now takes a dest_size and returns a negative value when the compressed output would not fit, instead of writing past the destination buffer. The check is lazy: before each block it verifies the remaining space still covers that block's worst case (mirrored by the new MaxCompressedBlockSize helper, consistent with MaxCompressedSize so a dest sized to MaxCompressedSize never fails). On overflow the dest content is undefined - no rescue. The receiver uses this to compress directly into the writer buffer slot and drop just the oversized frame instead of pre-reserving the full worst-case image size next to the per-image CBOR metadata. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-26 20:10:52 +02:00
leonarski_f	5aad009cd7	CI: Build all on Windows (not only jfjoch_viewer target)	2026-06-26 18:27:33 +02:00
leonarski_f	62a7fb3ab6	Use vcvars64.bat for all build steps	2026-06-26 18:26:46 +02:00
leonarski_f	8e40822e5b	Build viewer on Windows	2026-06-26 18:26:46 +02:00
leonarski_fandClaude Opus 4.8	2398330e52	Drop oversized-metadata frames instead of aborting the collection ... A serial-crystallography run on a detector with a large converted geometry (JF17T16, modules tiled vertically + horizontally) aborted with "Array out of bounds (Not enough memory to save image)". An indexed still on such a detector predicts/integrates close to the kMaxReflections (10000) cap; at ~170 B per serialized Reflection that is ~1.7 MB of per-image CBOR metadata, which overflowed the fixed 1 MB the buffer slot reserved on top of the image. The serialization guard then threw and cancelled the whole run. - Raise the per-image metadata headroom from 1 MB to 4 MB (GetImageBufferLocationSize). The worst case - 10000 reflections + 2000 spots (API max) + 65534 azimuthal bins - serializes to 2.78 MB, leaving margin while staying negligible next to the multi-MB image slot. - When metadata still does not fit, drop just that frame (log metadata/image/slot sizes + recycle the slot) instead of aborting, in both the FPGA and Lite receivers. - Add a regression test asserting the worst-case metadata fits the headroom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 17:11:41 +02:00
leonarski_fandClaude Opus 4.8	786af96b3b	SearchSpaceGroup: POINTLESS-style rewrite, pipeline integration, twinning test ... Space-group search (image_analysis/scale_merge/SearchSpaceGroup): - Two-stage POINTLESS-style determination. Stage A scores each distinct rotation operator once (was once per candidate space group, ~34x faster on lysozyme: ~26s -> <1s) and picks the largest point group all of whose operators confirm. Stage B picks the maximal space group whose predicted absences are confirmed weak, fixing the prototype's default to the symmorphic group (it returned P422 instead of P4(3)2(1)2). Enantiomorphic / origin-ambiguous pairs (P4(1) vs P4(3), I222 vs I2(1)2(1)2(1)) are reported as indistinguishable. - Constrain candidates to subgroups of the lattice (metric) holohedry and weigh centering only P-vs-metric, fed from rotation indexing's LatticeSearch result. Integration / pipeline: - With no user-fixed space group, predict in P (IndexAndRefine) so the centering-absent reflections are integrated and the search can confirm/deny centering (catching pseudo-centering / a missed superstructure) instead of trusting the metric; a user-fixed group still rejects absences in integration. - JFJochProcess: scale+merge in P1 -> determine the space group -> set it and re-scale+merge in it (statistics then come out in the right symmetry) -> write it to /entry/sample/space_group_number (new EndMessage.space_group_number, preferred by NXmx::Sample). jfjoch_scale no longer searches; it consumes the file's space group (and no longer clobbers it with an empty -S). Twinning (new image_analysis/scale_merge/TwinningAnalysis): Padilla-Yeates L-test (<|L|>, <L^2>; acentric-only, positive intensities so L is bounded) plus a shell-normalised <I^2>/<I>^2 second moment and a twin-fraction estimate. Reported after the final merge in jfjoch_process and jfjoch_scale, and surfaced in the jfjoch_viewer merge-statistics window with a red outline when twinning is suspected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 16:11:28 +02:00
leonarski_fandClaude Opus 4.8	d2159bde92	jfjoch_viewer: Grey out "Analyze dataset" on a live HTTP connection ... Dataset re-processing reads a stored HDF5 file, so it is unavailable for the live HTTP stream. Disable the "Analyze dataset" hero button while a source is connected (with an explanatory tooltip) instead of letting the user click through to a "open a file first" dialog afterwards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 11:42:44 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65 1.0.0-rc.155	2026-06-25 22:01:48 +02:00
leonarski_f	6136f858af	v1.0.0-rc.154 (#64) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability) Reviewed-on: #64 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.154	2026-06-25 18:12:00 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63 1.0.0-rc.153	2026-06-23 20:29:49 +02:00
leonarski_f	c49bd2ac3b	v1.0.0-rc.152 (#62) ... * jfjoch_broker: Fix bounds for azimuthal integration for Q spacing (allow Q of 1e-5) * jfjoch_viewer: Adjust Q bounds for azimuthal integration * jfjoch_azint: Add tool to do quick azimuthal integration Reviewed-on: #62 1.0.0-rc.152	2026-06-17 20:36:24 +02:00
leonarski_f	ef52dac2ee	v1.0.0-rc.151 (#61) ... * jfjoch_broker: For PSI EIGER detector allow to disable individual half-modules by putting empty hostname Reviewed-on: #61 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch> 1.0.0-rc.151	2026-06-16 14:13:29 +02:00
leonarski_f	90e804acd7	v1.0.0-rc.150 (#60) ... * jfjoch_broker: When in FPGA workflow (with PSI detectors) azimuthal integration might be forced to CPU - this will require more computational power, but it enables more integration bins and reports standard deviation of each bin. * jfjoch_broker: Raise error if one is in FPGA flow and there are too many azimuthal integration bins. Reviewed-on: #60 1.0.0-rc.150	2026-06-15 20:24:15 +02:00
leonarski_f	ea575f790a	v1.0.0-rc.149 (#59) ... * XDS plugin: Fix HDF5 mutex to run on multiple processors Reviewed-on: #59 1.0.0-rc.149	2026-06-13 21:27:41 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58 1.0.0-rc.148	2026-06-08 08:30:35 +02:00
leonarski_f	75de40f52b	v1.0.0-rc.147 (#57) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_viewer: Add reciprocal space viewer * jfjoch_process: Two pass algorithm that does spot finding/indexing + integration of full dataset * jfjoch_process: Improve logic for rotation indexer, to make execution more deterministic (still work in progress) Reviewed-on: #57 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-06-02 11:49:24 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56 1.0.0-rc.146	2026-05-28 18:48:35 +02:00
leonarski_f	75f1c5f954	SHIM library improvements from the HDF Group	2026-05-08 11:39:51 +02:00