v1.0.0-rc.157 #67

Merged

leonarski_f merged 41 commits from 2607-state-machine into main 2026-07-11 07:19:12 +02:00

Conversation 0 Commits 41 Files Changed 267

+3761 -1883

41 Commits

Author	SHA1	Message	Date
leonarski_fandClaude Opus 4.8	d4544bd3c8	rugnux: trim phase/timing log noise, keep step results ... Cut the started-phase and partial-timing chatter from the rugnux processing log while keeping the genuine step results and the final performance report: - Rugnux.cpp: the scaling/merging phase() lambda now only drives the GUI progress bar (observer->OnPhase); drop the per-transition "[timing] <phase>: X.XX s" lines (in the CLI the observer is null, so those timings were the only thing it printed). Also drop the two "started-phase" lines "Scaling+merging (<pass>) via RotationScaleMerge" and "Running scaling (<pass>) ..." - RotationScaleMerge already logs its own error model / combine / cutoff / merge-complete results, and the stills path still logs its error model and merge-complete. - rugnux_cli.cpp: drop the duplicate "Starting analysis of N images..." (Rugnux::Run already logs the richer "Processing N images ..." line) and the duplicate --max-spots log (kept where it is applied). Error-model params, adopted space group, lattice-search scores, ice-ring flag/mask results, auto resolution cutoff, and the final "Processed N images in X s (Y Hz)" report are all unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 22:24:35 +02:00
leonarski_fandClaude Opus 4.8	ef103d2ea2	rugnux: raise default min-captured-fraction to 0.7 (recover weak F-cubic space groups) ... The de-novo space-group search runs its point-group correlation only on reflections with I/sigma >= 3, but the Diederichs error model floors merged I/sigma at the asymptotic ISa = 1/b. On a weakly-diffracting crystal a few edge-of-sweep truncated fulls (rocking curve captured far below 1) inflate the error-model b term, dropping ISa below 3 so no reflection clears the gate: every operator gets zero pairs and the point group collapses to P1 (or a low subgroup). The chi^2 self-consistency guard is likewise corrupted by those zingers. Raising the rotation default of --min-captured-fraction from 0.5 to 0.7 drops those partiality-extrapolated fulls, lifting ISa back above the gate and cleaning the intensity statistics, so the search recovers the true symmetry. On the F-cubic Benas crystals: Benas_3 P4 -> F432 (CC1/2 78% -> 98%) and Benas_7 P1 -> P6122 (CC1/2 40% -> 99%). Benas_7's data supports hexagonal P6122, not cubic F432 - the cubic 3-fold/4-folds fail the merge self-consistency test - matching XDS. Full /data/rotation_test battery: every other crystal keeps its exact space group; the only cost is a little multiplicity from the extra dropped edge-fulls on a few weak/ narrow-sweep crystals (worst completeness -7% on EP_cs_01-24, whose CC1/2 nonetheless rises), while several crystals' CC1/2 improve. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 22:11:30 +02:00
leonarski_fandClaude Opus 4.8	1ea3846957	docs: CHANGELOG entries for weak-data indexing and min-captured-fraction ... Document the rc.157 changes: the Benas low-resolution rotation indexing fixes (gate diffracting-range denominator, multi-scheme first pass, space-group-search resolution guard) and the --min-captured-fraction / honest-R-meas rotation scaling change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 20:40:01 +02:00
leonarski_fandClaude Opus 4.8	7b8e08cd96	rugnux: index weakly-diffracting low-resolution rotation data (Benas F-cubic) ... Two F432 crystals (a~70.2 A) that diffract only to ~4-4.5 A on a detector reaching ~1.5 A failed to process de-novo. Three independent, physically motivated fixes, validated on the 24-crystal rotation_test battery with zero regressions (every other crystal keeps its cell, indexing rate and space group): 1. Per-frame indexing gate (AnalyzeIndexing): the indexed-fraction denominator counted every found spot, but for these crystals ~85% of spots are unindexable sub-diffraction ice/noise, making the 20% floor unreachable so every frame was rejected (0% indexed). Count only spots within the resolution range the lattice actually diffracts to (out to the highest-resolution indexed spot). This can only shrink the denominator, so it never rejects a frame that passes today. Fixes Benas_3 (0 -> 40%, correct F-cubic -> H32). 2. First-pass rotation indexing now runs multiple sampling schemes and selects the one that indexes the most frames on the real per-image path (spread over the whole rotation vs a consecutive wedge from the clean start). The long-standing spread scheme wins for typical crystals (unchanged), while the consecutive wedge keeps a reflection's rocking curve continuous across frames so the FFT can trace a long axis whose fine reciprocal spacing the coarse spread cannot resolve. Fixes Benas_7 (garbage cell 0.06% -> correct hex 69%). Scheme selection reuses ProcessImage's indexing path (extracted as DetermineRefineAnalyze / IndexFrameOnly) so validation cannot diverge from production. 3. Space-group-search resolution gate (Rugnux): the <I/sigma> >= 1 cut collapsed to ~the cell dimension when every shell is below the threshold (I/sigma floored by the error model though CC1/2 is high), leaving the re-merge with no reflections -> "resolution calculation failed" crash. Only cut when a good low-res region exists above the cutoff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 20:30:32 +02:00
leonarski_fandClaude Opus 4.8	f0799f780b	rugnux: drop edge-of-sweep truncated fulls + honest R-meas over accepted obs ... From a fresh decomposition of the low-resolution-shell R-meas gap vs XDS on the reference lysozyme rotation crystal (7.6% vs 2.9%), two rot3d scale/merge changes: - New --min-captured-fraction (default 0.5 for rotation, off otherwise): drop a combined full whose rocking curve was captured below this fraction. These edge-of-sweep, partiality-extrapolated fulls (sweep start/end) are biased and inflate the low-res R-meas. Dropping them lowers low-shell R-meas 7.6->5.5% and *improves* external accuracy (CC vs the F-model and vs an independent dataset, and the ANODE anomalous S-peak 0.93->0.94x XDS) - unlike --min-partiality, which lowers R-meas cosmetically while degrading accuracy. Threaded through ScalingSettings and both the GPU and CPU combine. - R-meas reporting fix: the R_meas re-walk counted observations the merge had rejected as outliers (XDS excludes its misfits before its R-factor). The accumulate pass now flags each rejected full and the R_meas pass skips it, in both the GPU and CPU paths. Merged intensities / ISa / CC1/2 unchanged (verified byte-identical external accuracy); low-shell R-meas 7.6->6.5%. No-op when --reject-outliers 0. Combined default rotation low-shell R-meas 7.6->5.1%, ISa 17.2->17.5, external accuracy equal-or-better than baseline and XDS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-10 20:13:06 +02:00
leonarski_fandClaude Opus 4.8	403e6e7a68	docs: bring the rc.157 CHANGELOG up to date with the branch ... Cover the de-novo symmetry / geometry / mmCIF rugnux work, the Bragg integration and preprocessing fixes, writer overwrite/reset handling, the jfjoch_viewer preview improvements, the web Frontend redesign + mask/pedestal viewers, and the installer naming — kept to one brief line each. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 15:43:34 +02:00
leonarski_fandClaude Opus 4.8	1b6831a2d3	rugnux: gate the P1 space-group-search scaling at <I/sigma> >= 1 ... The de-novo space-group determination (point-group intensity correlation plus the second-moment / twinning test) ran on the full resolution range, including noise-dominated high-res shells. There a single artifact - a zinger the box sum integrates whole, or a weak-reflection profile-fit runaway - has an astronomical resolution-normalised E (I / <I>_shell with <I>_shell ~ 0) and wrecks the normalised statistics, so the search collapses to P1. Re-scale the P1 search pass with a <I/sigma> >= 1 cut over thin resolution shells (POINTLESS-style: determine symmetry where there is signal). Only this pass is cut; the final in-symmetry merge keeps the full range, so no output resolution is lost. A manual --scaling-high-resolution, if coarser, wins. Battery of 24 rotation crystals: box-sum lysoC_14 P1 -> P41212; and with the default gaussian integrator three previously mis-assigned crystals now get the correct space group (cytC_10 155->152, EP_cs_01-24 5->4, Ins_I_3 196->197), with no space-group regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 15:20:10 +02:00
leonarski_fandClaude Opus 4.8	f6eca6c69e	bragg_integration: quantization-noise variance floor + summation fallback ... The Kabsch profile-fit weights floored the per-pixel variance at 1.0, far above the real detector noise floor (electronic noise is well below 1 for both EIGER and JUNGFRAU). The true floor is the quantization noise from rounding the charge-spread deposited energy to an integer: Var = 1/12. The 1.0 floor silently over-regularized — it inflated weak-reflection sigma ~6x at high resolution and pinned the scaling error model's `a` term at its 0.25 floor, suppressing ISa. Use PIXEL_VARIANCE_FLOOR = 1/12, and guard against the weak-reflection runaways it exposes: fall back to the box-sum (summation) intensity when the profile fit diverges from the summation seed by more than PROFILE_SUMMATION_MAX_NSIGMA box-sum sigmas. Applied identically in the CPU and GPU engines (GPU-vs-CPU equivalence test passes). Battery of 24 rotation crystals: broad ISa gains (lysoC_14 7.8->10.4, plus cytC/Ins/Myo), EcwtAL500 completeness 51->92% and CC1/2 35->79%, pding4_001 space group now correct; no space-group regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 14:30:15 +02:00
leonarski_fandClaude Opus 4.8	04bce94f4e	frontend: visualize mask + pedestal TIFFs in-browser ... Add an in-browser viewer for the broker's TIFF endpoints so the pixel mask and JUNGFRAU pedestal can be inspected without downloading files. - tiffDecode.ts: dependency-free decoder for the broker's baseline TIFFs (uncompressed, single-strip, single-channel; 8/16/32/64-bit uint/int/float). - TiffImageViewer.tsx: manual Load -> fetch -> decode -> canvas, with react-zoom-pan-pinch zoom/pan and a rect-based (zoom-agnostic) hover readout. `value` variant = colormap + editable min/max bounds + Auto + colorbar; `mask` variant = discrete fixed colours per bit-flag value with a decoded-flag legend (mask bits mirror common/PixelMask.h). - MaskVisualization.tsx: full-width panel on the Masks page (full/user mask toggle) fetching /config/mask.tiff and /config/user_mask.tiff. - Calibration.tsx: rebuilt with the shared card header; adds a pedestal viewer (gain-level + storage-cell selectors -> /preview/pedestal.tiff) above the per-module statistics grid, and degrades gracefully with no data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 14:17:57 +02:00
leonarski_fandClaude Opus 4.8	314caacee7	frontend: reflow ROI panel into readable rows ... The three ROI tables were crammed side-by-side; the 8-column Azimuthal one was unreadable at a third width. Put Box + Radial on the first row and the wide Azimuthal table full-width on its own row, then the buttons, under the shared titled card header. Trim the Box/Radial name columns so every column (including the delete action) fits at half width. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 14:17:57 +02:00
leonarski_fandClaude Opus 4.8	abf7d5f45d	frontend: visual polish pass (status pill, buttons, panels, empty states) ... Design pass across the frontend for a coherent, stronger-but-limited look (lime / coral / blue), keeping lime as the brand accent. - StatusBar: broker state is a fixed-width pill (lime=Idle, coral=Busy/ Pedestal/Error, light-blue=Inactive); while Measuring it fills like a progress bar (lime over the sidebar light-blue). Action buttons are a three-tier system (hot lime / active blue-on-blue / off ghost) so exactly one button reads as the thing to click now, ordered Cancel/Pedestal/ Initialize. Cancel is the emergency stop (hot while Measuring/Pedestal). - SettingsPanel: left-aligned card header on a light-blue band with a lime left accent (echoes the sidebar); inline <b> section labels restyled as uniform uppercase captions. DetectorSettings adopts the same header. - EmptyState: muted icon + text for "No plots"/"No preview". - DetectorStatus: semantic colour on state/power values. - Snackbars offset so action messages clear the fixed footer. - Footer PSI logo now uses the positive (dark) logos directly on the light footer instead of CSS-inverting the negative ones. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 14:17:57 +02:00
leonarski_fandClaude Opus 4.8	b474264a3f	frontend: PSI logo in footer + wider, shorter settings pages ... Add PSI branding and use the horizontal space on the expert pages, which previously stacked panels into one tall narrow column. - Footer carries the official PSI logo (one of four variants picked at page load), standalone with clear space, linking to psi.ch. Top bar is a clean "Jungfraujoch" wordmark. - PanelColumns masonry lays the processing/output/streaming/masks panels into responsive columns; ROI stays full width. - DetectorSettings regrouped from a fixed-height single column into a responsive two-column form (layout only; controls unchanged). - Softer corners (theme borderRadius 4 -> 8). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 14:17:57 +02:00
leonarski_fandClaude Opus 4.8	fd7e2ae6f9	image_preprocessing: treat the type-max marker as invalid for unsigned pixels ... The error/invalid pixel marker (the pixel type's extreme value, e.g. 0xFFFFFFFF for EIGER uint32) was only mapped to the internal error value for SIGNED types - the check was gated on std::is_signed<T>. For unsigned EIGER data a 0xFFFFFFFF pixel therefore fell through to the saturation test and, exceeding the (HDF5) saturation_value, was silently classified as saturated rather than invalid. Both are excluded from integration, so results are unchanged where such pixels are also in the static pixel mask (as in the JUNGFRAU test data - Thau_9 and lyso are bit-for-bit the same), but the classification/statistics were wrong and a stray 0xFFFFFFFF outside the static mask would be mis-accounted. Test the error marker before saturation and drop the is_signed gate in both the CPU and GPU preprocessors: the HDF5 saturation_value stays the saturation threshold, the pixel type's maximum is the invalid marker (priority masked > error > saturated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 14:17:56 +02:00
leonarski_fandClaude Opus 4.8	f0dc61ef58	rugnux: write refined geometry (beam, tilt, axis) to _process.h5 ... The master-file geometry is written from the StartMessage at file open, before indexing/refinement - the file is streamed per image, so it cannot wait for the refined values. rugnux therefore stored the nominal beam centre / detector tilt / rotation axis, not the geometry its own refinement found (jfjoch_broker, by contrast, deliberately keeps the user-provided values). Add optional refined geometry to EndMessage; at finalization the writer overwrites the beam_center_x/y, PONI rot1/2/3 and rotation-axis datasets in place with these. The broker leaves them empty, so its output is unchanged. For EcwtCQ244-E3 the _process.h5 now reports the refined beam 761.6/847.7 (nominal 764/850, matching XDS's 760.2) plus the refined tilt and rotation axis. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 14:17:56 +02:00
leonarski_fandClaude Opus 4.8	7d240a120c	rugnux: recover a centred Bravais lattice from intensities (de-novo C2) ... When de-novo rotation indexing demotes a pseudo-symmetric metric to a primitive/triclinic cell, the space-group search never tested the candidate symmetry operators - its operator set is bounded by the indexer's lattice, whose holohedry is trivial for P1 - so a genuinely centred lattice (C-centred monoclinic, EcwtCQ244) was locked to P1. A merely pseudo-symmetric triclinic cell (EcwtCQ066) is metrically indistinguishable from a real one (same 2x C-lattice, same absences), so the decision cannot be made from spot positions: the 2-fold is an intensity property and must be decided after P1 integration, as XDS does in CORRECT. After the P1 space-group search, if the metric is pseudo-symmetric, reindex the merged reflections to the candidate's conventional setting and re-run the search there. The I(h) vs I(Rh) correlation confirms a real centred lattice and rejects a pseudo-symmetric one. The centring itself cannot be read from absences here (integrated in the primitive cell, its centring-absent reflections do not exist), so among the point-group-equivalent candidates the intensities cannot separate, the one matching the metric candidate's centring is chosen. On confirmation the integrated reflections, lattice and cell are reindexed, the rotation merge is re-ingested, and the data are re-merged in the adopted space group. Scaling-only - the indexer is untouched. EcwtCQ244-B1-1 and -E3 -> C2 (2-fold CC 0.99); EcwtCQ066 stays P1 (CC 0.31); the rest of the 24-crystal de-novo battery is bit-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 14:17:56 +02:00
leonarski_fandClaude Opus 4.8	12efbc1beb	rugnux: emit non-standard mmCIF items under a "jfjoch" reserved prefix, not pdbx_ ... ISa and the twinning indicators (L-test means, second moment of I) have no standard mmCIF item. They were written as _reflns.pdbx_* names, borrowing the wwPDB PDBx/mmCIF dictionary's reserved prefix and so masquerading as standard items the dictionary does not actually define. Move them to the IUCr-sanctioned local-data-name extension: a private reserved prefix on the attribute of the existing _reflns category (_reflns.jfjoch_*), matching the DDL2 pattern the spec gives for private items. Write-only tags with no reader, so this is a self-contained, back-compatible rename. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-10 13:01:05 +02:00
leonarski_fandClaude Opus 4.8	09c1a04786	rugnux: measure total processing time incl. scaling, not just the per-image pass ... The reported "Processing time" / "Frame rate" stopped the clock right after the per-image analysis loop, excluding scaling, merging, resolution cutoff, twinning and file writing. Move the timing to the end of Rugnux::Run() so processing_time_s (and the derived frame_rate_hz / throughput_MBs) span the whole run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-10 12:27:50 +02:00
leonarski_fandClaude Opus 4.8	ca5ac9f838	Name the Windows NSIS installer jfjoch-viewer-*, not jfjoch-* ... Match the Linux viewer TGZ naming; update the Gitea upload glob to suit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-10 07:26:10 +02:00
leonarski_fandClaude Opus 4.8	a4cc908ff1	rugnux: run the L-test on rhombohedral data + write twinning stats to CIF ... The Padilla-Yeates L-test paired each reflection with a neighbour two steps along an axis ((h+2,k,l) etc.). A step of 2 preserves the reflection condition of P/I/C/F/A/B lattices but violates R-centring (-h+k+l = 0 mod 3, since 2 != 0 mod 3), so every neighbour was systematically absent -> zero pairs, and the L-test silently did not run on ANY rhombohedral crystal (twinning there was left to the weaker second-moment test alone). Add diagonal (1,1,0)/(1,1,3) fallback steps that preserve the mod-3 condition in both obverse and reverse settings, tried only when the axis steps find no present partner (P/I/C/F behaviour unchanged). On the twinned R3 insulin the L-test now runs: 25k pairs, <L^2> = 0.224 -> twinning detected. Also write the twinning indicators to the output mmCIF next to ISa: _reflns.pdbx_L_test_mean_abs_L / _L_test_mean_L_squared / _second_moment_I, as free-text pdbx values, guarded on pair/reflection count. Threaded the TwinningAnalysisResult into WriteReflections (both the full-pipeline and --scale callers). Note: on rhombohedral lattices the L-test reads ~0.03 lower than on P/I/C/F (its allowed step is slightly more local), so it separates the insulin twin pair only weakly (twinned 0.398 vs untwinned 0.427) and the second moment remains the cleaner discriminator there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-09 21:59:37 +02:00
leonarski_fandClaude Opus 4.8	8c288c8697	rugnux: recover genuine high symmetry in de-novo space-group search ... The POINTLESS-style symmetry search discarded a real point group on two hard rotation-test crystals: - Ins_I_3 (I23): one of the eight cubic 3-fold operators scored CC 0.483, just under the 0.5 per-operator floor, so I23 fell to I222 and merging collapsed (3-fold orientation ambiguity; CC1/2 1%, R-meas 558%). Gate A now scores each CONJUGACY CLASS of a point group's rotations by its mean CC - conjugate operators are symmetry-equivalent and must stand or fall together - so the cubic 3-fold class (mean 0.57) is confirmed. -> I23, CC1/2 99.9%. Degenerate (few-pair / NaN) class members are skipped rather than vetoing the class. - Benas_3/7 (F432): the chi^2 self-consistency gate rejected the true cubic group because the weak-data error model is badly miscalibrated (reduced chi^2 ~150), which inflates the chi^2 ratio with point-group order for genuine high symmetry too. The ratio bound is now widened by log10(chi2_ref) so a broken error model does not veto real symmetry, while a false operator whose ratio is far worse than its subgroup's is still caught. The bound stays a per-candidate chi^2 test - chi^2 still arbitrates and is never bypassed. Verified: F432 (chi2_ref 151, ratio 3.2 < bound 4.2) accepted; a calibrated pseudo/twin R32 (Ins_H_3: chi2_ref 2.6, ratio 4.2 > bound 2.4) still rejected, stays R3. SearchSpaceGroup.{cpp,h} only. Validated on the 24-crystal rotation-test battery: exactly these two space groups change (both now match the reference); zero regressions - every pseudo-symmetry P1 case and every correct assignment unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-09 21:14:18 +02:00
leonarski_fandClaude Opus 4.8	4f3153d6c7	rugnux: automatic CC1/2-logistic high-resolution cutoff ... Merge/report/write to the CC1/2 fall-off instead of the detector edge. After the final merge, fit a logistic to a fine-binned CC1/2 profile and cut one shell past cc_target (default 0.30) - generous, DIALS-style. Only the written MTZ/CIF and the reported shell table are trimmed; scaling, the 3D combine and the error model still run over the full range, and the per-image _process.h5 keeps full data for re-scaling. The de-novo P1 merge that feeds the space-group search is never cut (gated on the final merge). - ScalingSettings: resolution_cutoff {Off,CCHalfLogistic}, resolution_cc_target, report_shell_count (report shell count is now configurable). - New ResolutionCutoff.{h,cpp}: ComputeCCHalfLogisticCutoff (fine-bin CC1/2 + grid/refine logistic fit + one-shell extension + degeneracy/low-res guards). - Wired into RotationScaleMerge (rotation) and MergeOnTheFly/Rugnux (stills), plus the rugnux --scale path. - CLI: --resolution-cutoff cc-logistic|off, --resolution-cc-target, --resolution-shells. --scaling-high-resolution stays the manual override (wins and disables the auto-cut). Validated on the 24-crystal battery: no crashes (incl. Benas/cytC_10/Thau_9); --resolution-cutoff off is bit-identical to baseline; cutoffs reproduce the design table within ~0.05 A (lysoC 1.30, Ins_I 1.43, cytC_2 1.68, EcwtCQ244-E3 1.54); good-to-edge datasets (lyso_ref, EP_cs_02-10) correctly keep the full range. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-09 17:34:49 +02:00
leonarski_f	4df811bf73	Remove rugnux work-in-progress documents from Git	2026-07-09 16:18:08 +02:00
leonarski_fandClaude Fable 5	be937d0163	docs: rugnux automatic resolution-cutoff design (CC1/2 logistic, one shell too far) ... CC1/2 fall-off fit as a logistic in 1/d^2, cut one shell past CC1/2~0.3 - generous for ML, robust to the broken error model (works on 4 A Benas where I/sigma cannot), and avoids defining 'the' resolution. Includes experiments vs XDS and an implementation plan. Not yet implemented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 15:44:40 +02:00
leonarski_fandClaude Fable 5	c0f6522778	docs: rugnux de-novo symmetry/indexing investigation notes ... Records the C2-missed-as-P1 root cause + the (stashed) RotationOnly recovery fix and why it is a re-orientation step, the Benas F432 point-group/error-model finding, and the CC1/2-logistic resolution-cutoff design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 15:42:28 +02:00
leonarski_fandClaude Fable 5	ad73b4af99	writer: detect overwrite conflict at start for back-channel transports ... Previously an existing output file was only detected at the final rename, after the whole collection had run. Add an up-front check in the FileWriter constructor that fails before acquisition arms. Only the master file is checked, and only by the single writer that owns it (write_master_file). In a multi-writer TCP/ZMQ setup the per-image data files are staggered across writers by file number, and a writer does not even know the total writer count, so it cannot enumerate its own subset; checking all data files in every writer would make each writer stat files it never writes and race the writers already creating them. Data-file conflicts stay caught by their owning writer at the final rename. Gate the check on whether the transport can report the failure to the broker: the direct HDF5 pusher throws in-process and the TCP writer returns a START-failure ACK, so both abort the collection cleanly (verified: an asymmetric rejection cancels the already-started sibling). The ZeroMQ path is fire-and-forget with no back-channel, so StreamWriter opts out via image_puller.SupportsAck() and keeps writing .tmp files, failing only at the final rename (leaving the images on disk). Documented in docs/JFJOCH_WRITER.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 15:34:37 +02:00
leonarski_fandClaude Fable 5	36be60774f	image_pusher: reset writer when EndDataCollection finalization fails ... HDF5FilePusher::EndDataCollection released the FileWriter only after writer->WriteHDF5()/Finalize(), both of which throw at rename time when a target file already exists and overwrite is off. The throw skipped writer.reset(), leaving 'writer' non-null so the next StartDataCollection died with "Image pusher is already writing images". Tear the writer down on every exit path (try/catch + rethrow) so a failed collection no longer blocks the next one. The error still propagates and the orphaned temp files are intentionally left on disk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 14:43:42 +02:00
leonarski_fandClaude Fable 5	ec05a6c4d6	Viewer: coral secondary-lattice spots, ring arcs, legend, ROI swatches, save JPEG ... - Colour indexed spots from a secondary (or further) lattice in coral (#FA7268), the viewer's finishing accent, in the diffraction-image overlay. - Draw resolution rings by sweeping ResPhiToPxl into a QPainterPath (like the azimuthal ROI arcs), replacing the 4-point bounding-box ellipse and the poni_rot_3=0 hack, so rings follow the geometry on tilted detectors. - Add a spot/prediction colour legend to the Image-features side panel; the two user-adjustable swatches track the colour picker. - Show a colour swatch matching the on-image annotation on each ROI-list item, via a shared ROIAnnotationColor palette used by both the overlay and the list. - Add "Save image as JPEG..." / "...with overlay as JPEG..." to the image right-click menu, reusing a factored renderToImage() helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 12:04:12 +02:00
leonarski_fandClaude Fable 5	a4559e576d	Preview: geometry-following arcs, true beam center, unified colours, predictions ... Draw the resolution ring and azimuthal-ROI outlines by sweeping ResPhiToPxl through the geometry (a shared DrawArc/DrawThickLine helper), so they trace the true conic on a tilted detector instead of PONI-centred circles. Draw the beam crosshair at GetDirectBeam_pxl() (the real beam/detector intersection) rather than the stored PONI. Unify the spot overlay colours with the viewer: green = not indexed, magenta = indexed (primary lattice), cyan = ice ring, coral = secondary/further lattice. Add Bragg-prediction overlay (dark-red circles, skipping systematic absences) gated on a new show_predictions preview setting, wired through the OpenAPI spec, the image.jpeg HTTP handler, and the frontend toggle (+regenerated TS client). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 12:04:00 +02:00
leonarski_fandClaude Fable 5	03dad9fe93	image_analysis: remove dead RotationParameters; receiver opts out of retain_outcomes ... RotationParameters was a moving-average store (profile_radius, beam_x, beam_y, mosaicity) held by IndexAndRefine but never read or written - none of its four accessors had any call site. Remove the class, its member, the include, and the CMake entries. Also set retain_outcomes=false for the online receiver's IndexAndRefine: online scaling/merge at the end of a run is not enabled yet, so the whole-run integration_outcome is unused there. Revert to true when online scaling lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 11:28:16 +02:00
leonarski_fandClaude Fable 5	b2f0af12db	image_analysis: add retain_outcomes flag to IndexAndRefine; viewer opts out ... IndexAndRefine::integration_outcome is a whole-run vector (resize(GetImageNum())) holding up to ~0.8 MB of reflections per indexed image. It is consumed only by the offline scaling/merge pass in rugnux (ScaleAllImages / GetIntegrationOutcome). In the viewer's interactive/live analysis it is written but never read - dead storage that, with auto-reanalyze on during live follow, grew ~0.8 MB per frame for the whole run. Add a retain_outcomes constructor flag (default true). When false, skip the whole-run resize and the per-image store; the current image's reflections are still returned via the outgoing message. The viewer's live IndexAndRefine passes false. rugnux (including GUI processing jobs, which build their own Rugnux -> IndexAndRefine) and the online receiver keep the default true, so scaling/merge still has the accumulated data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 11:02:10 +02:00
leonarski_fandClaude Fable 5	dc8a80f3e0	viewer: fix HTTP live-follow OOM via datasetLoaded backpressure + shared pixel mask ... The viewer could grow to ~100 GB RAM when live-following an HTTP broker. The rc.153 images_in_flight backpressure only throttled imageLoaded; the heavy per-frame payload rides datasetLoaded, fanned out over ~10 queued cross-thread connections with no cap. In HTTPSyncDataset follow mode (entered when an operator clicks an image while following live) RefreshDatasetOnly_i emits a fresh full dataset every autoload tick with no imageLoaded, so the gate never engaged and the queued events - each pinning a full JFJochReaderDataset (full-detector PixelMask + per-image plots) - accumulated without bound. Backpressure datasetLoaded the same way as imageLoaded: a datasets_in_flight counter (cap 2), all emits routed through EmitDatasetLoaded_i, and AutoLoadTimerExpired gated on it (covers HTTPSyncDataset). The window routes the worker's datasetLoaded through a single OnDatasetReady sink that fans out synchronously via datasetReady and acks with datasetConsumed. Under load stale datasets are dropped; the next tick sends the latest. Share the pixel mask instead of deep-copying it: JFJochReaderDataset::pixel_mask is now shared_ptr<const PixelMask>, so per-frame dataset copies share the ~72 MB mask. UpdateUserMask does copy-on-write; JFJochHttpReader caches the mask by arm_date so a live refresh reuses one shared mask per acquisition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 10:58:10 +02:00
leonarski_fandClaude Fable 5	b4a29ebb32	Update grid scan_result test for dropped placeholder rows ... Commit 023a30cb0 intentionally filters uncollected placeholder rows (number = -1) out of ScanResultGenerator::GetResult() but left this test asserting the old full-grid vector size. The 3x3 grid collects only 7 of 9 cells, so scan_result now has 7 rows; the two uncollected placeholders sit at grid indices 5 and 8, after the checked positions, so the number assertions still hold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-09 09:44:21 +02:00
leonarski_fandClaude Fable 5	c6af375dc0	CHANGELOG: move branch changes from rc.156 to a new rc.157 section ... The rugnux consolidation bullet had been attributed to rc.156, which is the release this branch is based on. Restore rc.156 to match main and collect all of this branch's changes under a new rc.157 entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-08 20:43:44 +02:00
leonarski_f	1b5ffb4337	VERSION: 1.0.0-rc.157	2026-07-08 20:29:08 +02:00
leonarski_fandClaude Opus 4.8	023a30cb08	Drop uncollected placeholder rows from scan_result ... ScanResultGenerator pre-sizes its vector to the full expected image count, so GetResult() returned placeholder entries (number = -1, no angle) for images not yet collected. These leaked to the scan_result REST endpoint, where clients saw useless number=-1 rows and absent angle. Filter them out, matching the number<0 skip already used in FillEndMessage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-08 20:25:20 +02:00
leonarski_fandClaude Fable 5	4055e94b7d	Fix inverted per-image compression ratio in Lite receiver ... JFJochReceiverLite computed the per-image compression ratio as compressed/uncompressed (~0.15x) instead of uncompressed/compressed (~7x), disagreeing with the FPGA receiver and the aggregate statistic. Swap the ratio and guard against a zero compressed size. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-08 15:48:01 +02:00
leonarski_fandClaude Fable 5	51bfdd2b35	Fix acquisition error-handling and rugnux CLI issues from branch review ... Broker acquisition failure paths: - Report a truncated writer output / missing packets as errors ahead of the queue-full warning, so an incomplete dataset is no longer masked as a "reduce frame rate" warning. - Route PCIe/FPGA hardware faults to the Error state: make PCIeDeviceException a JFJochCriticalException, and stop JFJochServices::Stop() from slicing the caught exception (capture/rethrow via exception_ptr) so the critical type survives to the state machine. - Stop the (possibly partially-armed) detector, not just the receiver, when a detector arm fails, so the next start does not see it BUSY. - Return structured Error_message JSON for bind_json operation failures (e.g. a failed synchronous /start), matching the no-arg endpoints. rugnux CLI: - Default -N to all hardware threads, resolved once after parsing so full analysis, --azint-only and --scale behave the same. - Parse numeric option arguments strictly (reject non-numeric input and trailing garbage) instead of atof/stod, which silently yielded 0 or aborted the process; require --wavelength > 0. - Emit -R attached (-R100) in the reproduced command line so it re-parses. - Compare the reference MTZ against the cell that actually drives the merge in the --scale path. - Correct the --scaling-high-resolution help text and de-duplicate the offline-output experiment setup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-08 15:48:01 +02:00
leonarski_fandClaude Fable 5	77e53ac714	Fix exception handling in acquisition failure paths ... Follow-ups to the Idle-vs-Error acquisition failure split: - JFJochServices::Stop: raise the critical detector fault BEFORE the ordinary receiver/writer exception. Both can be set at once (detector not idle, then receiver->Stop() also throws); throwing the ordinary one first masked the detector fault and returned the broker to Idle instead of Error, skipping the required re-initialisation. - SLSDetectorWrapper::Stop/Deactivate: rethrow a JFJochException as-is (as Start already does) so InternalStop's JFJochCriticalException is not downgraded to an ordinary JFJochException by the generic catch. - JFJochServices::Start: wrap the whole best-effort receiver cleanup (Cancel + Stop) in the catch so a throwing Cancel cannot replace the original detector exception before it is re-raised. Also fix the "cannot create indexing pool" failure seen when retrying initialize after a failed acquisition start: a failed start leaves the receiver (and its GPU resources) alive until the next Start, but the retried initialize rebuilds the indexer pool first, so the fresh GPU indexer had to coexist with the stale receiver and its init failed. JFJochReceiverService::Indexing now releases the previous receiver before (re)building the pool (safe: only runs when the receiver is idle), which also removes a dangling pool pointer held by that receiver. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-08 14:36:25 +02:00
leonarski_fandClaude Fable 5	01e4f81e44	Consolidate offline tools into rugnux; merge on by default ... Fold jfjoch_azint and rugnux_scale into the single rugnux CLI, and flip scaling/merging from opt-in to on-by-default. - rugnux --azint-only replaces jfjoch_azint (reuses ProcessMode:: AzimuthalIntegration; adds the correction toggles and geometry overrides the old tool carried). - rugnux --scale replaces rugnux_scale (re-scale/merge stored reflections, same simpler scale/merge as before: no space-group search or rot3d defaults). Its workflow is folded into the CLI verbatim. - Merging is now on by default for rotation and stills; --no-merge disables it, replacing -M/--scale-merge. Also fix the batch ReadReflections reader so it works on rugnux's own NXmxIntegrated _process.h5: read per-image reflections/mosaicity/lattice from the master (global index) first, falling back to the source-data locator for legacy/VDS datasets. Without this, both --scale and the former rugnux_scale read zero reflections from an integrated snapshot. RugnuxCommandLine now emits `rugnux --azint-only` and `--no-merge`; docs and tests updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-08 14:09:15 +02:00
leonarski_fandClaude Opus 4.8	1caa57744a	Rebrand offline processing as rugnux (jfjoch_process -> rugnux) ... Split the naming: rugnux = data-processing subsystem, Jungfraujoch = streaming/acquisition. Executables jfjoch_process -> rugnux (source tools/rugnux_cli.cpp) and jfjoch_scale -> rugnux_scale; the processing library process/ -> rugnux/ with class/target JFJochProcess -> Rugnux (JFJochProcessObserver -> RugnuxObserver, JFJochProcessCommandLine -> RugnuxCommandLine). Doc JFJOCH_PROCESS.md -> RUGNUX.md, reconciled with the live usage message (drop dead -P/--partiality, -w/--wedge; --process-as-stills -> --force-still; add the real rot3d scaling knobs). New docs/NAMING.md explains both names, with pronunciation and a note on Romansh. rugnux now scales and merges rotation data automatically (implicit -M); stills still require an explicit -M. jfjoch_viewer and its classes keep their names (rename deferred); only their references to the renamed library are updated. The _process.h5 output suffix and ProcessConfig/Mode/Result are kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-07 22:46:58 +02:00
leonarski_fandClaude Opus 4.8	0a28da7c2e	Acquisition errors return to Idle instead of Error state ... Going to the Error state after any acquisition failure forced an expensive re-initialisation (reconnect + full detector config + pedestals), even when the detector was still perfectly configured and only a transient problem occurred. Split failures into two classes: - ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) return to Idle with an Error-severity message, so the run can be retried without re-initialising; - failures that leave the detector in an undefined state throw the new JFJochCriticalException and still go to Error, forcing re-initialisation. The detector layer is the single source of truth for "detector faulted": SLSDetectorWrapper::InternalStop and DectrisDetectorWrapper::CheckBusyOrError raise JFJochCriticalException on the ERROR state, and JFJochServices::Stop raises it when the detector does not return to idle at the end of a run. MeasurementThread dispatches purely on exception type - no detector-status field checks. Also: - JFJochServices::Start now stops the receiver it launched if the detector fails to start, so the service returns to idle cleanly and the run can be retried. - A synchronous Start rethrows the failure to the caller (HTTP layer) via a stored exception_ptr, so a failed start is no longer reported as HTTP 200. - JFJochServices::Stop no longer throws on a writer error; it is reported through receiver_output.writer_err and surfaced as an Idle + Error-severity message, which makes the previously-dead writer_err branch in MeasurementThread live. Initialisation, calibration/pedestal, SelectDetector and Deactivate keep the Error-state behaviour, as those genuinely leave the system in an undefined state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-07 14:10:48 +02:00