v1.0.0-rc.157 #67

Merged

leonarski_f merged 41 commits from 2607-state-machine into main 2026-07-11 07:19:12 +02:00

Conversation 0 Commits 41 Files Changed 267

+3761 -1883

leonarski_f commented 2026-07-08 20:53:44 +02:00

Owner

Copy Link

Copy Source

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

• rugnux: Rebrand the offline data-processing subsystem as rugnux and consolidate all offline analysis into the single rugnux binary - jfjoch_process is now rugnux, the former jfjoch_azint is now rugnux --azint-only, and jfjoch_scale is now rugnux --scale (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (--no-merge disables them), replacing the previous opt-in -M, --scale-merge.
• rugnux: CLI fixes - default -N to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require --wavelength > 0, and correct the reproduced command line and --scale reference-cell handling.
• rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics.
• rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1.
• rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below --min-captured-fraction (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group.
• rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved jfjoch prefix.
• jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation.
• jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning.
• jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint.
• jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed).
• jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types.
• jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails.
• jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak.
• Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation.
• CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.

leonarski_f added 9 commits 2026-07-08 20:53:45 +02:00

Acquisition errors return to Idle instead of Error state ... 0a28da7c2e

Going to the Error state after any acquisition failure forced an expensive
re-initialisation (reconnect + full detector config + pedestals), even when the
detector was still perfectly configured and only a transient problem occurred.

Split failures into two classes:
 - ordinary acquisition failures (receiver/writer/analysis problems, missed
   packets, writer disconnect) return to Idle with an Error-severity message, so
   the run can be retried without re-initialising;
 - failures that leave the detector in an undefined state throw the new
   JFJochCriticalException and still go to Error, forcing re-initialisation.

The detector layer is the single source of truth for "detector faulted":
SLSDetectorWrapper::InternalStop and DectrisDetectorWrapper::CheckBusyOrError
raise JFJochCriticalException on the ERROR state, and JFJochServices::Stop raises
it when the detector does not return to idle at the end of a run. MeasurementThread
dispatches purely on exception type - no detector-status field checks.

Also:
 - JFJochServices::Start now stops the receiver it launched if the detector fails
   to start, so the service returns to idle cleanly and the run can be retried.
 - A synchronous Start rethrows the failure to the caller (HTTP layer) via a
   stored exception_ptr, so a failed start is no longer reported as HTTP 200.
 - JFJochServices::Stop no longer throws on a writer error; it is reported through
   receiver_output.writer_err and surfaced as an Idle + Error-severity message,
   which makes the previously-dead writer_err branch in MeasurementThread live.

Initialisation, calibration/pedestal, SelectDetector and Deactivate keep the
Error-state behaviour, as those genuinely leave the system in an undefined state.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Rebrand offline processing as rugnux (jfjoch_process -> rugnux) ... 1caa57744a

Split the naming: rugnux = data-processing subsystem, Jungfraujoch = streaming/acquisition. Executables jfjoch_process -> rugnux (source tools/rugnux_cli.cpp) and jfjoch_scale -> rugnux_scale; the processing library process/ -> rugnux/ with class/target JFJochProcess -> Rugnux (JFJochProcessObserver -> RugnuxObserver, JFJochProcessCommandLine -> RugnuxCommandLine).

Doc JFJOCH_PROCESS.md -> RUGNUX.md, reconciled with the live usage message (drop dead -P/--partiality, -w/--wedge; --process-as-stills -> --force-still; add the real rot3d scaling knobs). New docs/NAMING.md explains both names, with pronunciation and a note on Romansh.

rugnux now scales and merges rotation data automatically (implicit -M); stills still require an explicit -M.

jfjoch_viewer and its classes keep their names (rename deferred); only their references to the renamed library are updated. The _process.h5 output suffix and ProcessConfig/Mode/Result are kept.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Consolidate offline tools into rugnux; merge on by default ... 01e4f81e44

Fold jfjoch_azint and rugnux_scale into the single rugnux CLI, and flip
scaling/merging from opt-in to on-by-default.

- rugnux --azint-only replaces jfjoch_azint (reuses ProcessMode::
  AzimuthalIntegration; adds the correction toggles and geometry overrides
  the old tool carried).
- rugnux --scale replaces rugnux_scale (re-scale/merge stored reflections,
  same simpler scale/merge as before: no space-group search or rot3d
  defaults). Its workflow is folded into the CLI verbatim.
- Merging is now on by default for rotation and stills; --no-merge disables
  it, replacing -M/--scale-merge.

Also fix the batch ReadReflections reader so it works on rugnux's own
NXmxIntegrated _process.h5: read per-image reflections/mosaicity/lattice
from the master (global index) first, falling back to the source-data
locator for legacy/VDS datasets. Without this, both --scale and the former
rugnux_scale read zero reflections from an integrated snapshot.

RugnuxCommandLine now emits `rugnux --azint-only` and `--no-merge`; docs and
tests updated to match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fix exception handling in acquisition failure paths ... 77e53ac714

Follow-ups to the Idle-vs-Error acquisition failure split:

- JFJochServices::Stop: raise the critical detector fault BEFORE the ordinary
  receiver/writer exception. Both can be set at once (detector not idle, then
  receiver->Stop() also throws); throwing the ordinary one first masked the
  detector fault and returned the broker to Idle instead of Error, skipping the
  required re-initialisation.
- SLSDetectorWrapper::Stop/Deactivate: rethrow a JFJochException as-is (as Start
  already does) so InternalStop's JFJochCriticalException is not downgraded to an
  ordinary JFJochException by the generic catch.
- JFJochServices::Start: wrap the whole best-effort receiver cleanup (Cancel +
  Stop) in the catch so a throwing Cancel cannot replace the original detector
  exception before it is re-raised.

Also fix the "cannot create indexing pool" failure seen when retrying
initialize after a failed acquisition start: a failed start leaves the receiver
(and its GPU resources) alive until the next Start, but the retried initialize
rebuilds the indexer pool first, so the fresh GPU indexer had to coexist with
the stale receiver and its init failed. JFJochReceiverService::Indexing now
releases the previous receiver before (re)building the pool (safe: only runs
when the receiver is idle), which also removes a dangling pool pointer held by
that receiver.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fix acquisition error-handling and rugnux CLI issues from branch review ... 51bfdd2b35

Broker acquisition failure paths:
- Report a truncated writer output / missing packets as errors ahead of the
  queue-full warning, so an incomplete dataset is no longer masked as a
  "reduce frame rate" warning.
- Route PCIe/FPGA hardware faults to the Error state: make PCIeDeviceException a
  JFJochCriticalException, and stop JFJochServices::Stop() from slicing the
  caught exception (capture/rethrow via exception_ptr) so the critical type
  survives to the state machine.
- Stop the (possibly partially-armed) detector, not just the receiver, when a
  detector arm fails, so the next start does not see it BUSY.
- Return structured Error_message JSON for bind_json operation failures (e.g. a
  failed synchronous /start), matching the no-arg endpoints.

rugnux CLI:
- Default -N to all hardware threads, resolved once after parsing so full
  analysis, --azint-only and --scale behave the same.
- Parse numeric option arguments strictly (reject non-numeric input and trailing
  garbage) instead of atof/stod, which silently yielded 0 or aborted the
  process; require --wavelength > 0.
- Emit -R attached (-R100) in the reproduced command line so it re-parses.
- Compare the reference MTZ against the cell that actually drives the merge in
  the --scale path.
- Correct the --scaling-high-resolution help text and de-duplicate the
  offline-output experiment setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fix inverted per-image compression ratio in Lite receiver ... 4055e94b7d

JFJochReceiverLite computed the per-image compression ratio as
compressed/uncompressed (~0.15x) instead of uncompressed/compressed (~7x),
disagreeing with the FPGA receiver and the aggregate statistic. Swap the ratio
and guard against a zero compressed size.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Drop uncollected placeholder rows from scan_result ... 023a30cb08

ScanResultGenerator pre-sizes its vector to the full expected image
count, so GetResult() returned placeholder entries (number = -1, no
angle) for images not yet collected. These leaked to the scan_result
REST endpoint, where clients saw useless number=-1 rows and absent
angle. Filter them out, matching the number<0 skip already used in
FillEndMessage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

VERSION: 1.0.0-rc.157 1b5ffb4337

CHANGELOG: move branch changes from rc.156 to a new rc.157 section ... c6af375dc0

The rugnux consolidation bullet had been attributed to rc.156, which is
the release this branch is based on. Restore rc.156 to match main and
collect all of this branch's changes under a new rc.157 entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

leonarski_f added 10 commits 2026-07-09 15:50:35 +02:00

Update grid scan_result test for dropped placeholder rows ... b4a29ebb32

Commit 023a30cb0 intentionally filters uncollected placeholder rows
(number = -1) out of ScanResultGenerator::GetResult() but left this
test asserting the old full-grid vector size. The 3x3 grid collects
only 7 of 9 cells, so scan_result now has 7 rows; the two uncollected
placeholders sit at grid indices 5 and 8, after the checked positions,
so the number assertions still hold.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

viewer: fix HTTP live-follow OOM via datasetLoaded backpressure + shared pixel mask ... dc8a80f3e0

The viewer could grow to ~100 GB RAM when live-following an HTTP broker. The
rc.153 images_in_flight backpressure only throttled imageLoaded; the heavy
per-frame payload rides datasetLoaded, fanned out over ~10 queued cross-thread
connections with no cap. In HTTPSyncDataset follow mode (entered when an operator
clicks an image while following live) RefreshDatasetOnly_i emits a fresh full
dataset every autoload tick with no imageLoaded, so the gate never engaged and
the queued events - each pinning a full JFJochReaderDataset (full-detector
PixelMask + per-image plots) - accumulated without bound.

Backpressure datasetLoaded the same way as imageLoaded: a datasets_in_flight
counter (cap 2), all emits routed through EmitDatasetLoaded_i, and
AutoLoadTimerExpired gated on it (covers HTTPSyncDataset). The window routes the
worker's datasetLoaded through a single OnDatasetReady sink that fans out
synchronously via datasetReady and acks with datasetConsumed. Under load stale
datasets are dropped; the next tick sends the latest.

Share the pixel mask instead of deep-copying it: JFJochReaderDataset::pixel_mask
is now shared_ptr<const PixelMask>, so per-frame dataset copies share the ~72 MB
mask. UpdateUserMask does copy-on-write; JFJochHttpReader caches the mask by
arm_date so a live refresh reuses one shared mask per acquisition.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

image_analysis: add retain_outcomes flag to IndexAndRefine; viewer opts out ... b2f0af12db

IndexAndRefine::integration_outcome is a whole-run vector (resize(GetImageNum()))
holding up to ~0.8 MB of reflections per indexed image. It is consumed only by
the offline scaling/merge pass in rugnux (ScaleAllImages / GetIntegrationOutcome).
In the viewer's interactive/live analysis it is written but never read - dead
storage that, with auto-reanalyze on during live follow, grew ~0.8 MB per frame
for the whole run.

Add a retain_outcomes constructor flag (default true). When false, skip the
whole-run resize and the per-image store; the current image's reflections are
still returned via the outgoing message. The viewer's live IndexAndRefine passes
false. rugnux (including GUI processing jobs, which build their own Rugnux ->
IndexAndRefine) and the online receiver keep the default true, so scaling/merge
still has the accumulated data.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

image_analysis: remove dead RotationParameters; receiver opts out of retain_outcomes ... 03dad9fe93

RotationParameters was a moving-average store (profile_radius, beam_x, beam_y,
mosaicity) held by IndexAndRefine but never read or written - none of its four
accessors had any call site. Remove the class, its member, the include, and the
CMake entries.

Also set retain_outcomes=false for the online receiver's IndexAndRefine: online
scaling/merge at the end of a run is not enabled yet, so the whole-run
integration_outcome is unused there. Revert to true when online scaling lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Preview: geometry-following arcs, true beam center, unified colours, predictions ... a4559e576d

Draw the resolution ring and azimuthal-ROI outlines by sweeping ResPhiToPxl
through the geometry (a shared DrawArc/DrawThickLine helper), so they trace the
true conic on a tilted detector instead of PONI-centred circles. Draw the beam
crosshair at GetDirectBeam_pxl() (the real beam/detector intersection) rather
than the stored PONI.

Unify the spot overlay colours with the viewer: green = not indexed, magenta =
indexed (primary lattice), cyan = ice ring, coral = secondary/further lattice.
Add Bragg-prediction overlay (dark-red circles, skipping systematic absences)
gated on a new show_predictions preview setting, wired through the OpenAPI spec,
the image.jpeg HTTP handler, and the frontend toggle (+regenerated TS client).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Viewer: coral secondary-lattice spots, ring arcs, legend, ROI swatches, save JPEG ... ec05a6c4d6

- Colour indexed spots from a secondary (or further) lattice in coral (#FA7268),
  the viewer's finishing accent, in the diffraction-image overlay.
- Draw resolution rings by sweeping ResPhiToPxl into a QPainterPath (like the
  azimuthal ROI arcs), replacing the 4-point bounding-box ellipse and the
  poni_rot_3=0 hack, so rings follow the geometry on tilted detectors.
- Add a spot/prediction colour legend to the Image-features side panel; the two
  user-adjustable swatches track the colour picker.
- Show a colour swatch matching the on-image annotation on each ROI-list item,
  via a shared ROIAnnotationColor palette used by both the overlay and the list.
- Add "Save image as JPEG..." / "...with overlay as JPEG..." to the image
  right-click menu, reusing a factored renderToImage() helper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

image_pusher: reset writer when EndDataCollection finalization fails ... 36be60774f

HDF5FilePusher::EndDataCollection released the FileWriter only after
writer->WriteHDF5()/Finalize(), both of which throw at rename time when a
target file already exists and overwrite is off. The throw skipped
writer.reset(), leaving 'writer' non-null so the next StartDataCollection
died with "Image pusher is already writing images".

Tear the writer down on every exit path (try/catch + rethrow) so a failed
collection no longer blocks the next one. The error still propagates and the
orphaned temp files are intentionally left on disk.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

writer: detect overwrite conflict at start for back-channel transports ... ad73b4af99

Previously an existing output file was only detected at the final rename,
after the whole collection had run. Add an up-front check in the FileWriter
constructor that fails before acquisition arms.

Only the master file is checked, and only by the single writer that owns it
(write_master_file). In a multi-writer TCP/ZMQ setup the per-image data files
are staggered across writers by file number, and a writer does not even know
the total writer count, so it cannot enumerate its own subset; checking all
data files in every writer would make each writer stat files it never writes
and race the writers already creating them. Data-file conflicts stay caught by
their owning writer at the final rename.

Gate the check on whether the transport can report the failure to the broker:
the direct HDF5 pusher throws in-process and the TCP writer returns a
START-failure ACK, so both abort the collection cleanly (verified: an asymmetric
rejection cancels the already-started sibling). The ZeroMQ path is fire-and-forget
with no back-channel, so StreamWriter opts out via image_puller.SupportsAck() and
keeps writing .tmp files, failing only at the final rename (leaving the images on
disk). Documented in docs/JFJOCH_WRITER.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs: rugnux de-novo symmetry/indexing investigation notes ... c0f6522778

Records the C2-missed-as-P1 root cause + the (stashed) RotationOnly recovery fix
and why it is a re-orientation step, the Benas F432 point-group/error-model finding,
and the CC1/2-logistic resolution-cutoff design.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs: rugnux automatic resolution-cutoff design (CC1/2 logistic, one shell too far) ... be937d0163

CC1/2 fall-off fit as a logistic in 1/d^2, cut one shell past CC1/2~0.3 - generous
for ML, robust to the broken error model (works on 4 A Benas where I/sigma cannot),
and avoids defining 'the' resolution. Includes experiments vs XDS and an
implementation plan. Not yet implemented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

leonarski_f added 2 commits 2026-07-09 17:46:13 +02:00

Remove rugnux work-in-progress documents from Git 4df811bf73

rugnux: automatic CC1/2-logistic high-resolution cutoff ... 4f3153d6c7

Merge/report/write to the CC1/2 fall-off instead of the detector edge.
After the final merge, fit a logistic to a fine-binned CC1/2 profile and
cut one shell past cc_target (default 0.30) - generous, DIALS-style. Only
the written MTZ/CIF and the reported shell table are trimmed; scaling, the
3D combine and the error model still run over the full range, and the
per-image _process.h5 keeps full data for re-scaling. The de-novo P1 merge
that feeds the space-group search is never cut (gated on the final merge).

- ScalingSettings: resolution_cutoff {Off,CCHalfLogistic}, resolution_cc_target,
  report_shell_count (report shell count is now configurable).
- New ResolutionCutoff.{h,cpp}: ComputeCCHalfLogisticCutoff (fine-bin CC1/2 +
  grid/refine logistic fit + one-shell extension + degeneracy/low-res guards).
- Wired into RotationScaleMerge (rotation) and MergeOnTheFly/Rugnux (stills),
  plus the rugnux --scale path.
- CLI: --resolution-cutoff cc-logistic|off, --resolution-cc-target,
  --resolution-shells. --scaling-high-resolution stays the manual override
  (wins and disables the auto-cut).

Validated on the 24-crystal battery: no crashes (incl. Benas/cytC_10/Thau_9);
--resolution-cutoff off is bit-identical to baseline; cutoffs reproduce the
design table within ~0.05 A (lysoC 1.30, Ins_I 1.43, cytC_2 1.68, EcwtCQ244-E3
1.54); good-to-edge datasets (lyso_ref, EP_cs_02-10) correctly keep the full
range.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 2 commits 2026-07-09 22:03:39 +02:00

rugnux: recover genuine high symmetry in de-novo space-group search ... 8c288c8697

The POINTLESS-style symmetry search discarded a real point group on two
hard rotation-test crystals:

- Ins_I_3 (I23): one of the eight cubic 3-fold operators scored CC 0.483,
  just under the 0.5 per-operator floor, so I23 fell to I222 and merging
  collapsed (3-fold orientation ambiguity; CC1/2 1%, R-meas 558%). Gate A
  now scores each CONJUGACY CLASS of a point group's rotations by its mean
  CC - conjugate operators are symmetry-equivalent and must stand or fall
  together - so the cubic 3-fold class (mean 0.57) is confirmed. -> I23,
  CC1/2 99.9%. Degenerate (few-pair / NaN) class members are skipped
  rather than vetoing the class.

- Benas_3/7 (F432): the chi^2 self-consistency gate rejected the true
  cubic group because the weak-data error model is badly miscalibrated
  (reduced chi^2 ~150), which inflates the chi^2 ratio with point-group
  order for genuine high symmetry too. The ratio bound is now widened by
  log10(chi2_ref) so a broken error model does not veto real symmetry,
  while a false operator whose ratio is far worse than its subgroup's is
  still caught. The bound stays a per-candidate chi^2 test - chi^2 still
  arbitrates and is never bypassed. Verified: F432 (chi2_ref 151, ratio
  3.2 < bound 4.2) accepted; a calibrated pseudo/twin R32 (Ins_H_3:
  chi2_ref 2.6, ratio 4.2 > bound 2.4) still rejected, stays R3.

SearchSpaceGroup.{cpp,h} only. Validated on the 24-crystal rotation-test
battery: exactly these two space groups change (both now match the
reference); zero regressions - every pseudo-symmetry P1 case and every
correct assignment unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: run the L-test on rhombohedral data + write twinning stats to CIF ... a4cc908ff1

The Padilla-Yeates L-test paired each reflection with a neighbour two
steps along an axis ((h+2,k,l) etc.). A step of 2 preserves the
reflection condition of P/I/C/F/A/B lattices but violates R-centring
(-h+k+l = 0 mod 3, since 2 != 0 mod 3), so every neighbour was
systematically absent -> zero pairs, and the L-test silently did not run
on ANY rhombohedral crystal (twinning there was left to the weaker
second-moment test alone). Add diagonal (1,1,0)/(1,1,3) fallback steps
that preserve the mod-3 condition in both obverse and reverse settings,
tried only when the axis steps find no present partner (P/I/C/F
behaviour unchanged). On the twinned R3 insulin the L-test now runs:
25k pairs, <L^2> = 0.224 -> twinning detected.

Also write the twinning indicators to the output mmCIF next to ISa:
_reflns.pdbx_L_test_mean_abs_L / _L_test_mean_L_squared /
_second_moment_I, as free-text pdbx values, guarded on pair/reflection
count. Threaded the TwinningAnalysisResult into WriteReflections (both
the full-pipeline and --scale callers).

Note: on rhombohedral lattices the L-test reads ~0.03 lower than on
P/I/C/F (its allowed step is slightly more local), so it separates the
insulin twin pair only weakly (twinned 0.398 vs untwinned 0.427) and the
second moment remains the cleaner discriminator there.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-10 07:26:18 +02:00

Name the Windows NSIS installer jfjoch-viewer-*, not jfjoch-* ... ca5ac9f838

Match the Linux viewer TGZ naming; update the Gitea upload glob to suit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 2 commits 2026-07-10 13:03:45 +02:00

rugnux: measure total processing time incl. scaling, not just the per-image pass ... 09c1a04786

The reported "Processing time" / "Frame rate" stopped the clock right after the
per-image analysis loop, excluding scaling, merging, resolution cutoff, twinning
and file writing. Move the timing to the end of Rugnux::Run() so processing_time_s
(and the derived frame_rate_hz / throughput_MBs) span the whole run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

rugnux: emit non-standard mmCIF items under a "jfjoch" reserved prefix, not pdbx_ ... 12efbc1beb

ISa and the twinning indicators (L-test means, second moment of I) have no
standard mmCIF item. They were written as _reflns.pdbx_* names, borrowing the
wwPDB PDBx/mmCIF dictionary's reserved prefix and so masquerading as standard
items the dictionary does not actually define.

Move them to the IUCr-sanctioned local-data-name extension: a private reserved
prefix on the attribute of the existing _reflns category (_reflns.jfjoch_*),
matching the DDL2 pattern the spec gives for private items. Write-only tags with
no reader, so this is a self-contained, back-compatible rename.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 8 commits 2026-07-10 14:33:14 +02:00

rugnux: recover a centred Bravais lattice from intensities (de-novo C2) ... 7d240a120c

When de-novo rotation indexing demotes a pseudo-symmetric metric to a
primitive/triclinic cell, the space-group search never tested the candidate
symmetry operators - its operator set is bounded by the indexer's lattice, whose
holohedry is trivial for P1 - so a genuinely centred lattice (C-centred
monoclinic, EcwtCQ244) was locked to P1. A merely pseudo-symmetric triclinic
cell (EcwtCQ066) is metrically indistinguishable from a real one (same 2x
C-lattice, same absences), so the decision cannot be made from spot positions:
the 2-fold is an intensity property and must be decided after P1 integration,
as XDS does in CORRECT.

After the P1 space-group search, if the metric is pseudo-symmetric, reindex the
merged reflections to the candidate's conventional setting and re-run the search
there. The I(h) vs I(Rh) correlation confirms a real centred lattice and rejects
a pseudo-symmetric one. The centring itself cannot be read from absences here
(integrated in the primitive cell, its centring-absent reflections do not
exist), so among the point-group-equivalent candidates the intensities cannot
separate, the one matching the metric candidate's centring is chosen. On
confirmation the integrated reflections, lattice and cell are reindexed, the
rotation merge is re-ingested, and the data are re-merged in the adopted space
group. Scaling-only - the indexer is untouched.

EcwtCQ244-B1-1 and -E3 -> C2 (2-fold CC 0.99); EcwtCQ066 stays P1 (CC 0.31); the
rest of the 24-crystal de-novo battery is bit-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: write refined geometry (beam, tilt, axis) to _process.h5 ... f0dc61ef58

The master-file geometry is written from the StartMessage at file open, before
indexing/refinement - the file is streamed per image, so it cannot wait for the
refined values. rugnux therefore stored the nominal beam centre / detector tilt /
rotation axis, not the geometry its own refinement found (jfjoch_broker, by
contrast, deliberately keeps the user-provided values).

Add optional refined geometry to EndMessage; at finalization the writer overwrites
the beam_center_x/y, PONI rot1/2/3 and rotation-axis datasets in place with these.
The broker leaves them empty, so its output is unchanged.

For EcwtCQ244-E3 the _process.h5 now reports the refined beam 761.6/847.7 (nominal
764/850, matching XDS's 760.2) plus the refined tilt and rotation axis.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

image_preprocessing: treat the type-max marker as invalid for unsigned pixels ... fd7e2ae6f9

The error/invalid pixel marker (the pixel type's extreme value, e.g. 0xFFFFFFFF for
EIGER uint32) was only mapped to the internal error value for SIGNED types - the check
was gated on std::is_signed<T>. For unsigned EIGER data a 0xFFFFFFFF pixel therefore
fell through to the saturation test and, exceeding the (HDF5) saturation_value, was
silently classified as saturated rather than invalid.

Both are excluded from integration, so results are unchanged where such pixels are also
in the static pixel mask (as in the JUNGFRAU test data - Thau_9 and lyso are bit-for-bit
the same), but the classification/statistics were wrong and a stray 0xFFFFFFFF outside
the static mask would be mis-accounted.

Test the error marker before saturation and drop the is_signed gate in both the CPU and
GPU preprocessors: the HDF5 saturation_value stays the saturation threshold, the pixel
type's maximum is the invalid marker (priority masked > error > saturated).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: PSI logo in footer + wider, shorter settings pages ... b474264a3f

Add PSI branding and use the horizontal space on the expert pages,
which previously stacked panels into one tall narrow column.

- Footer carries the official PSI logo (one of four variants picked at
  page load), standalone with clear space, linking to psi.ch. Top bar is
  a clean "Jungfraujoch" wordmark.
- PanelColumns masonry lays the processing/output/streaming/masks panels
  into responsive columns; ROI stays full width.
- DetectorSettings regrouped from a fixed-height single column into a
  responsive two-column form (layout only; controls unchanged).
- Softer corners (theme borderRadius 4 -> 8).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: visual polish pass (status pill, buttons, panels, empty states) ... abf7d5f45d

Design pass across the frontend for a coherent, stronger-but-limited look
(lime / coral / blue), keeping lime as the brand accent.

- StatusBar: broker state is a fixed-width pill (lime=Idle, coral=Busy/
  Pedestal/Error, light-blue=Inactive); while Measuring it fills like a
  progress bar (lime over the sidebar light-blue). Action buttons are a
  three-tier system (hot lime / active blue-on-blue / off ghost) so exactly
  one button reads as the thing to click now, ordered Cancel/Pedestal/
  Initialize. Cancel is the emergency stop (hot while Measuring/Pedestal).
- SettingsPanel: left-aligned card header on a light-blue band with a lime
  left accent (echoes the sidebar); inline <b> section labels restyled as
  uniform uppercase captions. DetectorSettings adopts the same header.
- EmptyState: muted icon + text for "No plots"/"No preview".
- DetectorStatus: semantic colour on state/power values.
- Snackbars offset so action messages clear the fixed footer.
- Footer PSI logo now uses the positive (dark) logos directly on the light
  footer instead of CSS-inverting the negative ones.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: reflow ROI panel into readable rows ... 314caacee7

The three ROI tables were crammed side-by-side; the 8-column Azimuthal one
was unreadable at a third width. Put Box + Radial on the first row and the
wide Azimuthal table full-width on its own row, then the buttons, under the
shared titled card header. Trim the Box/Radial name columns so every column
(including the delete action) fits at half width.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: visualize mask + pedestal TIFFs in-browser ... 04bce94f4e

Add an in-browser viewer for the broker's TIFF endpoints so the pixel mask
and JUNGFRAU pedestal can be inspected without downloading files.

- tiffDecode.ts: dependency-free decoder for the broker's baseline TIFFs
  (uncompressed, single-strip, single-channel; 8/16/32/64-bit uint/int/float).
- TiffImageViewer.tsx: manual Load -> fetch -> decode -> canvas, with
  react-zoom-pan-pinch zoom/pan and a rect-based (zoom-agnostic) hover
  readout. `value` variant = colormap + editable min/max bounds + Auto +
  colorbar; `mask` variant = discrete fixed colours per bit-flag value with
  a decoded-flag legend (mask bits mirror common/PixelMask.h).
- MaskVisualization.tsx: full-width panel on the Masks page (full/user mask
  toggle) fetching /config/mask.tiff and /config/user_mask.tiff.
- Calibration.tsx: rebuilt with the shared card header; adds a pedestal
  viewer (gain-level + storage-cell selectors -> /preview/pedestal.tiff)
  above the per-module statistics grid, and degrades gracefully with no data.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bragg_integration: quantization-noise variance floor + summation fallback ... f6eca6c69e

The Kabsch profile-fit weights floored the per-pixel variance at 1.0, far
above the real detector noise floor (electronic noise is well below 1 for
both EIGER and JUNGFRAU). The true floor is the quantization noise from
rounding the charge-spread deposited energy to an integer: Var = 1/12. The
1.0 floor silently over-regularized — it inflated weak-reflection sigma ~6x
at high resolution and pinned the scaling error model's `a` term at its 0.25
floor, suppressing ISa.

Use PIXEL_VARIANCE_FLOOR = 1/12, and guard against the weak-reflection
runaways it exposes: fall back to the box-sum (summation) intensity when the
profile fit diverges from the summation seed by more than
PROFILE_SUMMATION_MAX_NSIGMA box-sum sigmas. Applied identically in the CPU
and GPU engines (GPU-vs-CPU equivalence test passes).

Battery of 24 rotation crystals: broad ISa gains (lysoC_14 7.8->10.4, plus
cytC/Ins/Myo), EcwtAL500 completeness 51->92% and CC1/2 35->79%, pding4_001
space group now correct; no space-group regressions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-10 15:23:08 +02:00

rugnux: gate the P1 space-group-search scaling at <I/sigma> >= 1 ... 1b6831a2d3

The de-novo space-group determination (point-group intensity correlation plus
the second-moment / twinning test) ran on the full resolution range, including
noise-dominated high-res shells. There a single artifact - a zinger the box sum
integrates whole, or a weak-reflection profile-fit runaway - has an
astronomical resolution-normalised E (I / <I>_shell with <I>_shell ~ 0) and
wrecks the normalised statistics, so the search collapses to P1.

Re-scale the P1 search pass with a <I/sigma> >= 1 cut over thin resolution
shells (POINTLESS-style: determine symmetry where there is signal). Only this
pass is cut; the final in-symmetry merge keeps the full range, so no output
resolution is lost. A manual --scaling-high-resolution, if coarser, wins.

Battery of 24 rotation crystals: box-sum lysoC_14 P1 -> P41212; and with the
default gaussian integrator three previously mis-assigned crystals now get the
correct space group (cytC_10 155->152, EP_cs_01-24 5->4, Ins_I_3 196->197),
with no space-group regressions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-10 15:54:29 +02:00

docs: bring the rc.157 CHANGELOG up to date with the branch ... 403e6e7a68

Cover the de-novo symmetry / geometry / mmCIF rugnux work, the Bragg
integration and preprocessing fixes, writer overwrite/reset handling, the
jfjoch_viewer preview improvements, the web Frontend redesign + mask/pedestal
viewers, and the installer naming — kept to one brief line each.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-10 20:13:58 +02:00

rugnux: drop edge-of-sweep truncated fulls + honest R-meas over accepted obs ... f0799f780b

From a fresh decomposition of the low-resolution-shell R-meas gap vs XDS on the
reference lysozyme rotation crystal (7.6% vs 2.9%), two rot3d scale/merge changes:

- New --min-captured-fraction (default 0.5 for rotation, off otherwise): drop a
  combined full whose rocking curve was captured below this fraction. These
  edge-of-sweep, partiality-extrapolated fulls (sweep start/end) are biased and
  inflate the low-res R-meas. Dropping them lowers low-shell R-meas 7.6->5.5% and
  *improves* external accuracy (CC vs the F-model and vs an independent dataset,
  and the ANODE anomalous S-peak 0.93->0.94x XDS) - unlike --min-partiality, which
  lowers R-meas cosmetically while degrading accuracy. Threaded through
  ScalingSettings and both the GPU and CPU combine.

- R-meas reporting fix: the R_meas re-walk counted observations the merge had
  rejected as outliers (XDS excludes its misfits before its R-factor). The
  accumulate pass now flags each rejected full and the R_meas pass skips it, in
  both the GPU and CPU paths. Merged intensities / ISa / CC1/2 unchanged (verified
  byte-identical external accuracy); low-shell R-meas 7.6->6.5%. No-op when
  --reject-outliers 0.

Combined default rotation low-shell R-meas 7.6->5.1%, ISa 17.2->17.5, external
accuracy equal-or-better than baseline and XDS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 4 commits 2026-07-10 22:27:37 +02:00

rugnux: index weakly-diffracting low-resolution rotation data (Benas F-cubic) ... 7b8e08cd96

Two F432 crystals (a~70.2 A) that diffract only to ~4-4.5 A on a detector
reaching ~1.5 A failed to process de-novo. Three independent, physically
motivated fixes, validated on the 24-crystal rotation_test battery with zero
regressions (every other crystal keeps its cell, indexing rate and space group):

1. Per-frame indexing gate (AnalyzeIndexing): the indexed-fraction denominator
   counted every found spot, but for these crystals ~85% of spots are
   unindexable sub-diffraction ice/noise, making the 20% floor unreachable so
   every frame was rejected (0% indexed). Count only spots within the resolution
   range the lattice actually diffracts to (out to the highest-resolution indexed
   spot). This can only shrink the denominator, so it never rejects a frame that
   passes today. Fixes Benas_3 (0 -> 40%, correct F-cubic -> H32).

2. First-pass rotation indexing now runs multiple sampling schemes and selects
   the one that indexes the most frames on the real per-image path (spread over
   the whole rotation vs a consecutive wedge from the clean start). The
   long-standing spread scheme wins for typical crystals (unchanged), while the
   consecutive wedge keeps a reflection's rocking curve continuous across frames
   so the FFT can trace a long axis whose fine reciprocal spacing the coarse
   spread cannot resolve. Fixes Benas_7 (garbage cell 0.06% -> correct hex 69%).
   Scheme selection reuses ProcessImage's indexing path (extracted as
   DetermineRefineAnalyze / IndexFrameOnly) so validation cannot diverge from
   production.

3. Space-group-search resolution gate (Rugnux): the <I/sigma> >= 1 cut collapsed
   to ~the cell dimension when every shell is below the threshold (I/sigma floored
   by the error model though CC1/2 is high), leaving the re-merge with no
   reflections -> "resolution calculation failed" crash. Only cut when a good
   low-res region exists above the cutoff.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: CHANGELOG entries for weak-data indexing and min-captured-fraction ... 1ea3846957

Document the rc.157 changes: the Benas low-resolution rotation indexing fixes
(gate diffracting-range denominator, multi-scheme first pass, space-group-search
resolution guard) and the --min-captured-fraction / honest-R-meas rotation
scaling change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: raise default min-captured-fraction to 0.7 (recover weak F-cubic space groups) ... ef103d2ea2

The de-novo space-group search runs its point-group correlation only on reflections
with I/sigma >= 3, but the Diederichs error model floors merged I/sigma at the
asymptotic ISa = 1/b. On a weakly-diffracting crystal a few edge-of-sweep truncated
fulls (rocking curve captured far below 1) inflate the error-model b term, dropping
ISa below 3 so no reflection clears the gate: every operator gets zero pairs and the
point group collapses to P1 (or a low subgroup). The chi^2 self-consistency guard is
likewise corrupted by those zingers.

Raising the rotation default of --min-captured-fraction from 0.5 to 0.7 drops those
partiality-extrapolated fulls, lifting ISa back above the gate and cleaning the
intensity statistics, so the search recovers the true symmetry. On the F-cubic Benas
crystals: Benas_3 P4 -> F432 (CC1/2 78% -> 98%) and Benas_7 P1 -> P6122 (CC1/2 40% ->
99%). Benas_7's data supports hexagonal P6122, not cubic F432 - the cubic 3-fold/4-folds
fail the merge self-consistency test - matching XDS.

Full /data/rotation_test battery: every other crystal keeps its exact space group; the
only cost is a little multiplicity from the extra dropped edge-fulls on a few weak/
narrow-sweep crystals (worst completeness -7% on EP_cs_01-24, whose CC1/2 nonetheless
rises), while several crystals' CC1/2 improve.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: trim phase/timing log noise, keep step results ... d4544bd3c8

Cut the started-phase and partial-timing chatter from the rugnux
processing log while keeping the genuine step results and the final
performance report:

- Rugnux.cpp: the scaling/merging phase() lambda now only drives the
  GUI progress bar (observer->OnPhase); drop the per-transition
  "[timing] <phase>: X.XX s" lines (in the CLI the observer is null, so
  those timings were the only thing it printed). Also drop the two
  "started-phase" lines "Scaling+merging (<pass>) via RotationScaleMerge"
  and "Running scaling (<pass>) ..." - RotationScaleMerge already logs
  its own error model / combine / cutoff / merge-complete results, and
  the stills path still logs its error model and merge-complete.
- rugnux_cli.cpp: drop the duplicate "Starting analysis of N images..."
  (Rugnux::Run already logs the richer "Processing N images ..." line)
  and the duplicate --max-spots log (kept where it is applied).

Error-model params, adopted space group, lattice-search scores,
ice-ring flag/mask results, auto resolution cutoff, and the final
"Processed N images in X s (Y Hz)" report are all unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f merged commit 54c0100e8e into main 2026-07-11 07:19:11 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#67