v1.0.0-rc.158 #68

Merged

leonarski_f merged 47 commits from 2607-win-viewer into main 2026-07-12 19:42:29 +02:00

Conversation 0 Commits 47 Files Changed 205

+2010 -1179

leonarski_f commented 2026-07-11 12:53:18 +02:00

Owner

Copy Link

Copy Source

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

• Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (cos^3 of that angle) instead of cos^3(2*theta), so it is correct for a tilted detector and matches PyFAI solidAngleArray and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (XtalOptimizer) no longer silently ignores an imported PONI rot3 (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured rot3 through the full PONI rotation.
• jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting \\); write processing outputs next to the input file, with a Browse button and independent _process.h5 / merged .mtz/.cif toggles; and show the determined space group in the merge-statistics window.
• jfjoch_viewer: Connect to a broker over https (an http/https selector in the connect dialog), and keep the HTTP connection alive across reads for faster live-follow.
• jfjoch_viewer: Time out stalled HTTP requests (5 s) so an unreachable broker cannot hang the reader thread, and drop the cached pixel mask when switching data source.
• rugnux: Accept an absolute -o output prefix in offline processing.
• rugnux: Faster two-pass rotation indexing - the first pass now runs its FFT indexing and geometry refinement in parallel (results unchanged).
• rugnux: Rotation indexing now works on standard DECTRIS datasets that store no spots - the first pass finds spots itself instead of failing.
• rugnux: De-novo symmetry robustness - don't over-promote a merohedral twin to the holohedral group (keep e.g. R3, not R32), make the intensity second-moment twinning statistic robust on weak/mis-integrated data, and don't flag twinning in holohedral Laue classes where no twin law can exist.
• jfjoch_writer: Fold the refined beam centre into the NXmx detector translation vector too (not only the informational beam_center fields), so a reprocessed _process.h5 has a self-consistent refined geometry.
• Robustness: Harden size handling of untrusted input in TIFF reading and raw-TCP frames.
• Packaging: The self-contained Linux viewer .tgz now bundles cuFFT, so it runs without a system CUDA toolkit (.deb/.rpm are unchanged, distro-managed).
• Docs: Documentation updated to match the current analysis code and CLI.

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * jfjoch_viewer: Connect to a broker over `https` (an http/https selector in the connect dialog), and keep the HTTP connection alive across reads for faster live-follow. * jfjoch_viewer: Time out stalled HTTP requests (5 s) so an unreachable broker cannot hang the reader thread, and drop the cached pixel mask when switching data source. * rugnux: Accept an absolute `-o` output prefix in offline processing. * rugnux: Faster two-pass rotation indexing - the first pass now runs its FFT indexing and geometry refinement in parallel (results unchanged). * rugnux: Rotation indexing now works on standard DECTRIS datasets that store no spots - the first pass finds spots itself instead of failing. * rugnux: De-novo symmetry robustness - don't over-promote a merohedral twin to the holohedral group (keep e.g. R3, not R32), make the intensity second-moment twinning statistic robust on weak/mis-integrated data, and don't flag twinning in holohedral Laue classes where no twin law can exist. * jfjoch_writer: Fold the refined beam centre into the NXmx detector `translation` vector too (not only the informational `beam_center` fields), so a reprocessed `_process.h5` has a self-consistent refined geometry. * Robustness: Harden size handling of untrusted input in TIFF reading and raw-TCP frames. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Documentation updated to match the current analysis code and CLI.

leonarski_f added 18 commits 2026-07-11 12:53:18 +02:00

viewer: open WSL/UNC paths by converting to native separators ... 4292c990ea

Qt's QFileDialog returns '/'-separated paths even on Windows, so a UNC path
like \\wsl.localhost\Ubuntu\... arrives as //wsl.localhost/Ubuntu/... and
H5Fopen (via the Win32 file layer) does not recognise the forward-slash form as
UNC, while C:/... still works - exactly the reported symptom. Normalise the
file-dialog result with QDir::toNativeSeparators before opening. No-op on Linux.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

rugnux: allow absolute output paths for offline processing ... 8858951419

The FilePrefix/CheckPath guard forbids absolute paths and '..' traversal so a
remote client cannot make the broker/writer write outside their run directory.
Offline processing (rugnux -o, and the viewer writing next to the input file)
supplies its own trusted local path, so that guard should not apply to it.

Add a trusted opt-in that leaves the broker/writer path untouched:
- DatasetSettings/DiffractionExperiment::FilePrefixTrusted() sets the prefix
  without CheckPath (FilePrefix() = CheckPath + FilePrefixTrusted).
- FileWriter gains a trusted_path ctor flag that skips its own CheckPath.
- Rugnux uses both. Broker/writer never set them, so their behaviour is
  identical; this only widens what the offline CLI/viewer may write.

Previously 'rugnux -o /abs/path' threw ('Path cannot start with slash') while
building the _process.h5; the .mtz/.cif already bypassed the guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: write processing output next to the input file ... 343af8deed

The New-job dialog seeded the output prefix from QFileInfo::completeBaseName(),
which drops the directory, so _process.h5/.cif/.mtz landed in the viewer's
working directory (wherever it was installed) instead of beside the dataset.
Seed it with the input file's absolute directory instead; the field stays
editable. Relies on the trusted output-path support so the absolute prefix is
accepted by the writer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: control _process.h5 and merged .mtz/.cif output independently ... 9b1ab02c41

The 'Save _process.h5' checkbox drove output_prefix, so unchecking it also
suppressed the merged .mtz/.cif, and there was no way to run for the ISa/R-meas
numbers only. Split it into two checkboxes wired to independent flags:
- Save _process.h5  -> ProcessConfig::write_process_h5 (also drives the viewer
  snapshot, so unchecking it means the viewer is not updated with spots/results)
- Write merged .mtz/.cif -> new ProcessConfig::write_merged

output_prefix is set when either is wanted; both off = process for stats only.
Scaling/merge statistics (hence the ISa/R-meas window) are independent of file
writing, so they show in every case. RugnuxCommandLine now emits
--write-process-h5 so a copied command matches the GUI's choice.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

rugnux: honour an absolute -o output prefix ... 228841870d

configure_offline_output still routed the -o prefix through FilePrefix (the
CheckPath guard), so 'rugnux -o /abs/path' threw 'Path cannot start with slash'
before processing even though the writer path already supported it. Use the
trusted setter here too. Completes the offline absolute-path support (the writer
and Rugnux::Run already skip the guard); the .mtz/.cif and _process.h5 now land
at an absolute -o. The --scale path shares this function and writes via
WriteReflections (no FileWriter), so it is covered as well.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: show the space group in the merge-statistics window ... 0ed943dd4d

The de-novo space-group search (SearchSpaceGroup) already ran inside Rugnux but
only its chosen number survived on ProcessResult, and the viewer dropped even
that. Carry the structured SearchSpaceGroupResult (point group + ranked
candidate scores) on ProcessResult and thread it into JFJochMergeStatsWindow,
which now shows a 'Space group' hero card (the final group, searched or fixed)
plus a compact table of candidate groups and their absence scores.

The library no longer renders the search to text (it used to embed it in
merge_statistics_text); rugnux_cli formats it for stdout instead, so the CLI
keeps its text table while the viewer draws a proper table and does not spew it
to stdout. Also surface a user-fixed space group on the result so the card
shows it too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: bundle cuFFT in the Linux .tar.gz like the Windows build ... 936d2efd4a

The Windows package ships cufft64_*.dll next to the viewer so a host with only
an NVIDIA driver (no CUDA toolkit) can run it; the Linux .tar.gz shipped nothing
equivalent and had no rpath, so a -linux-cuda archive needed cuFFT on the system.

Mirror the Windows behaviour ONLY for the self-contained .tar.gz
(JFJOCH_VIEWER_ONLY): install libcufft.so (with its SONAME/version symlink chain)
into bin next to the binary and set INSTALL_RPATH=$ORIGIN so the loader picks the
bundled copy. The .deb/.rpm builds deliberately do NOT bundle it - CUDA there is
centrally managed by the distro's packages, and the .tar.gz is the one with no
package manager, where we want the dependency set really minimal. cuFFT is the
only dynamically-linked CUDA component (cudart and the fast-feedback indexer are
static). macOS has no CUDA so it is excluded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: add a Browse button to the processing output prefix ... 4374babe26

The output prefix was a plain text field; add a 'Browse…' button that opens a
file dialog to pick the output location. The chosen name is treated as a prefix
(the _process.h5 / .mtz / .cif suffixes are appended), so a trailing extension is
dropped, and the result is stored with native separators.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs: changelog for rc.158 viewer/rugnux/packaging changes ... 85cf1b04d1

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

ci: upload the Windows/CUDA viewer installer as a build artifact ... 61eb661eca

The Windows installer was only uploaded on tagged releases, so testing an
arbitrary branch build meant cutting a tag. Also upload the CUDA installer as a
workflow artifact on every run, with a short 3-day retention (it is a throwaway
test build). Only the cuda variant, since that is the one we test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: show the merge resolution range in the stats hero row ... 668855a9e7

The high-resolution limit is now usually set by the automatic CC1/2 logistic
cutoff (when no scaling resolution is given), so it varies per run. Add a
'Resolution [Å]' hero card (low-res .. high-res edge from the overall shell) as
the first card in the merge-statistics window.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: show only the high-resolution limit in the stats hero card ... 54de990100

Drop the low-resolution edge from the Resolution card - only the high-res limit
(the CC1/2-cutoff value) is of interest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

ci: use gitea-upload-artifact@v4 for the Windows/CUDA artifact ... 7da60c3385

Switch to the internal Gitea artifact action (actions/gitea-upload-artifact@v4,
hosted at gitea.psi.ch) instead of actions/upload-artifact@v3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

ci: use actions/upload-artifact@v4 for the Windows/CUDA artifact ... c3468eef6c

The internal Gitea action is referenced as actions/upload-artifact@v4 (no
gitea- prefix), per its own documentation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs: update analysis references to match current code ... e7fbeb527f

Reconcile docs/CPU_DATA_ANALYSIS.md and docs/RUGNUX.md with the current
image_analysis/ and rugnux CLI (verified section-by-section against the code):
unified profile-fit Bragg integration engine, multi-lattice indexing,
azimuthal phi binning, radial parallax/bandwidth profile with sub-pixel
centring, rot3d capture-fraction handling, automatic CC1/2 resolution cutoff,
and the new rugnux options; fix the section numbering and cross-references;
remove the never-implemented French-Wilson and still-partiality descriptions.

Delete the stale in-source design notes (ICE_RING_DETECTION,
BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) and fix the two code comments
that pointed at them; the BraggIntegrationEngine header no longer claims it is
'not yet wired'.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

azint: tilt-correct solid-angle correction; honour imported rot3 in refinement ... acd2025676

The azimuthal-integration solid-angle correction used cos^3(2*theta), where
2*theta is the true scattering angle (from LabCoord, including detector tilt).
The solid angle of a flat pixel actually depends on the incidence angle to the
detector normal, cos(alpha) = det_distance / |detector-frame position|, which is
invariant under detector tilt (rot1/rot2/rot3). Only for an untilted detector do
the two agree. Switch CalcAzIntSolidAngleCorr(x,y) to the tilt-invariant form,
matching PyFAI solidAngleArray and MAX IV azint. Drop the q-only overload (it can
only ever be the untilted approximation and was used only in tests) and move its
test onto the (x,y) form; add a tilt-invariance test.

XtalOptimizer's residual reconstructed each spot's lab position from rot1/rot2
only, hardcoding rot3 = 0, while the rest of the pipeline (and its own spot
selection) used the full PONI rotation. An imported non-zero rot3 was therefore
silently dropped during refinement. Bake rot3 into the residual as a fixed
Rz(-rot3) so refinement stays consistent (no-op when rot3 == 0).

Polarization and azimuthal binning already honoured rot3 via the full PONI
rotation (Phi_rad), validated against PyFAI chi() by the existing rot3 phi tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

harden untrusted-input size handling in TIFF read and raw-TCP frames ... 4b72a89249

Two memory-safety/robustness fixes for input whose size is attacker- or
peer-controlled:

- ReadTIFF: a large IMAGELENGTH could overflow scanline_bytes * lines,
  undersizing the buffer that TIFFReadScanline then writes past. Guard the
  product against overflow before resize and reuse scanline_bytes in the loop.

- Raw-TCP image path: an uncapped header payload_size drove a huge resize()
  that took down the receive thread (wedging the writer-facing acceptor).
  Add JFJOCH_TCP_MAX_PAYLOAD_SIZE and reject oversized frames at the single
  receive-loop choke point on both the pusher and puller sides.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

VERSION: 1.0.0-rc.158 05feb74602

leonarski_f added 1 commit 2026-07-11 13:06:44 +02:00

docs: tilt-invariant solid-angle correction; brief changelog docs entry ... 6c9a22d7f3

Update CPU_DATA_ANALYSIS.md 2.2 to document the azimuthal-integration
solid-angle correction as cos^3(alpha) with alpha the incidence angle to the
detector normal (cos(alpha) = D / sqrt(u^2+v^2+D^2)), tilt-invariant and
matching PyFAI solidAngleArray / MAX IV azint, reducing to cos^3(2*theta)
only for an untilted detector - matching the code change in acd20256.

Shorten the over-detailed rc.158 "Docs" changelog bullet to a brief line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-11 13:13:00 +02:00

Remove upload artifact from Gitea for jfjoch_viewer on Windows (upload doesn't work) efd5f3513a

leonarski_f added 2 commits 2026-07-11 15:20:23 +02:00

viewer: move JFJochHttpReader to the viewer and switch it to libcurl ... 49d078563c

JFJochHttpReader (the viewer's live-broker HTTP client) is the only HTTP client
in the reader library and the only piece that needs TLS + Kerberos. Moving it out
of the always-built JFJochReader keeps the broker/writer free of a TLS/Kerberos
stack; switching it from cpp-httplib to libcurl gives the viewer HTTPS and, later,
GSSAPI/Bearer auth.

- Move reader/JFJochHttpReader.{cpp,h} -> viewer/, compiled into jfjoch_viewer.
- Rewrite the 10 httplib call sites onto one libcurl easy-handle Request() helper
  (GET + PUT, binary/JSON bodies; status/transport handling preserved).
- CMake: FetchContent libcurl, STATIC, HTTP(S)-only, only for viewer builds.
  OS-native backends keep Windows self-contained (Schannel+SSPI); Linux uses
  system OpenSSL + GSSAPI (krb5). Drop httplib from JFJochReader.
- Block httplib OpenSSL auto-link (HTTPLIB_USE_OPENSSL_IF_AVAILABLE OFF) so the
  broker stops pulling in libssl/libcrypto for HTTPS it never serves.
- docker: add krb5 dev headers (libkrb5-dev / krb5-devel) for CURL_USE_GSSAPI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

viewer: accept https:// endpoints and add an http/https selector to the connect dialog ... 007e65cdfe

- Route https:// URLs to the HTTP reader (LoadFile_i scheme check), not only http://.
- Replace the fixed "http://" label in the Open HTTP Connection dialog with a small
  http/https combo box and build the URL from the chosen scheme.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-11 17:53:58 +02:00

viewer: reuse one libcurl handle so the HTTP reader keeps the connection alive ... cdcd5d28ca

JFJochHttpReader::Request previously did curl_easy_init()/curl_easy_cleanup()
around every call, which destroys the handle's connection cache -- so each
request reconnected from scratch (full TCP + TLS handshake, and a fresh
Negotiate/Kerberos round-trip once auth is on).

Keep one persistent easy handle instead: create it lazily on the first request
(i.e. as soon as an address is selected via ReadURL) and reuse it, calling
curl_easy_reset between requests. reset clears the previous request's options
but deliberately keeps the live connection and DNS/TLS-session caches, so a
same-host request reuses the open socket; if that socket has been closed
(idle timeout, server session limit, broker restart) libcurl reconnects and
retries on its own -- no manual reconnect logic needed. The handle is torn down
on disconnect (Close), when a new address is selected (ReadURL), and in the
destructor. Access is serialised by a dedicated leaf mutex, since a single easy
handle must not be used from two threads at once.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 2 commits 2026-07-11 18:47:07 +02:00

Dockerfile: For rocky8 one should NOT run dnf update, because it will like to update cuda to 13.3 4fcbb09771

viewer/curl: clean stale CI build dir + guard against system libcurl shadowing ... 53ffb56aae

Self-hosted runners reuse the workspace Docker volume, so a stale build/ dir
could leave jfjoch_viewer linking a system libcurl instead of the vendored
FetchContent static one (curl configured but never compiled; viewer linked the
OS curl). Add a "Clean previous build" step (rm -rf build) to the two Linux
viewer jobs (build-viewer-tgz, build-rpm), and assert libcurl_static exists
after FetchContent_MakeAvailable(curl) so any future shadowing fails loudly
instead of silently linking the OS curl with the wrong TLS/Kerberos backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-11 18:57:08 +02:00

tools: add rugnux_vs_xds.py — rugnux vs XDS regression table ... 7fd2105f96

Self-contained script (self-bootstrapping venv + gemmi) that runs rugnux
de-novo over a directory of test crystals and tabulates its merging statistics
against the XDS CORRECT.LP reference: number of reflections, observations,
space group (H-M + number), R_meas / CC1/2 overall, R_meas low shell,
CC1/2 high shell, ISa, and rugnux wall-clock time — with a per-crystal
point-group-level space-group match (OK/DIFF).

rugnux stats are read from its mmCIF output (the stable interface, not the
console log); resolution limit and Friedel/anomalous setting are matched to
XDS for a like-for-like comparison. Default output dir rugnux_cmp/ is ignored.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-11 19:04:53 +02:00

viewer/curl: assert CURL::libcurl is the vendored libcurl_static, and log it ... 180f93341a

The CI runner uses a fresh per-task workspace volume (not a reused build dir),
so the earlier rm -rf build was a no-op and curl still wasn't compiled. The
real symptom: on a fresh build the viewer's CURL::libcurl resolves to a system
libcurl (libcurl_static exists but is never compiled/linked). The previous
guard only checked that libcurl_static exists, which it does. Instead print
what CURL::libcurl aliases and hard-fail unless it is the vendored
libcurl_static, so a system-curl shadow surfaces in the configure log instead
of silently shipping a viewer linked against the OS curl.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-11 19:11:48 +02:00

viewer/curl: dump CURL::libcurl type + imported path when it shadows ... 3b02156fba

The CI image has only runtime /usr/lib64/libcurl.so.4 (no libcurl-devel, no
.so symlink, no CMake config, no toolchain/dependency provider), so a standard
find_package(CURL)/-lcurl shadow is impossible -- whatever provides CURL::libcurl
must point at a full path. Expand the guard to print ALIASED_TARGET, TYPE,
IMPORTED_LOCATION and INTERFACE_LINK_LIBRARIES so a single configure run reveals
exactly what CURL::libcurl resolves to before the fatal error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-11 19:17:57 +02:00

viewer/curl: link vendored libcurl_static directly, not the CURL::libcurl alias ... b3b4f5824e

The viewer's CURL::libcurl was resolving to a system libcurl on the CI image
(curl configured but never compiled; viewer linked the OS curl). The alias is
shadowable; the concrete FetchContent target name libcurl_static is not. Link
it directly -- this forces the vendored static curl into the build graph and
pulls its include dirs + CURL_STATICLIB define via the target's usage
requirements. Simplify the top-level guard to just assert libcurl_static exists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-11 21:47:05 +02:00

viewer/curl: keep <curl/curl.h> out of JFJochHttpReader.h (MSVC build fix) ... c37bc2d005

curl.h transitively includes <windows.h>, whose min/max/ERROR/LoadImage
macros clobbered every viewer TU that includes this widely-used header
(std::min/max in gemmi, the TaskState::ERROR enum, the LoadImage() calls,
the Qt/fftw cascade). It even broke JFJochHttpReader.cpp itself, since inside
the header curl.h preceded the reader/common includes that pull in gemmi.

Forward-declare the opaque easy handle (CURL is `typedef void` in libcurl)
so the header no longer needs curl.h, and include curl.h only in the .cpp,
last, after all std/project headers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

leonarski_f added 3 commits 2026-07-11 23:03:55 +02:00

rugnux: parallelize two-pass rotation first phase ... b1cd49c6b5

The first pass of two-pass rotation indexing was the slow "getting there" phase.
Profiling RunIndexing showed the cost is the serial Ceres candidate refinement
(up to 4 FFT candidate cells x 2 XtalOptimizer solves each), not spot finding or
the cuFFT kernel.

- XtalOptimizer: add a num_threads arg (default 1) so a caller running a few
  refinements concurrently can give each several cores.
- RotationIndexer::RunIndexing: seed the candidates serially, run their up-to-8
  XtalOptimizer solves concurrently (4 Ceres threads each), select serially in
  candidate order (identical result). ~10-13x on the candidate loop.
- Rugnux: feed both first-pass schemes (spread + wedge) serially, then run their
  RunIndexing() passes in parallel - overlapping one scheme's cuFFT with the
  other's Ceres. Size the FFT indexer pool to 2 for the rotation path (it fires
  only twice) instead of the default 4, avoiding wasted cuFFT-plan init.

First-pass wall 2.1-6.5x faster across a 12-crystal battery; whole lysoC run
17.7->10.5s. All 12 crystals bit-identical (scheme, validation count, space group).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: move rugnux_cli.cpp from tools/ to rugnux/ ... f737424bdf

The offline analysis CLI belongs next to its Rugnux library. Move the source and
the ADD_EXECUTABLE(rugnux ...) target (plus the Windows wingetopt link) into
rugnux/CMakeLists.txt; the ../reader, ../common, ../image_analysis includes are
unchanged (both dirs sit one level under the root) and ../rugnux/Rugnux.h becomes
a local Rugnux.h. rugnux/ is added before tools/ in every build config, so
viewer-only and Windows packages still get the CLI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: don't reuse absent spots on plain DECTRIS datasets ... 0dfaf0a58a

reuse_rotation_spots defaults on, so the two-pass first phase read stored spots
from /entry/MX. A standard DECTRIS dataset has no spot-finding results, so every
frame came back with zero spots and indexing failed ("Two-pass rotation indexing
failed") instead of falling back to finding spots.

Add HasStoredSpots() to the reader (HDF5MetadataSource::HasSpots): spots exist if
/entry/MX/nPeaks is present in the master (integrated _process.h5) or the per-image
data file (VDS/legacy) - the same master-first-then-source path ReadSpots uses.
Rugnux reuses stored spots only when they exist, otherwise it runs spot finding
(and logs it); --redo-rotation-spots already forced finding.

Verified: a true no-/entry/MX rotation dataset fails on the old code and indexes
(P21) on the new; all jungfraujoch files (spots in master or in data files) still
reuse unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 1 commit 2026-07-11 23:38:58 +02:00

docs: changelog for rc.158 rugnux/viewer/robustness changes ... 27802a30a3

Faster parallel two-pass rotation indexing, rotation indexing on spot-less DECTRIS
datasets, viewer https connection, and hardened TIFF/raw-TCP input handling.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 3 commits 2026-07-12 10:18:33 +02:00

rugnux: fix resolution-cutoff over-extension on low-res crystals ... a854523fee

The CC1/2-logistic auto resolution cutoff extended the fall-off crossing
by "one shell" = s_range/10, where s_range spans to the detector edge. On
a low-res crystal read out by a high-res-configured detector (integration
runs to the ~1.4 A edge), that range is dominated by high-res noise, so
one "shell" is a huge step in s that overshoots the true fall-off by ~1 A:
Benas_3's CC1/2=0.30 crossing at 3.77 A was pushed to 2.98 A, Benas_7's
4.04 A to 2.97 A. The logistic fit itself is accurate; only the extension
was wrong.

Anchor the extension to the range actually kept and reported (low-res
plateau -> the crossing), (s_cross - s_lo)/10, instead of the detector-edge
range. Benas_3 -> 3.60 A, Benas_7 -> 3.85 A. The change is monotonic in
(crossing - edge) and always coarser (never adds noise): zero change for
crystals diffracting to the edge, negligible for well-diffracting ones,
and only meaningful where the detector over-reaches the diffraction limit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: exclude merge-rejected outliers from the error-model fit ... dee679380b

The rotation error-model (a,b) fit built its sample pool from all usable
fulls, while the merge separately dropped symmetry outliers (> Nsigma from
the per-reflection median). So the fitted sigmas were calibrated against a
dirtier pool than the reflections that actually entered the merge.

Refactor the intensity-binned median fit into a `fit_ab` lambda and re-run
it once on the misfit-free pool (samples with dev2 <= reject_nsigma^2 *
model variance), matching the merge's own rejection. It operates on the
shared `samples` array, so the CPU and GPU error-model paths stay
bit-identical.

Neutral on lyso_ref (the median-binned fit is already robust to the few
outliers) - a correctness fix that matters more on dirtier data. Gated on
reject_outliers (default 6 sigma for rot3d).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: optional decay + absorption correction surfaces for rotation scaling ... bab52a8077

After scale-fulls, RotationScaleMerge can now fit two optional
multiplicative corrections on the combined fulls (both off by default,
rotation only):

- -B (decay): a global Debye-Waller relative-B rate for radiation damage,
  ln(Iref/Iobs) = 2 (dB/dn)(n - nbar) s^2, folded into the per-full scale.
  Engages only above a physical floor (total relative-B > 2 A^2), so it is
  a no-op on low-dose data. (-B was previously rejected for rotation.)
- --absorption[=iter]: a smooth factor over the diffracted-beam direction
  in the goniometer frame (predicted detector position de-rotated by the
  spindle, so a crystal-frame direction is sampled at many angles). Wires
  the existing ScalingSettings::absorption_iter scaffold. Negligible at
  hard X-rays; matters at low energy. On lyso_ref it is ISa-neutral (+0.5)
  but drops R-free 0.219 -> 0.212 against the deposited model: a smooth
  absorption error cancels among symmetry mates (small ISa effect) yet
  still biases the intensities (real R-free effect).

Both are CROSS-VALIDATED: the surface is fit on even frames and applied
only if it improves the held-out odd-frame equivalent agreement by a clear
margin (and vice versa), so a fit to noise where the systematic is absent
does not generalize and is discarded - the correction never adds scatter.
Battery: EcwtCQ066S +2.4 ISa, lyso_ref R-free win, others neutral.

Also fixes a latent bug this exposed: the GPU->host corr download before
the CPU 3D-combine was gated on --dump-observations only, so any other
forced-CPU-combine path (the new -B/--absorption) read a stale host corr
and mis-scaled every full. The download condition now matches the
CPU-combine condition.

Predicted detector position (px, py) is carried on the combined fulls for
the absorption surface (CPU combine only; unused by the GPU SoA path).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f added 10 commits 2026-07-12 17:19:41 +02:00

rugnux: run decay + absorption surfaces on the GPU and enable by default ... fbebc0d56e

The decay/absorption correction surfaces previously forced the CPU 3D
combine (absorption needs each full's predicted detector position, which
the GPU SoA did not carry), so they could not be default-on without losing
the fast GPU path. Put them on the GPU:

- carry px/py (peak partial) through the GPU combine and download them with
  the fulls (GetFullsPxPy);
- add SetFullsCorr to re-upload the host-corrected corr to the resident
  fulls before the merge.

The surfaces themselves stay as cheap host fits on the downloaded fulls;
only px/py download + corr upload are added (~two O(n_fulls) transfers).
Measured +2% wall-time on lyso_ref (9.7 -> 9.9 s), and the GPU-combine and
CPU-combine (--dump-observations) paths give identical results.

Enable by default via a single master toggle ScalingSettings::
CorrectionSurfaces (default true; decoupled from the stills-only -B, which
is rejected for rotation again). Both surfaces are cross-validated, so they
no-op where their systematic is absent - safe to leave on. Off-switch wired
through the rugnux CLI (--no-scaling-corrections; --absorption removed) and
the viewer (a "Correction surfaces" checkbox). Documented in docs/RUGNUX.md.

Battery (24 rotation crystals), default-on vs --no-scaling-corrections:
EcwtCQ066S +2.4 ISa, lyso_ref +0.5 (and R-free 0.194 -> 0.185, better than
XDS); every other crystal within run-to-run noise; corrections skip (CV /
dB-floor) wherever there is no signal. lyso_ref anomalous S/Cl peaks are
already ~45% above XDS's - the ISa gap is metric convention, not data.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rugnux: keep merohedral twins in true symmetry, robustify twinning stats ... 4aab8078d6

SearchSpaceGroup: drop the log10(chi2_ref) widening of the point-group
chi^2-ratio bound and tighten max_merge_chi2_ratio 2.0 -> 1.85. The
variance-floor fix removed the error-model miscalibration the widening
compensated for, so the widening now only let a partial merohedral twin
through (Ins_H_2 R3->R32 twin 2-fold: ratio 2.01). Every genuine high
symmetry across the rotation-test battery stays within ~1.7x (worst real
case Thau P41212 at 1.71), so 1.85 keeps R3 in its true lower symmetry.

TwinningAnalysis: make the <I^2>/<I>^2 second moment robust - skip
noise-only shells (<I/sigma> < 1) and reject Wilson outliers (E^2 > 8)
with one shell-mean re-iteration, so a single strong reflection in a
collapsed-mean shell no longer dominates the moment. Add
MerohedralTwinningPossible: in a holohedral Laue class (4/mmm, 6/mmm,
m-3m, rhombohedral -3m) no twin law exists, so a low <|L|> there is a
statistical artefact and is no longer flagged as twinning.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

writer: write refined detector geometry once at the end, incl. translation ... 6c80dbdc32

The refined beam centre / detector rotations were written from the
StartMessage at file open and then partially overwritten in Finalize -
but the overwrite updated beam_center_x/y and rot1/2/3 and NOT the
transformations/translation vector, which is what actually encodes the
beam centre in the NXmx geometry chain read by XDS/DIALS/dxtbx. The
reprocessed _process.h5 therefore had an internally inconsistent geometry
(refined rotations combined with the stale, unrefined beam centre).

The master file is never read before Finalize, so write the whole
refinable geometry (beam_center_x/y + translation + rot1/2/3) once there,
from the refined values when present and the StartMessage otherwise. This
drops the open-time write plus in-place overwrite and, unlike the old
path, keeps translation consistent with the refined beam centre.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

broker: make the writer-error status message self-explanatory ... d033b378ae

When the writer fails mid-run the status carried only the raw writer_err
string. Prefix it with "Writer error, written data may be incomplete: "
so the operator sees the dataset may be truncated, not just a bare error
string. Priority order is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

frontend: guard the TIFF value colour span against zero/negative ... 1aac0495e1

`span = hi - lo || 1` only replaced an exactly-zero span, so Min>Max (or a
cleared Min/Max field coercing to 0 via Number("")) gave a negative span
and rendered the pedestal image with an inverted/collapsed colour scale.
Use `hi - lo > 0 ? hi - lo : 1` so a non-positive span falls back to 1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

viewer: give the libcurl HTTP reader a 5 s connect/transfer timeout ... d90888b5f9

The single reused easy handle set no timeout, so a stalled or black-holed
broker blocked curl_easy_perform indefinitely - wedging the reader thread
and any Close()/ResetConnection() waiting on curl_mutex (the previous
per-call httplib client timed out and recovered). Set CURLOPT_CONNECTTIMEOUT
and CURLOPT_TIMEOUT to 5 s; all viewer requests are small and go to a
nearby broker.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

viewer: drop the cached pixel mask when switching HTTP source ... 8ecfa780eb

cached_pixel_mask is keyed only on arm_date and was cleared only in
Close(); ReadURL() reused it, so switching to a source whose start message
has an empty or duplicate arm_date silently kept the previous source's
mask (wrong pixels masked in the live preview and fed to spot-finding /
azimuthal integration). Clear it in ReadURL() too, as Close() does.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

scale_merge: hoist the resolution-cutoff apply/trim into one shared helper ... 4d20298523

The "resolve effective_d_min (manual override, else the auto CC1/2 cutoff
unless for_search) then erase merged reflections beyond it" block was
copy-pasted in three places - RotationScaleMerge::MergeAndStats, the stills
merge in Rugnux.cpp and the offline --scale path in rugnux_cli.cpp - and had
already started to diverge (the CLI copy had dropped the for_search guard and
the "no cut" log line). Move it into ResolutionCutoff::ApplyResolutionCutoff
so there is one implementation.

No behaviour change to the cutoff policy: the two full-analysis paths are
identical to before, and the CLI path keeps its semantics (for_search is
always false when re-scaling a stored _process.h5) - it only gains the same
"keeping the full resolution range" info line the other two already logged.
RotationScaleMerge still recomputes its own d_min/d_max span after the trim.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: note the symmetry/twinning and refined-geometry fixes ... 523301bc6b

CHANGELOG rc.158: add the de-novo symmetry robustness fix (don't
over-promote a merohedral twin, robust second moment, holohedral no-twin
guard), the writer refined-beam-centre/translation fix, and the viewer
HTTP timeout + pixel-mask-on-source-switch fixes.

CPU_DATA_ANALYSIS: the space-group self-consistency test keeps a
merohedral twin in its true lower symmetry; the twinning second moment is
computed per shell with noise-shell/Wilson-outlier rejection, and twinning
is only flagged in Laue classes where a twin law can exist.

HDF5: state that the detector transformations carry the refined geometry
(refined beam centre folds into translation) in a rugnux _process.h5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: drop specific sample names/results from the rc.157 changelog note ... 575d5d09e4

Replace the Benas-crystal example (and the reference-battery validation
claim) in the de-novo space-group entry with a generic description; this
is general documentation, not a record of which datasets were tested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f merged commit 451310f43d into main 2026-07-12 19:42:29 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#68