The x86/ARM split asked for on macOS already exists: BitShuffleBlock.h (fbc507839) sends
the per-block transform to bitshuffle_hperf everywhere except aarch64 with NEON, where it
uses the classic bitshuffle's NEON path, and both call sites go through it. Nothing new is
needed for Apple Silicon - clang --target=arm64-apple-macos defines __aarch64__ and
__ARM_NEON exactly as aarch64 Linux does, both vendored files compile to arm64 Mach-O
objects, the seam resolves to bshuf_(un)trans_bit_elem and bshuf_using_NEON() is 1. The
header now says why the switch is a preprocessor one: a universal build compiles it once
per architecture, which a CMake-time answer cannot follow.
What had never happened is the NEON code actually running. It has now, under
qemu-aarch64 in the project's cross image: 504 blocks (elem 1/2/4/8, 8 elements up to the
128 kB block, odd multiples of 8, random and detector-like data) plus 54 whole-buffer
bitshuffle/LZ4 streams with element counts that are no multiple of 8 or of the block. The
output is byte-identical (same md5) across aarch64 NEON, aarch64 without NEON (seam falls
back to hperf's portable code), x86 hperf AVX2 via ifunc, hperf SSE2, hperf portable
fallback, and classic SSE2 / AVX2 / scalar; every implementation decodes every other.
Throughput, one thread on a loaded Zen 3, 16/32-bit, GB/s encode/decode: hperf AVX2
~9-12/~10-11, classic AVX2 ~8/~5.5-8, classic SSE2 ~4-5/~4.5-5.5, hperf portable fallback
auto-vectorised ~2.8/~5, not vectorised ~1/~1.8. Classic 128-bit SIMD beating hperf's
fallback is the x86 stand-in for the choice the seam makes on ARM; no ARM hardware was
available, so the NEON-vs-fallback ranking on a real core is still unmeasured.
The one thing that would have failed on aarch64 is the Bshuf_SSE test, which required
SSE2 outright. It now accepts NEON as well, which also makes it the check that a Mac or
DGX Spark build did not end up on the scalar path.
Not built: no CMake configure or jfjoch_test build was run on this machine (busy); the
test expression was compiled and run standalone on x86 only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 5957b15493dfd0cc8a18fb780a18c2573a6eccde)
None of this has been built on a Mac - there is none yet. It is the list a read-only audit of the
viewer/rugnux subtree produced, plus a serial -fsyntax-only pass of every reachable .cpp with
clang 16 + libc++ on Linux, which found exactly one error (the first item).
- JFJochDatasetInfoChartView: std::vector<fftwf_complex> does not compile with libc++, whose
construct_at refuses an array element type (float[2]). Use std::vector<std::complex<float>> and
the reinterpret_cast every other FFTW call site already uses.
- libcurl: GSSAPI off on Linux, and neither TLS nor GSSAPI on macOS. The viewer never sets
CURLOPT_HTTPAUTH, so Negotiate was dead weight that cost a krb5-devel build dependency; on macOS
curl's FindGSS refuses the system Heimdal outright, and with Secure Transport gone from curl
(8.15) TLS would mean a Homebrew OpenSSL - the only host library a Mac build would need.
Linux keeps OpenSSL. CURL_USE_GSSAPI is forced OFF rather than left unset so an existing build
tree drops its cached ON.
- libjpeg-turbo ExternalProject: CMAKE_SYSTEM_NAME/PROCESSOR were forwarded unconditionally, which
puts even a native sub-build into cross-compiling mode, and CMAKE_OSX_ARCHITECTURES / SYSROOT /
DEPLOYMENT_TARGET were not forwarded at all. Now the same rule the zlib-ng sub-build follows.
- ShadowAccumulatorGPU.cu was added on the JFJOCH_USE_CUDA option (default ON) instead of
JFJOCH_CUDA_AVAILABLE like every other .cu, so a machine without nvcc got a CUDA source in a
target with no CUDA language.
- CMAKE_OSX_DEPLOYMENT_TARGET defaults to 12.0 (overridable). Left unset, CMake takes the build
machine's OS version and the .dmg starts nowhere older.
- Standard headers that were only arriving transitively (<chrono>, <cmath>, <limits>, <cstring>,
<atomic>, <thread>, <string>); newer libc++ releases keep removing such transitive includes.
Checked: the seven changed sources pass clang 16 + libc++ -fsyntax-only. The CMake changes are
not configured or built.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015eAE2K7i5JGDwgwifiCfuA
(cherry picked from commit 5a7282759a)
* Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases.
* rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures.
* rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination.
* rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands.
* rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings.
* The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`).
* `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed.
* Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself.
Reviewed-on: #79
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
* `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands.
* `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion.
* Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants.
* `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing.
* A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed.
* `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing.
* Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences.
* The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to.
* The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after.
* The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution.
* `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have.
* Twinning is no longer reported when the L-test contradicts it.
* The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's.
* `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots.
* The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area.
Reviewed-on: #76
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.
* **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster.
* **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory.
* **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again.
**Breaking change to the rugnux command line:**
* `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.
* `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride.
**Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional:
* `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve.
* `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.
**Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional:
* The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more.
* `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.**
Reviewed-on: #71
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144.
* jfjoch_broker: Improve azimuthal integration (add <I^2> calculation)
* jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated)
* jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin
Reviewed-on: #58
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4)
* jfjoch_viewer: Improve image render performance
Reviewed-on: #44
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.
* jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long
* jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata
* XDS plugin: Add generation of Jungfraujoch plugin for XDS
* CI: Add tests with XDS and DIALS (`xia2.ssx`)
Reviewed-on: #43
leonarski_f