Commit Graph
8 Commits
Author SHA1 Message Date
leonarski_fandClaude Opus 5 fbc5078399 compression: use the NEON bitshuffle on aarch64
bitshuffle_hperf is an x86-only implementation - its entire vector body sits behind
__i386__/__x86_64__, so on aarch64 every entry point compiles down to the scalar
fallback. Measured against its own SIMD path that costs 8.3x on encode and 3.8x on
decode, and it is the transform behind every compressed image the writer produces and
every one the reader, preview and XDS plugin take apart again.

The classic bitshuffle vendored beside it does have an aarch64 NEON path, and is already
compiled into the same target, so this costs nothing new. BitShuffleBlock.h picks
bshuf_trans_bit_elem/bshuf_untrans_bit_elem there and keeps bitshuf_encode_block /
bitshuf_decode_block everywhere else, where hperf is about twice classic SSE2 and remains
the better choice. The expected aarch64 gain is ~2.5x encode / ~1.7x decode: classic NEON
is 128-bit and carries an extra pass, so it recovers part of the gap rather than all of
it. The condition mirrors USEARMNEON in bitshuffle_core.c exactly, because with NEON off
the classic scalar path is slower than hperf's and must not be selected.

Swapping implementations is only safe while the two agree bit for bit - otherwise an ARM
build would write files an x86 build could not read. They do: verified byte-identical
output and mutual cross-decoding for elem_size 1/2/4/8 over block sizes from 8 to 65536
elements. Both are always compiled in, so the new test holds them to it on every
architecture, not just the one that would notice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 21:27:27 +02:00
leonarski_f d6389e12da v1.0.0-rc.156 (#66)
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 15m31s
Build Packages / build:viewer-tgz:cpu (push) Successful in 5m46s
Build Packages / build:viewer-tgz:cuda (push) Successful in 6m9s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 9m25s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 10m21s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 9m41s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 9m18s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 10m26s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 11m33s
Build Packages / build:rpm (rocky8) (push) Successful in 10m32s
Build Packages / build:rpm (rocky9) (push) Successful in 12m23s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 10m12s
Build Packages / DIALS test (push) Successful in 12m6s
Build Packages / XDS test (durin plugin) (push) Successful in 8m15s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 7m12s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m35s
Build Packages / Generate python client (push) Successful in 27s
Build Packages / Build documentation (push) Successful in 54s
Build Packages / Create release (push) Skipped
Build Packages / build:windows:cuda (push) Successful in 12m37s
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output.
* jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling.
* Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool.
* jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection.
* jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata.
* CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-07-03 19:18:56 +02:00
leonarski_f 4a852b4d6b v1.0.0-rc.134 (#43)
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 12m57s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 13m4s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 11m18s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 13m12s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 13m51s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 13m59s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 10m45s
Build Packages / build:rpm (rocky8) (push) Successful in 12m29s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 12m2s
Build Packages / Generate python client (push) Successful in 24s
Build Packages / XDS test (durin plugin) (push) Successful in 9m50s
Build Packages / Create release (push) Has been skipped
Build Packages / build:rpm (rocky9) (push) Successful in 14m15s
Build Packages / Build documentation (push) Successful in 1m6s
Build Packages / DIALS test (push) Successful in 13m10s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 6m45s
Build Packages / XDS test (neggia plugin) (push) Successful in 5m58s
Build Packages / Unit tests (push) Successful in 1h20m42s
This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132.

* jfjoch_broker: Add better locking for detector object - should help, when detector initialization takes too long
* jfjoch_writer: Enable writing single, integrated HDF5 file with both data and metadata
* XDS plugin: Add generation of Jungfraujoch plugin for XDS
* CI: Add tests with XDS and DIALS (`xia2.ssx`)

Reviewed-on: #43
2026-04-09 13:30:47 +02:00
leonarski_f 91fd44bff7 Improve release/versioning of Jungfraujoch repository 2024-05-15 11:29:01 +02:00
leonarski_f 64dbd2291e Further enhacements 2024-04-29 14:13:35 +02:00
leonarski_f 767dc8e1b3 JFJochCompressor: Use Kal's bitshuffle compressor 2023-08-04 16:03:45 +02:00
leonarski_f 9144058f0c JFJochCompressor: Use Kal's bitshuffle compressor 2023-08-04 15:47:32 +02:00
leonarski_f 1757d42182 Initial commit
Signed-off-by: Filip Leonarski <filip.leonarski@psi.ch>
2023-04-06 11:17:59 +02:00