From 76e97b16da984907aee0c792eba37e3fa3b97d90 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 19 Sep 2026 12:55:13 +0200 Subject: [PATCH] compression: verify the bitshuffle seam for Apple Silicon; SIMD test knows NEON The x86/ARM split asked for on macOS already exists: BitShuffleBlock.h (fbc507839) sends the per-block transform to bitshuffle_hperf everywhere except aarch64 with NEON, where it uses the classic bitshuffle's NEON path, and both call sites go through it. Nothing new is needed for Apple Silicon - clang --target=arm64-apple-macos defines __aarch64__ and __ARM_NEON exactly as aarch64 Linux does, both vendored files compile to arm64 Mach-O objects, the seam resolves to bshuf_(un)trans_bit_elem and bshuf_using_NEON() is 1. The header now says why the switch is a preprocessor one: a universal build compiles it once per architecture, which a CMake-time answer cannot follow. What had never happened is the NEON code actually running. It has now, under qemu-aarch64 in the project's cross image: 504 blocks (elem 1/2/4/8, 8 elements up to the 128 kB block, odd multiples of 8, random and detector-like data) plus 54 whole-buffer bitshuffle/LZ4 streams with element counts that are no multiple of 8 or of the block. The output is byte-identical (same md5) across aarch64 NEON, aarch64 without NEON (seam falls back to hperf's portable code), x86 hperf AVX2 via ifunc, hperf SSE2, hperf portable fallback, and classic SSE2 / AVX2 / scalar; every implementation decodes every other. Throughput, one thread on a loaded Zen 3, 16/32-bit, GB/s encode/decode: hperf AVX2 ~9-12/~10-11, classic AVX2 ~8/~5.5-8, classic SSE2 ~4-5/~4.5-5.5, hperf portable fallback auto-vectorised ~2.8/~5, not vectorised ~1/~1.8. Classic 128-bit SIMD beating hperf's fallback is the x86 stand-in for the choice the seam makes on ARM; no ARM hardware was available, so the NEON-vs-fallback ranking on a real core is still unmeasured. The one thing that would have failed on aarch64 is the Bshuf_SSE test, which required SSE2 outright. It now accepts NEON as well, which also makes it the check that a Mac or DGX Spark build did not end up on the scalar path. Not built: no CMake configure or jfjoch_test build was run on this machine (busy); the test expression was compiled and run standalone on x86 only. Co-Authored-By: Claude Fable 5.1 (cherry picked from commit 5957b15493dfd0cc8a18fb780a18c2573a6eccde) --- compression/BitShuffleBlock.h | 3 +++ tests/FrameTransformationTest.cpp | 6 ++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/compression/BitShuffleBlock.h b/compression/BitShuffleBlock.h index b740a4f6a..a4a73f76a 100644 --- a/compression/BitShuffleBlock.h +++ b/compression/BitShuffleBlock.h @@ -18,6 +18,9 @@ // // The condition mirrors USEARMNEON in bitshuffle_core.c exactly. With NEON off the classic path // falls back to a scalar of its own that is slower than hperf's, so it must not be selected then. +// It is a preprocessor test rather than a CMake one on purpose: Apple Silicon defines the same two +// macros as aarch64 Linux, and a macOS universal build compiles this header once per architecture, +// which a single configure-time answer could not follow. #if (defined(__ARM_NEON__) || (__ARM_NEON)) && defined(__aarch64__) // The classic entry points allocate their own block-sized scratch, so the caller's goes unused. diff --git a/tests/FrameTransformationTest.cpp b/tests/FrameTransformationTest.cpp index 22f997521..3abebda59 100644 --- a/tests/FrameTransformationTest.cpp +++ b/tests/FrameTransformationTest.cpp @@ -22,8 +22,10 @@ inline uint64_t read_be64(const void *ptr) { return __builtin_bswap64(ptr64[0]); } -TEST_CASE("Bshuf_SSE", "[bitshuffle]") { - REQUIRE (bshuf_using_SSE2() == 1); +// The classic bitshuffle runs behind the HDF5 filter everywhere, and on aarch64 it is also the one +// BitShuffleBlock.h picks, so a build that left it on its scalar path must not go unnoticed. +TEST_CASE("Bshuf_SIMD", "[bitshuffle]") { + REQUIRE ((bshuf_using_SSE2() == 1 || bshuf_using_NEON() == 1)); } TEST_CASE("FrameTransformation_Raw_NoCompression" ,"") {