olex2: bundle image libs, run the cctbx first start at build time

- sysroot += libwebp, libdeflate, libtiff 4.6 (+libjpeg-turbo, jbigkit-libs,
  liblerc): from a NEEDED scan of every ELF object in the zip these are the
  libs RHEL8/9 lack at the required version (PIL _imaging, vendor
  libtiff.so.5, GUI binary libtiff.so.6).
- install step 9: run olex2c once as the owner so Olex2 writes
  cctbx/cctbx_build/libtbx_env into the shared prefix (its first-start cold
  start; users cannot write there). The upstream fable/sdrive.f error on
  that run is expected and documented: libtbx_env is pickled before it.
- install step 10: second start + refine of the sucrose sample; die if
  cctbx is still reported broken, warn on other errors.
- launcher: PYTHONNOUSERSITE=1.

olex2/README.md

@@ -77,16 +77,24 @@ A newer glibc can **not** be dropped in via `LD_LIBRARY_PATH`: the dynamic
 loader path (`/lib64/ld-linux-x86-64.so.2`) is hard-wired into each
 executable and must match its libc. So the recipe:
 
-- ships glibc 2.39 + GCC 14 `libstdc++`/`libgcc_s` + readline 8.2 under
-  `$PREFIX/sysroot` (frozen Fedora 40 rpms from kojipkgs, Fedora's
-  permanent build archive; same glibc generation as the upstream build
-  host). readline because the binaries need `libreadline.so.8`: RHEL8 has
-  `.so.7`, RHEL9 `.so.8`, the vendor `lib/` only `.so.6` — bundled, olex2c
-  needs just `libtinfo.so.6` (ncurses-libs) from the host. glibc, libstdc++
-  and readline unpack to `usr/lib64`, libgcc to `/lib64` (Fedora relies on
-  the filesystem package's `lib64 -> usr/lib64` symlink, which we do not
-  extract), so `configure` merges `lib64` into `usr/lib64` and recreates
-  the symlink — one lib dir for the RPATH;
+- ships under `$PREFIX/sysroot` (frozen Fedora 40 rpms from kojipkgs,
+  Fedora's permanent build archive; same glibc generation as the upstream
+  build host):
+
+  | package | why |
+  |---|---|
+  | glibc 2.39, libstdc++/libgcc 14.2 | GLIBC_2.38 / GLIBCXX_3.4.32 |
+  | readline 8.2 | `libreadline.so.8`: RHEL8 has .so.7, vendor `lib/` .so.6; host needs only `libtinfo.so.6` |
+  | libwebp 1.3.2 | `libwebp.so.7` for PIL `_imaging` and libtiff (RHEL: only if the libwebp package is installed) |
+  | libdeflate 1.19 | needed by the vendor `libtiff.so.5`, absent on RHEL8 |
+  | libtiff 4.6 (+ libjpeg-turbo, jbigkit-libs, liblerc) | `libtiff.so.6` for the **GUI** binary; RHEL8/9 ship .so.5 |
+
+  The list comes from a scan of the `NEEDED` entries of every ELF object in
+  the zip minus what the zip itself carries. Most packages unpack to
+  `usr/lib64`, libgcc to `/lib64` (Fedora relies on the filesystem
+  package's `lib64 -> usr/lib64` symlink, which we do not extract), so
+  `configure` merges `lib64` into `usr/lib64` and recreates the symlink —
+  one lib dir for the RPATH;
 - runs `patchelf --set-interpreter <sysroot ld.so> --force-rpath --set-rpath
   <sysroot libdir>:$PREFIX/lib:$PREFIX/ilib:$PREFIX/cctbx/cctbx_build/lib:$PREFIX:$ORIGIN`
   on every dynamic executable (`hart`, `hart_mpi` are static and skipped).
@@ -105,9 +113,34 @@ Rules that keep this safe:
 - the originals are untouched inside the cached zip; a rebuild re-patches
   from scratch.
 
+### cctbx first run happens at build time
+
+On its first start Olex2 regenerates `cctbx/cctbx_build/libtbx_env` and
+the libtbx dispatchers by running cctbx's `configure.py` *into BaseDir*
+(`util/pyUtil/PyToolLib/path_utils.py`, `cold_start`). On a per-user
+install that is invisible; on a shared read-only prefix it would fail for
+every user. So `install` runs `olex2c` once as the owner (step 9, isolated
+`HOME`), checks `libtbx_env` exists, then starts it a second time and
+refines the shipped sucrose sample (step 10), dying if cctbx is still
+reported broken. Expect in the first-run log, verbatim from upstream:
+
+```
+Using fable to convert scitbx/lbfgs.f
+There is a problem with the cctbx: [Errno 2] No such file or directory: '.../scitbx/lbfgs/sdrive.f'
+```
+
+That is an upstream quirk: the vendor ships SCons in its site-packages, so
+libtbx believes it can build and the scitbx refresh wants Fortran sources
+that are not shipped. It happens *after* `libtbx_env` is pickled
+(`env_config.py`), which is why the second start is clean. Every Linux
+user of the vendor zip goes through the same first start.
+
 Unverified until the first RHEL9 run: NSS lookups through the host's
-`libnss_sss` under the newer libc (expected fine, NSS ABI is stable), and
-`hart_mpi` against the host MPI.
+`libnss_sss` under the newer libc (expected fine, NSS ABI is stable),
+`hart_mpi` against the host MPI, and NoSpherA2 (it references Intel MKL
+kernel libraries `libmkl_*.so.3` that are neither shipped nor on the
+hosts; whether they are hard requirements or optional dlopen targets is
+untested).
 
 ### Why the binary archive and not a source build
 
@@ -164,10 +197,14 @@ $OLEX2_ROOT/sysroot/usr/lib64/ld-linux-x86-64.so.2 --list $OLEX2_ROOT/olex2c | g
 ```
 
 `libc.so.6`/`libstdc++.so.6` must resolve under `$OLEX2_ROOT/sysroot`; every `not
-found` line is a host package to `dnf install` (expected candidates on a
-minimal RHEL9: `gtk2`, `mesa-libGL`, `mesa-libGLU`, `libSM`, `libXxf86vm`,
-`libxkbcommon`). Plain `ldd` on the raw binary also works after the patch
-(it honours the new interpreter).
+found` line is a host package to `dnf install`. From the NEEDED scan, the
+host must provide (all standard RHEL8/9 packages): `gtk2`, `gdk-pixbuf2`,
+`pango`, `cairo`, `glib2`, `fontconfig`, `freetype`, `libX11`, `libSM`,
+`libxcb`, `libxkbcommon`, `libwayland-client`, `ncurses-libs`
+(libtinfo), `libgomp` (cctbx refinement!), `libgfortran` + `libquadmath`
+(openblas/scipy), `zlib`, `xz-libs`, `libzstd`, `libuuid`, `libxcrypt`;
+GUI additionally `mesa-libGL`, `mesa-libGLU`. Plain `ldd` on the raw
+binary also works after the patch (it honours the new interpreter).
 
 ## Verifying the install
 
@@ -215,11 +252,13 @@ not to edit the hash:
 2. check the new binaries' glibc need still fits the bundled 2.39:
    `unzip -p olex2-linux64.zip olex2/olex2c | strings | grep -o 'GLIBC_2\.[0-9]*' | sort -uV | tail -1`
    (and `GLIBCXX_3\.4\.[0-9]*` ≤ 3.4.33). If upstream moved past that, bump
-   the four rpms (glibc, libstdc++, libgcc, readline) in `urls:`/`shasums:`
-   to a newer frozen Fedora release (kojipkgs
-   `packages/glibc/<ver>/<rel>/x86_64/`, `packages/gcc/...`,
-   `packages/readline/...`) — they must come from the same release so
-   readline's glibc requirement is met;
+   all sysroot rpms in `urls:`/`shasums:` to a newer frozen Fedora release
+   — they must come from the same release so their mutual glibc
+   requirements are met. kojipkgs has no browsable index (bot gate) but
+   serves any `packages/<src>/<ver>/<rel>.fcNN/x86_64/<nvr>.x86_64.rpm`;
+   find the NVR via the Koji XML-RPC hub (`listTagged`, tag `fNN`,
+   `latest=1`) and note that the source name can differ from the lib name
+   (`liblerc`, `jbigkit` → `jbigkit-libs`);
 3. add `1.5-7800` under `versions:` (`relstage: unstable`) and
    `olex2-1.5-7800-linux64.zip: <sha256>` under `shasums:` — the `urls:`
    entry needs no change (`${V_PKG}` = series dir, `${V}` = cache name);

olex2/build

@@ -33,6 +33,12 @@ OLEX2_SYSROOT_RPMS=(
     libstdc++-14.2.1-3.fc40.x86_64.rpm
     libgcc-14.2.1-3.fc40.x86_64.rpm
     readline-8.2-8.fc40.x86_64.rpm      # libreadline.so.8: RHEL8 has .so.7, vendor lib/ .so.6
+    libwebp-1.3.2-5.fc40.x86_64.rpm     # libwebp.so.7: PIL + libtiff
+    libdeflate-1.19-2.fc40.x86_64.rpm   # libdeflate.so.0: vendor libtiff.so.5 (absent on RHEL8)
+    libtiff-4.6.0-2.fc40.x86_64.rpm     # libtiff.so.6: GUI binary (RHEL ships .so.5) + its deps:
+    libjpeg-turbo-3.0.2-1.fc40.x86_64.rpm
+    jbigkit-libs-2.1-29.fc40.x86_64.rpm
+    liblerc-4.0.0-6.fc40.x86_64.rpm
 )
 OLEX2_PATCHELF_TGZ="patchelf-0.18.0-x86_64.tar.gz"
 # ELF executables in the vendor tree that get the bundled loader (static ones,
@@ -192,8 +198,49 @@ pbuild::install() {
     # 8. what the modulefile / launchers rely on
     local f
     for f in olex2 olex2c cctbx/cctbx_build/lib lib/libpython3.8.so.1.0 lib/python3.8 util/pyUtil/initpy.py bin/olex2c \
-             sysroot/usr/lib64/libc.so.6 sysroot/usr/lib64/libstdc++.so.6 sysroot/usr/lib64/libgcc_s.so.1 sysroot/usr/lib64/libreadline.so.8; do
+             sysroot/usr/lib64/libc.so.6 sysroot/usr/lib64/libstdc++.so.6 sysroot/usr/lib64/libgcc_s.so.1 \
+             sysroot/usr/lib64/libreadline.so.8 sysroot/usr/lib64/libwebp.so.7 sysroot/usr/lib64/libdeflate.so.0 \
+             sysroot/usr/lib64/libtiff.so.6 sysroot/usr/lib64/libLerc.so.4; do
         [[ -e "${PREFIX}/${f}" ]] || std::die 42 "olex2: ${PREFIX}/${f} missing after install - module would be broken"
     done
+
+    # 9. cctbx first-run setup. On its FIRST start Olex2 regenerates
+    #    cctbx/cctbx_build/libtbx_env (+ dispatchers) by running libtbx's
+    #    configure.py into BaseDir (util/pyUtil/PyToolLib/path_utils.py,
+    #    cold_start). Users cannot write there, so do it now as the owner.
+    #    Upstream quirk: the vendor ships SCons in its site-packages, so
+    #    libtbx believes it can build and the scitbx refresh then fails on
+    #    Fortran sources that are not shipped ("[Errno 2] ... scitbx/lbfgs/
+    #    sdrive.f"). That happens AFTER libtbx_env is pickled (env_config.py),
+    #    so the first run printing "There is a problem with the cctbx" is
+    #    expected; what matters is that the second start is clean.
+    local -r home="${SRC_DIR}/home"            # keep ~/.olex2 of the build user out of it
+    mkdir -p "${home}"
+    printf 'exit\n' | HOME="${home}" timeout 900 "${PREFIX}/bin/olex2c" > "${SRC_DIR}/olex2c.firstrun.log" 2>&1 || true
+    if [[ ! -f "${PREFIX}/cctbx/cctbx_build/libtbx_env" ]]; then
+        cat "${SRC_DIR}/olex2c.firstrun.log"
+        std::die 42 "olex2: first start did not create cctbx/cctbx_build/libtbx_env (log above) - host libs missing? see step 7"
+    fi
+
+    # 10. second start must be clean, and refine the shipped sucrose sample
+    #     (real cctbx/smtbx use: libgomp, openblas, PIL ... all exercised)
+    local -r smoke="${SRC_DIR}/smoke"
+    rm -rf "${smoke}"
+    mkdir -p "${smoke}"
+    cp "${PREFIX}/sample_data/sucrose/"* "${smoke}/"
+    ( cd "${smoke}" && printf 'reap sucrose.res\nrefine 2\nexit\n' | \
+        HOME="${home}" timeout 900 "${PREFIX}/bin/olex2c" > "${SRC_DIR}/olex2c.smoke.log" 2>&1 ) || true
+    echo "olex2: --- second start + refine smoke test, tail of ${SRC_DIR}/olex2c.smoke.log ---"
+    tail -n 40 "${SRC_DIR}/olex2c.smoke.log"
+    echo "olex2: --- end ---"
+    grep -q 'problem with the cctbx' "${SRC_DIR}/olex2c.smoke.log" && \
+        std::die 42 "olex2: cctbx still not usable on the second start - full log: ${SRC_DIR}/olex2c.smoke.log"
+    if grep -qE 'ImportError|cannot open shared object|Traceback' "${SRC_DIR}/olex2c.smoke.log"; then
+        echo "olex2: WARNING - errors during the smoke test on THIS host (full log: ${SRC_DIR}/olex2c.smoke.log):"
+        grep -E 'ImportError|cannot open shared object|Error' "${SRC_DIR}/olex2c.smoke.log" | head -n 20
+    fi
+    # the first run wrote libtbx_env, dispatchers, .pyc files as the build user
+    chmod -R a+rX "${PREFIX}" || true
+
     echo "olex2: installed upstream build '$(cat "${PREFIX}/version.txt" 2>/dev/null)' (tag $(cat "${PREFIX}/olex2.tag" 2>/dev/null))"
 }

olex2/files/config.yaml

@@ -34,6 +34,30 @@ olex2:
       - url: https://kojipkgs.fedoraproject.org/packages/readline/8.2/8.fc40/x86_64/readline-8.2-8.fc40.x86_64.rpm
         name: readline-8.2-8.fc40.x86_64.rpm
         unpacker: none
+      # Image libs the host does not have at the needed version (full NEEDED
+      # scan of every ELF object in the zip, 2026-10-16):
+      #   libwebp.so.7    PIL _imaging + vendor libtiff.so.5 (RHEL: only if libwebp installed)
+      #   libdeflate.so.0 vendor libtiff.so.5 (absent on RHEL8)
+      #   libtiff.so.6    the GUI binary olex2 (RHEL8/9 ship .so.5) -> pulls
+      #                   libjpeg.so.62, libjbig.so.2.1, libLerc.so.4 (same release)
+      - url: https://kojipkgs.fedoraproject.org/packages/libwebp/1.3.2/5.fc40/x86_64/libwebp-1.3.2-5.fc40.x86_64.rpm
+        name: libwebp-1.3.2-5.fc40.x86_64.rpm
+        unpacker: none
+      - url: https://kojipkgs.fedoraproject.org/packages/libdeflate/1.19/2.fc40/x86_64/libdeflate-1.19-2.fc40.x86_64.rpm
+        name: libdeflate-1.19-2.fc40.x86_64.rpm
+        unpacker: none
+      - url: https://kojipkgs.fedoraproject.org/packages/libtiff/4.6.0/2.fc40/x86_64/libtiff-4.6.0-2.fc40.x86_64.rpm
+        name: libtiff-4.6.0-2.fc40.x86_64.rpm
+        unpacker: none
+      - url: https://kojipkgs.fedoraproject.org/packages/libjpeg-turbo/3.0.2/1.fc40/x86_64/libjpeg-turbo-3.0.2-1.fc40.x86_64.rpm
+        name: libjpeg-turbo-3.0.2-1.fc40.x86_64.rpm
+        unpacker: none
+      - url: https://kojipkgs.fedoraproject.org/packages/jbigkit/2.1/29.fc40/x86_64/jbigkit-libs-2.1-29.fc40.x86_64.rpm
+        name: jbigkit-libs-2.1-29.fc40.x86_64.rpm
+        unpacker: none
+      - url: https://kojipkgs.fedoraproject.org/packages/liblerc/4.0.0/6.fc40/x86_64/liblerc-4.0.0-6.fc40.x86_64.rpm
+        name: liblerc-4.0.0-6.fc40.x86_64.rpm
+        unpacker: none
       # static patchelf: re-points the ELF executables at the bundled loader
       - url: https://github.com/NixOS/patchelf/releases/download/0.18.0/patchelf-0.18.0-x86_64.tar.gz
         name: patchelf-0.18.0-x86_64.tar.gz
@@ -48,6 +72,12 @@ olex2:
     libstdc++-14.2.1-3.fc40.x86_64.rpm: 6b6c07bef7ecf7af4a4042bc53eca6c188e07567b3ae710568495a5ca89ccede
     libgcc-14.2.1-3.fc40.x86_64.rpm: 6d84546060cf58bd1ea2456a0ac52659bf9670782258b664dfc0aa046ff3a50d
     readline-8.2-8.fc40.x86_64.rpm: 4e84cb13e24caa3f2b445cef19bf8756d97f17bc64a0958144abc3df284bf038
+    libwebp-1.3.2-5.fc40.x86_64.rpm: 243a0bbcb2475eb787ac34fdb4f7ae5f38ec1de8eb51798dceec84e56edee19f
+    libdeflate-1.19-2.fc40.x86_64.rpm: dcfdd38e20743d799a9627fef64325e984c698a0c9b692e6c84b8904f92acb11
+    libtiff-4.6.0-2.fc40.x86_64.rpm: be959cf9a6f76b7ae963a9bed07ce21bc5dac7bd1aeb136d3222be69e2365e69
+    libjpeg-turbo-3.0.2-1.fc40.x86_64.rpm: 496d2d8c86f183bc0a7bc0b2d8aa9b5b91f20b5621da0b3511d6cdcc6208ff6f
+    jbigkit-libs-2.1-29.fc40.x86_64.rpm: fe6508c217d32185afab13574a8f756626c50ecb9191f9192107803e777c7371
+    liblerc-4.0.0-6.fc40.x86_64.rpm: 14f8e5b8e7f040b46ca7a3b643f65e1260d5c89ab1e2ad8a65233edf26397df0
     patchelf-0.18.0-x86_64.tar.gz: ce84f2447fb7a8679e58bc54a20dc2b01b37b5802e12c57eece772a6f14bf3f0
   versions:
     # 1.5: legacy Ra/RHEL8 module built by the previous recipe from a

olex2/files/olex2-launcher

@@ -44,8 +44,10 @@ export LD_LIBRARY_PATH="$base/lib:$base/ilib:$OLEX2_CCTBX_DIR/cctbx_build/lib${L
 # embedded CPython 3.8 (lib/python3.8, lib/libpython3.8.so.1.0)
 export PYTHONHOME="$base"
 # a PYTHONPATH inherited from another module/conda env would be imported by
-# the embedded 3.8 interpreter and break it - never let it leak in
+# the embedded 3.8 interpreter and break it - never let it leak in; same for
+# the user's ~/.local/lib/python3.8/site-packages
 unset PYTHONPATH
+export PYTHONNOUSERSITE=1
 
 if [[ "$exe" == "olex2" ]]; then
     # GUI only: software OpenGL avoids "Vulkan drivers missing" / GLX errors