diff --git a/olex2/README.md b/olex2/README.md index 89f5394..17cd8e2 100644 --- a/olex2/README.md +++ b/olex2/README.md @@ -77,16 +77,24 @@ A newer glibc can **not** be dropped in via `LD_LIBRARY_PATH`: the dynamic loader path (`/lib64/ld-linux-x86-64.so.2`) is hard-wired into each executable and must match its libc. So the recipe: -- ships glibc 2.39 + GCC 14 `libstdc++`/`libgcc_s` + readline 8.2 under - `$PREFIX/sysroot` (frozen Fedora 40 rpms from kojipkgs, Fedora's - permanent build archive; same glibc generation as the upstream build - host). readline because the binaries need `libreadline.so.8`: RHEL8 has - `.so.7`, RHEL9 `.so.8`, the vendor `lib/` only `.so.6` — bundled, olex2c - needs just `libtinfo.so.6` (ncurses-libs) from the host. glibc, libstdc++ - and readline unpack to `usr/lib64`, libgcc to `/lib64` (Fedora relies on - the filesystem package's `lib64 -> usr/lib64` symlink, which we do not - extract), so `configure` merges `lib64` into `usr/lib64` and recreates - the symlink — one lib dir for the RPATH; +- ships under `$PREFIX/sysroot` (frozen Fedora 40 rpms from kojipkgs, + Fedora's permanent build archive; same glibc generation as the upstream + build host): + + | package | why | + |---|---| + | glibc 2.39, libstdc++/libgcc 14.2 | GLIBC_2.38 / GLIBCXX_3.4.32 | + | readline 8.2 | `libreadline.so.8`: RHEL8 has .so.7, vendor `lib/` .so.6; host needs only `libtinfo.so.6` | + | libwebp 1.3.2 | `libwebp.so.7` for PIL `_imaging` and libtiff (RHEL: only if the libwebp package is installed) | + | libdeflate 1.19 | needed by the vendor `libtiff.so.5`, absent on RHEL8 | + | libtiff 4.6 (+ libjpeg-turbo, jbigkit-libs, liblerc) | `libtiff.so.6` for the **GUI** binary; RHEL8/9 ship .so.5 | + + The list comes from a scan of the `NEEDED` entries of every ELF object in + the zip minus what the zip itself carries. Most packages unpack to + `usr/lib64`, libgcc to `/lib64` (Fedora relies on the filesystem + package's `lib64 -> usr/lib64` symlink, which we do not extract), so + `configure` merges `lib64` into `usr/lib64` and recreates the symlink — + one lib dir for the RPATH; - runs `patchelf --set-interpreter --force-rpath --set-rpath :$PREFIX/lib:$PREFIX/ilib:$PREFIX/cctbx/cctbx_build/lib:$PREFIX:$ORIGIN` on every dynamic executable (`hart`, `hart_mpi` are static and skipped). @@ -105,9 +113,34 @@ Rules that keep this safe: - the originals are untouched inside the cached zip; a rebuild re-patches from scratch. +### cctbx first run happens at build time + +On its first start Olex2 regenerates `cctbx/cctbx_build/libtbx_env` and +the libtbx dispatchers by running cctbx's `configure.py` *into BaseDir* +(`util/pyUtil/PyToolLib/path_utils.py`, `cold_start`). On a per-user +install that is invisible; on a shared read-only prefix it would fail for +every user. So `install` runs `olex2c` once as the owner (step 9, isolated +`HOME`), checks `libtbx_env` exists, then starts it a second time and +refines the shipped sucrose sample (step 10), dying if cctbx is still +reported broken. Expect in the first-run log, verbatim from upstream: + +``` +Using fable to convert scitbx/lbfgs.f +There is a problem with the cctbx: [Errno 2] No such file or directory: '.../scitbx/lbfgs/sdrive.f' +``` + +That is an upstream quirk: the vendor ships SCons in its site-packages, so +libtbx believes it can build and the scitbx refresh wants Fortran sources +that are not shipped. It happens *after* `libtbx_env` is pickled +(`env_config.py`), which is why the second start is clean. Every Linux +user of the vendor zip goes through the same first start. + Unverified until the first RHEL9 run: NSS lookups through the host's -`libnss_sss` under the newer libc (expected fine, NSS ABI is stable), and -`hart_mpi` against the host MPI. +`libnss_sss` under the newer libc (expected fine, NSS ABI is stable), +`hart_mpi` against the host MPI, and NoSpherA2 (it references Intel MKL +kernel libraries `libmkl_*.so.3` that are neither shipped nor on the +hosts; whether they are hard requirements or optional dlopen targets is +untested). ### Why the binary archive and not a source build @@ -164,10 +197,14 @@ $OLEX2_ROOT/sysroot/usr/lib64/ld-linux-x86-64.so.2 --list $OLEX2_ROOT/olex2c | g ``` `libc.so.6`/`libstdc++.so.6` must resolve under `$OLEX2_ROOT/sysroot`; every `not -found` line is a host package to `dnf install` (expected candidates on a -minimal RHEL9: `gtk2`, `mesa-libGL`, `mesa-libGLU`, `libSM`, `libXxf86vm`, -`libxkbcommon`). Plain `ldd` on the raw binary also works after the patch -(it honours the new interpreter). +found` line is a host package to `dnf install`. From the NEEDED scan, the +host must provide (all standard RHEL8/9 packages): `gtk2`, `gdk-pixbuf2`, +`pango`, `cairo`, `glib2`, `fontconfig`, `freetype`, `libX11`, `libSM`, +`libxcb`, `libxkbcommon`, `libwayland-client`, `ncurses-libs` +(libtinfo), `libgomp` (cctbx refinement!), `libgfortran` + `libquadmath` +(openblas/scipy), `zlib`, `xz-libs`, `libzstd`, `libuuid`, `libxcrypt`; +GUI additionally `mesa-libGL`, `mesa-libGLU`. Plain `ldd` on the raw +binary also works after the patch (it honours the new interpreter). ## Verifying the install @@ -215,11 +252,13 @@ not to edit the hash: 2. check the new binaries' glibc need still fits the bundled 2.39: `unzip -p olex2-linux64.zip olex2/olex2c | strings | grep -o 'GLIBC_2\.[0-9]*' | sort -uV | tail -1` (and `GLIBCXX_3\.4\.[0-9]*` ≤ 3.4.33). If upstream moved past that, bump - the four rpms (glibc, libstdc++, libgcc, readline) in `urls:`/`shasums:` - to a newer frozen Fedora release (kojipkgs - `packages/glibc///x86_64/`, `packages/gcc/...`, - `packages/readline/...`) — they must come from the same release so - readline's glibc requirement is met; + all sysroot rpms in `urls:`/`shasums:` to a newer frozen Fedora release + — they must come from the same release so their mutual glibc + requirements are met. kojipkgs has no browsable index (bot gate) but + serves any `packages///.fcNN/x86_64/.x86_64.rpm`; + find the NVR via the Koji XML-RPC hub (`listTagged`, tag `fNN`, + `latest=1`) and note that the source name can differ from the lib name + (`liblerc`, `jbigkit` → `jbigkit-libs`); 3. add `1.5-7800` under `versions:` (`relstage: unstable`) and `olex2-1.5-7800-linux64.zip: ` under `shasums:` — the `urls:` entry needs no change (`${V_PKG}` = series dir, `${V}` = cache name); diff --git a/olex2/build b/olex2/build index aafef86..fdaeba5 100755 --- a/olex2/build +++ b/olex2/build @@ -33,6 +33,12 @@ OLEX2_SYSROOT_RPMS=( libstdc++-14.2.1-3.fc40.x86_64.rpm libgcc-14.2.1-3.fc40.x86_64.rpm readline-8.2-8.fc40.x86_64.rpm # libreadline.so.8: RHEL8 has .so.7, vendor lib/ .so.6 + libwebp-1.3.2-5.fc40.x86_64.rpm # libwebp.so.7: PIL + libtiff + libdeflate-1.19-2.fc40.x86_64.rpm # libdeflate.so.0: vendor libtiff.so.5 (absent on RHEL8) + libtiff-4.6.0-2.fc40.x86_64.rpm # libtiff.so.6: GUI binary (RHEL ships .so.5) + its deps: + libjpeg-turbo-3.0.2-1.fc40.x86_64.rpm + jbigkit-libs-2.1-29.fc40.x86_64.rpm + liblerc-4.0.0-6.fc40.x86_64.rpm ) OLEX2_PATCHELF_TGZ="patchelf-0.18.0-x86_64.tar.gz" # ELF executables in the vendor tree that get the bundled loader (static ones, @@ -192,8 +198,49 @@ pbuild::install() { # 8. what the modulefile / launchers rely on local f for f in olex2 olex2c cctbx/cctbx_build/lib lib/libpython3.8.so.1.0 lib/python3.8 util/pyUtil/initpy.py bin/olex2c \ - sysroot/usr/lib64/libc.so.6 sysroot/usr/lib64/libstdc++.so.6 sysroot/usr/lib64/libgcc_s.so.1 sysroot/usr/lib64/libreadline.so.8; do + sysroot/usr/lib64/libc.so.6 sysroot/usr/lib64/libstdc++.so.6 sysroot/usr/lib64/libgcc_s.so.1 \ + sysroot/usr/lib64/libreadline.so.8 sysroot/usr/lib64/libwebp.so.7 sysroot/usr/lib64/libdeflate.so.0 \ + sysroot/usr/lib64/libtiff.so.6 sysroot/usr/lib64/libLerc.so.4; do [[ -e "${PREFIX}/${f}" ]] || std::die 42 "olex2: ${PREFIX}/${f} missing after install - module would be broken" done + + # 9. cctbx first-run setup. On its FIRST start Olex2 regenerates + # cctbx/cctbx_build/libtbx_env (+ dispatchers) by running libtbx's + # configure.py into BaseDir (util/pyUtil/PyToolLib/path_utils.py, + # cold_start). Users cannot write there, so do it now as the owner. + # Upstream quirk: the vendor ships SCons in its site-packages, so + # libtbx believes it can build and the scitbx refresh then fails on + # Fortran sources that are not shipped ("[Errno 2] ... scitbx/lbfgs/ + # sdrive.f"). That happens AFTER libtbx_env is pickled (env_config.py), + # so the first run printing "There is a problem with the cctbx" is + # expected; what matters is that the second start is clean. + local -r home="${SRC_DIR}/home" # keep ~/.olex2 of the build user out of it + mkdir -p "${home}" + printf 'exit\n' | HOME="${home}" timeout 900 "${PREFIX}/bin/olex2c" > "${SRC_DIR}/olex2c.firstrun.log" 2>&1 || true + if [[ ! -f "${PREFIX}/cctbx/cctbx_build/libtbx_env" ]]; then + cat "${SRC_DIR}/olex2c.firstrun.log" + std::die 42 "olex2: first start did not create cctbx/cctbx_build/libtbx_env (log above) - host libs missing? see step 7" + fi + + # 10. second start must be clean, and refine the shipped sucrose sample + # (real cctbx/smtbx use: libgomp, openblas, PIL ... all exercised) + local -r smoke="${SRC_DIR}/smoke" + rm -rf "${smoke}" + mkdir -p "${smoke}" + cp "${PREFIX}/sample_data/sucrose/"* "${smoke}/" + ( cd "${smoke}" && printf 'reap sucrose.res\nrefine 2\nexit\n' | \ + HOME="${home}" timeout 900 "${PREFIX}/bin/olex2c" > "${SRC_DIR}/olex2c.smoke.log" 2>&1 ) || true + echo "olex2: --- second start + refine smoke test, tail of ${SRC_DIR}/olex2c.smoke.log ---" + tail -n 40 "${SRC_DIR}/olex2c.smoke.log" + echo "olex2: --- end ---" + grep -q 'problem with the cctbx' "${SRC_DIR}/olex2c.smoke.log" && \ + std::die 42 "olex2: cctbx still not usable on the second start - full log: ${SRC_DIR}/olex2c.smoke.log" + if grep -qE 'ImportError|cannot open shared object|Traceback' "${SRC_DIR}/olex2c.smoke.log"; then + echo "olex2: WARNING - errors during the smoke test on THIS host (full log: ${SRC_DIR}/olex2c.smoke.log):" + grep -E 'ImportError|cannot open shared object|Error' "${SRC_DIR}/olex2c.smoke.log" | head -n 20 + fi + # the first run wrote libtbx_env, dispatchers, .pyc files as the build user + chmod -R a+rX "${PREFIX}" || true + echo "olex2: installed upstream build '$(cat "${PREFIX}/version.txt" 2>/dev/null)' (tag $(cat "${PREFIX}/olex2.tag" 2>/dev/null))" } diff --git a/olex2/files/config.yaml b/olex2/files/config.yaml index 45617c9..0cd4ba4 100644 --- a/olex2/files/config.yaml +++ b/olex2/files/config.yaml @@ -34,6 +34,30 @@ olex2: - url: https://kojipkgs.fedoraproject.org/packages/readline/8.2/8.fc40/x86_64/readline-8.2-8.fc40.x86_64.rpm name: readline-8.2-8.fc40.x86_64.rpm unpacker: none + # Image libs the host does not have at the needed version (full NEEDED + # scan of every ELF object in the zip, 2026-10-16): + # libwebp.so.7 PIL _imaging + vendor libtiff.so.5 (RHEL: only if libwebp installed) + # libdeflate.so.0 vendor libtiff.so.5 (absent on RHEL8) + # libtiff.so.6 the GUI binary olex2 (RHEL8/9 ship .so.5) -> pulls + # libjpeg.so.62, libjbig.so.2.1, libLerc.so.4 (same release) + - url: https://kojipkgs.fedoraproject.org/packages/libwebp/1.3.2/5.fc40/x86_64/libwebp-1.3.2-5.fc40.x86_64.rpm + name: libwebp-1.3.2-5.fc40.x86_64.rpm + unpacker: none + - url: https://kojipkgs.fedoraproject.org/packages/libdeflate/1.19/2.fc40/x86_64/libdeflate-1.19-2.fc40.x86_64.rpm + name: libdeflate-1.19-2.fc40.x86_64.rpm + unpacker: none + - url: https://kojipkgs.fedoraproject.org/packages/libtiff/4.6.0/2.fc40/x86_64/libtiff-4.6.0-2.fc40.x86_64.rpm + name: libtiff-4.6.0-2.fc40.x86_64.rpm + unpacker: none + - url: https://kojipkgs.fedoraproject.org/packages/libjpeg-turbo/3.0.2/1.fc40/x86_64/libjpeg-turbo-3.0.2-1.fc40.x86_64.rpm + name: libjpeg-turbo-3.0.2-1.fc40.x86_64.rpm + unpacker: none + - url: https://kojipkgs.fedoraproject.org/packages/jbigkit/2.1/29.fc40/x86_64/jbigkit-libs-2.1-29.fc40.x86_64.rpm + name: jbigkit-libs-2.1-29.fc40.x86_64.rpm + unpacker: none + - url: https://kojipkgs.fedoraproject.org/packages/liblerc/4.0.0/6.fc40/x86_64/liblerc-4.0.0-6.fc40.x86_64.rpm + name: liblerc-4.0.0-6.fc40.x86_64.rpm + unpacker: none # static patchelf: re-points the ELF executables at the bundled loader - url: https://github.com/NixOS/patchelf/releases/download/0.18.0/patchelf-0.18.0-x86_64.tar.gz name: patchelf-0.18.0-x86_64.tar.gz @@ -48,6 +72,12 @@ olex2: libstdc++-14.2.1-3.fc40.x86_64.rpm: 6b6c07bef7ecf7af4a4042bc53eca6c188e07567b3ae710568495a5ca89ccede libgcc-14.2.1-3.fc40.x86_64.rpm: 6d84546060cf58bd1ea2456a0ac52659bf9670782258b664dfc0aa046ff3a50d readline-8.2-8.fc40.x86_64.rpm: 4e84cb13e24caa3f2b445cef19bf8756d97f17bc64a0958144abc3df284bf038 + libwebp-1.3.2-5.fc40.x86_64.rpm: 243a0bbcb2475eb787ac34fdb4f7ae5f38ec1de8eb51798dceec84e56edee19f + libdeflate-1.19-2.fc40.x86_64.rpm: dcfdd38e20743d799a9627fef64325e984c698a0c9b692e6c84b8904f92acb11 + libtiff-4.6.0-2.fc40.x86_64.rpm: be959cf9a6f76b7ae963a9bed07ce21bc5dac7bd1aeb136d3222be69e2365e69 + libjpeg-turbo-3.0.2-1.fc40.x86_64.rpm: 496d2d8c86f183bc0a7bc0b2d8aa9b5b91f20b5621da0b3511d6cdcc6208ff6f + jbigkit-libs-2.1-29.fc40.x86_64.rpm: fe6508c217d32185afab13574a8f756626c50ecb9191f9192107803e777c7371 + liblerc-4.0.0-6.fc40.x86_64.rpm: 14f8e5b8e7f040b46ca7a3b643f65e1260d5c89ab1e2ad8a65233edf26397df0 patchelf-0.18.0-x86_64.tar.gz: ce84f2447fb7a8679e58bc54a20dc2b01b37b5802e12c57eece772a6f14bf3f0 versions: # 1.5: legacy Ra/RHEL8 module built by the previous recipe from a diff --git a/olex2/files/olex2-launcher b/olex2/files/olex2-launcher index 7787644..c066f38 100644 --- a/olex2/files/olex2-launcher +++ b/olex2/files/olex2-launcher @@ -44,8 +44,10 @@ export LD_LIBRARY_PATH="$base/lib:$base/ilib:$OLEX2_CCTBX_DIR/cctbx_build/lib${L # embedded CPython 3.8 (lib/python3.8, lib/libpython3.8.so.1.0) export PYTHONHOME="$base" # a PYTHONPATH inherited from another module/conda env would be imported by -# the embedded 3.8 interpreter and break it - never let it leak in +# the embedded 3.8 interpreter and break it - never let it leak in; same for +# the user's ~/.local/lib/python3.8/site-packages unset PYTHONPATH +export PYTHONNOUSERSITE=1 if [[ "$exe" == "olex2" ]]; then # GUI only: software OpenGL avoids "Vulkan drivers missing" / GLX errors