leonarski_fandjungfrau
4dc2534dbf
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
7.5 KiB
7.5 KiB
Security
Jungfraujoch is a data-acquisition and analysis system for X-ray detectors, designed to run inside a controlled facility network. This document describes what the software does and does not protect against, the current known limitations, and the authentication work in progress.
Threat model and scope
The security model targets a semi-trusted internal facility network. The concern is a peer on that network reaching a Jungfraujoch service with little or no effort — a mistyped host/port, a curious colleague, a mis-pointed script, a stray browser tab — not a determined attacker and not passive wire capture (which is the responsibility of the network layer: 802.1x, VLANs, facility infrastructure).
The asset that matters most is the confidentiality of live analysis data: the diffraction images and derived metadata (unit cell, resolution, spot counts, sample name) that reveal which sample is being measured. This matters for industrial and proprietary experiments. By contrast, acquisition control (start / stop / configure) is treated as low risk — scientists operate their own experiments and there is little to gain from restricting it.
Security is best-effort: measures that materially impede normal operation get turned off, so the design favours a few high-value, low-friction controls over comprehensive lockdown.
Out of scope. Jungfraujoch is not designed to be exposed to an untrusted network or the public internet. Do not do this.
1. Good practice — what is and is not protected
What you can secure (and should)
These controls work and a deployment should apply them (see also DEPLOYMENT.md):
- Network isolation. Keep the broker and its data streams on a controlled segment. The broker ↔ writer ↔ receiver traffic should run on a dedicated back-end network, with the data-socket addresses pinned to that interface and the ports firewalled.
- Reverse proxy for TLS. The broker speaks plain HTTP. To get HTTPS, put a reverse proxy
(Apache / nginx) in front that terminates TLS and pin the broker to
localhostbehind it. The desktop viewer supportshttps://endpoints — choose the scheme in the Open HTTP Connection dialog. - Filesystem confinement of written data. The writer creates NXmx HDF5 files on shared storage. Confidentiality of that data at rest is enforced by the filesystem: run the writer under a dedicated identity and use directory ownership / ACLs (and setgid) so that only the owning experiment can read its files.
- Firewall the ZeroMQ ports. The image / preview / metadata / republish streams have no access control of their own (see below), so restrict who can reach those ports at the network layer.
What the software does NOT provide
Be explicit about the gaps so a deployment does not assume protection that is not there:
- No authentication or authorization in the broker. The HTTP/REST API currently has no login, token, or access control. Anyone who can reach the broker's host and port has full read access (live images, unit cell, resolution, sample metadata) and full write access (start, cancel, reconfigure). Confidentiality currently depends entirely on network/firewall isolation. This is being addressed — see §3.
- No transport encryption in the broker. The broker serves plain HTTP; there is no built-in TLS. Encryption must be provided by a reverse proxy.
- No access control or encryption on the ZeroMQ streams. The preview, metadata, image, and
republish streams are unauthenticated sockets. Any peer that can connect can subscribe to live
data. For the image
PUSHstream specifically, an accidental extra consumer does not merely eavesdrop — aPULLpeer is load-balanced into the stream and will divert images away from the real writer. - No per-user isolation. The broker has no concept of users; it cannot separate one operator's access from another's.
- No application-level audit trail of who accessed or changed what.
Recommended deployment checklist
- Broker and back-end streams on an isolated network; never exposed to a general/untrusted network.
- ZeroMQ data-socket addresses pinned to the back-end interface; ports firewalled to known peers.
- TLS terminated by a reverse proxy; broker bound to
localhostbehind it. - Writer run under a dedicated identity; data directories owned / ACL'd per experiment (setgid) so users read only their own data.
- ZeroMQ compatibility streams (preview / metadata / republish) enabled only if actually consumed, and only on the trusted back-end.
2. Known issues
| # | Issue | Impact | Mitigation today |
|---|---|---|---|
| 1 | Broker HTTP API has no authentication | Anyone who can reach it has full read (confidential live data) + write (control) | Network / firewall isolation; §3 in progress |
| 2 | Broker binds all interfaces, plain HTTP | Reachable from anywhere routable; no encryption | Expose only on the trusted segment; TLS via reverse proxy |
| 3 | ZeroMQ preview / metadata / image / republish streams are unauthenticated and unencrypted | Live-data exfiltration; a rogue PULL on the image stream diverts/steals images |
Firewall the ports; run only on the back-end network |
| 4 | Web frontend assumes an open API | The bundled UI has no auth and expects to reach an open broker | Serve and reach it only on the trusted network |
Input robustness. Services parse framed data from peers on the (trusted) data path. Hardening of untrusted-frame handling (size caps, overflow guards) is ongoing; these paths are not intended to face an untrusted network.
3. Work in progress — authenticated read access
The main gap (issue #1) is being closed with a best-effort, low-friction scheme that protects the confidential read endpoints while leaving acquisition control open.
Enabling step (done): viewer HTTP client on libcurl. The desktop viewer's broker client was
migrated from a plain-HTTP library to libcurl, which gives it HTTPS plus the ability to
authenticate. The build stays self-contained per platform: on Windows, TLS and Kerberos come
from the operating system (Schannel + SSPI, no external dependencies); on Linux, from system
OpenSSL + Kerberos (GSSAPI / krb5). This is why the Linux build now needs the Kerberos
development headers (libkrb5-dev on Debian/Ubuntu, krb5-devel on RHEL/Rocky).
Planned enforcement (not yet implemented). Two complementary options, both gating only the sensitive read endpoints (live images, scan result, preview plots, statistics) and leaving writes open:
- Per-experiment Bearer token. An optional token supplied at acquisition start (
/start); when set, the broker requires it (Authorization: Bearer …) on the read endpoints. No token set → no enforcement (backward compatible). The token is minted externally and handed to the authorised viewer(s); the broker only compares strings. A rotated per-experiment token segments one experiment's data from the next. - Kerberos / GSSAPI single sign-on. A pass-through reverse proxy performs Kerberos/SPNEGO
authentication (service keytab) and forwards the authenticated user name to a
localhost-pinned broker as a trusted header. This gives Active Directory single sign-on with no token to carry, at the cost of a proxy deployment. The viewer's libcurl client can negotiate Kerberos directly against such a proxy.
Neither enforcement path is in the broker yet. Until then, confidentiality relies on the network and filesystem controls in §1.