Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
105 lines
4.5 KiB
Python
105 lines
4.5 KiB
Python
#!/usr/bin/env python
|
|
"""Check that DIALS places a TILTED detector where Jungfraujoch does.
|
|
|
|
Run under the DIALS environment:
|
|
|
|
source /opt/dials-v3-27-0/dials_env.sh
|
|
dials.python tests/nxmx_geometry_dials_test.py <master.h5>
|
|
|
|
Why this exists: the NXmx transformation chain the writer produces is only exercised by the rest of
|
|
CI at zero tilt, where every plausible encoding of the tilt - right axes or swapped, right
|
|
composition order or reversed, right pivot or not - gives exactly the same answer. The errors are
|
|
second order and only appear with two angles non-zero at once, or with a non-zero distance being
|
|
rotated. A previous encoding placed the detector 40 mm out at a few degrees of tilt while passing
|
|
every zero-tilt check.
|
|
|
|
The reference positions below are the same ones pinned in
|
|
tests/DiffractionGeometryTest.cpp ("DiffractionGeometry_Tilted_vs_PyFAI_and_DIALS"), which checks
|
|
them against pyFAI as well; that comment explains how to regenerate them. Keep the two in step.
|
|
|
|
The script patches the geometry into an existing master rather than requiring the writing tool to
|
|
take tilt options: the angle VALUES are just scalars, while the axis vectors, the depends_on chain
|
|
and the pivot - the parts that were wrong - stay exactly as the writer emitted them, so they remain
|
|
under test.
|
|
"""
|
|
|
|
import shutil
|
|
import sys
|
|
|
|
import h5py
|
|
import numpy as np
|
|
from dxtbx.model.experiment_list import ExperimentListFactory
|
|
|
|
# Geometry to impose. Mixed signs on purpose, and all three angles non-zero.
|
|
BEAM_X, BEAM_Y, DISTANCE_M = 1000.0, 1275.0, 0.150
|
|
ROT1, ROT2, ROT3 = 0.05, -0.03, 0.02
|
|
|
|
# Expected lab position per pixel, in the imgCIF frame DIALS reports (mm).
|
|
EXPECTED = {
|
|
(0, 0): (-69.399544982, 98.819979800, -150.623559832),
|
|
(1000, 1275): (7.405508016, 4.642730847, -149.745128473),
|
|
(300, 1800): (-44.232877406, -35.676610671, -153.548927191),
|
|
(1700, 400): (58.519164629, 71.195014535, -145.153947836),
|
|
}
|
|
|
|
# 2 um, matching the C++ test. A wrong axis, order or pivot is a millimetre-scale error.
|
|
TOLERANCE_MM = 2e-3
|
|
|
|
|
|
def main(master):
|
|
patched = "geometry_check_master.h5"
|
|
shutil.copy(master, patched)
|
|
|
|
with h5py.File(patched, "r+") as h:
|
|
det = h["/entry/instrument/detector"]
|
|
det["beam_center_x"][()] = BEAM_X
|
|
det["beam_center_y"][()] = BEAM_Y
|
|
det["distance"][()] = DISTANCE_M
|
|
transformations = det["transformations"]
|
|
transformations["rot1"][()] = ROT1
|
|
transformations["rot2"][()] = ROT2
|
|
transformations["rot3"][()] = ROT3
|
|
# The writer derives the translation from the beam centre and the distance, so it has to be
|
|
# recomputed here - both the magnitude and the direction it points in.
|
|
pixel_m = float(det["x_pixel_size"][()])
|
|
vector = np.array([BEAM_X * pixel_m, BEAM_Y * pixel_m, DISTANCE_M])
|
|
length = np.linalg.norm(vector)
|
|
transformations["translation"][()] = length
|
|
transformations["translation"].attrs["vector"] = vector / length
|
|
|
|
panel = ExperimentListFactory.from_filenames([patched])[0].detector[0]
|
|
origin = np.array(panel.get_origin())
|
|
fast = np.array(panel.get_fast_axis())
|
|
slow = np.array(panel.get_slow_axis())
|
|
pixel_mm = panel.get_pixel_size()[0]
|
|
|
|
# Deliberately NOT get_pixel_lab_coord(): it applies a parallax correction from the sensor
|
|
# thickness and material that Jungfraujoch does not model, worth ~0.1 mm at the detector edge.
|
|
failures = []
|
|
for (x, y), expected in sorted(EXPECTED.items()):
|
|
got = origin + x * pixel_mm * fast + y * pixel_mm * slow
|
|
error = np.max(np.abs(got - np.array(expected)))
|
|
status = "ok" if error <= TOLERANCE_MM else "FAILED"
|
|
print(f" pixel ({x:5d},{y:5d}): max error {error:.2e} mm {status}")
|
|
if error > TOLERANCE_MM:
|
|
failures.append(f"pixel ({x},{y}): got {got}, expected {expected}")
|
|
|
|
if failures:
|
|
print("\nDIALS does not place the tilted detector where Jungfraujoch does:")
|
|
for failure in failures:
|
|
print(" " + failure)
|
|
print("\nThe NXmx transformation chain in writer/HDF5NXmx.cpp is the thing to look at: the "
|
|
"rot1/rot2/rot3 axis vectors, the order of the depends_on chain, and whether "
|
|
"translation sits inside the rotations so the tilt pivots about the sample.")
|
|
return 1
|
|
|
|
print(f"\nAll {len(EXPECTED)} pixels agree within {TOLERANCE_MM} mm.")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
if len(sys.argv) != 2:
|
|
print(__doc__)
|
|
sys.exit(2)
|
|
sys.exit(main(sys.argv[1]))
|