Files
Jungfraujoch/detector_control/DectrisSimplonClient.cpp
T
leonarski_fandjungfrau 4dc2534dbf
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
v1.0.0.rc-162 (#72)
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one.

* HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected.
* HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it.
* HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset.
* HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts.
* HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout.
* A grid scan and a goniometer axis can both be set; they are no longer alternatives.
* `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000.
* The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned.
* The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer.
* rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it.
* rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`.
* rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way.
* rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound.
* rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached.
* rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use.
* rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own.
* rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement.
* rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged.
* rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged.
* Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged.
* A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses.
* rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given.
* CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer.
* The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2.
* Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact.

**Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`):
* `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file.
* `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them.

---------

Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch>
Reviewed-on: #72
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-08-25 08:21:39 +02:00

342 lines
13 KiB
C++

// SPDX-FileCopyrightText: 2025 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include <base64/Base64.h>
#include <httplib.h>
#include "DectrisSimplonClient.h"
std::string to_string(SimplonState state) {
switch (state) {
case SimplonState::Na: return "Na";
case SimplonState::Ready: return "Ready";
case SimplonState::Initialize: return "Initialize";
case SimplonState::Configure: return "Configure";
case SimplonState::Acquire: return "Acquire";
case SimplonState::Idle: return "Idle";
case SimplonState::Test: return "Test";
case SimplonState::Error: return "Error";
default: return "Unknown";
}
}
DectrisSimplonClient::DectrisSimplonClient(const std::string &addr, uint16_t in_port)
: hostname(addr), port(in_port) {
}
void DectrisSimplonClient::SendDetectorCommand(SimplonDetectorCommand cmd) {
std::string key;
switch (cmd) {
case SimplonDetectorCommand::Initialize:
key = "initialize";
break;
case SimplonDetectorCommand::Arm:
key = "arm";
break;
case SimplonDetectorCommand::Disarm:
key = "disarm";
break;
case SimplonDetectorCommand::Trigger:
key = "trigger";
break;
case SimplonDetectorCommand::Cancel:
key = "cancel";
break;
}
auto addr = GenAddr(SimplonModule::Detector, SimplonTask::Command, key);
httplib::Client cli_cmd(hostname, port);
cli_cmd.set_connection_timeout(std::chrono::milliseconds(500));
cli_cmd.set_read_timeout(std::chrono::minutes(20)); // Initialize or disarm might take a lot of time
cli_cmd.set_write_timeout(std::chrono::minutes(20)); // Initialize or disarm might take a lot of time
auto res = cli_cmd.Post(addr);
if (!res || res->status != httplib::StatusCode::OK_200) {
std::string err_msg = "Command failed " + key;
if (res) {
err_msg += "; HTTP" + std::to_string(res->status);
if (!res->body.empty())
err_msg += "; " + res->body;
}
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, err_msg);
}
}
void DectrisSimplonClient::SetConfig(SimplonModule element, const std::string &key, const nlohmann::json &value) {
std::string addr = GenAddr(element, SimplonTask::Config, key);
nlohmann::json j;
j["value"] = value;
httplib::Client cli_cmd(hostname, port);
cli_cmd.set_connection_timeout(std::chrono::seconds(1));
cli_cmd.set_read_timeout(std::chrono::minutes(20)); // Initialize or disarm might take a lot of time
cli_cmd.set_write_timeout(std::chrono::minutes(20)); // Initialize or disarm might take a lot of time
auto res = cli_cmd.Post(addr, j.dump(), "application/json");
if (!res || res->status != httplib::StatusCode::OK_200) {
std::string err_msg = "Configure failed " + key;
if (res) {
err_msg += "; HTTP" + std::to_string(res->status);
if (!res->body.empty())
err_msg += "; " + res->body;
}
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, err_msg);
}
}
void DectrisSimplonClient::SetConfigIfDifferent(SimplonModule element, const std::string &key, float value, float tolerance) {
auto curr_val = GetConfig(element, key).val.get<float>();
if (std::abs(curr_val - value) > tolerance)
SetConfig(element, key, value);
}
void DectrisSimplonClient::SetConfigIfDifferent(SimplonModule element, const std::string &key, int64_t value) {
auto curr_val = GetConfig(element, key).val.get<int64_t>();
if (curr_val != value)
SetConfig(element, key, value);
}
std::string DectrisSimplonClient::GenAddr(SimplonModule element, SimplonTask task, const std::string& key) {
std::string addr;
switch (element) {
case SimplonModule::Detector:
addr += "/detector/";
break;
case SimplonModule::Stream:
addr += "/stream/";
break;
case SimplonModule::Filewriter:
addr += "/filewriter/";
break;
case SimplonModule::Monitor:
addr += "/monitor/";
break;
}
addr += "api/";
addr += api_version;
switch (task) {
case SimplonTask::Config:
addr += "/config/";
break;
case SimplonTask::Command:
addr += "/command/";
break;
case SimplonTask::Status:
addr += "/status/";
break;
}
addr += key;
return addr;
}
SimplonConfig
DectrisSimplonClient::GetConfig(SimplonModule element, SimplonTask task, const std::string &key) {
httplib::Client cli_simple(hostname, port);
cli_simple.set_connection_timeout(std::chrono::milliseconds(500));
cli_simple.set_read_timeout(std::chrono::seconds(1));
cli_simple.set_write_timeout(std::chrono::seconds(1));
auto res = cli_simple.Get(GenAddr(element, task, key));
if (!res) {
throw JFJochException(JFJochExceptionCategory::SimplonError,
"Get key " + key + " timeout/error: " + to_string(res.error()));
}
if (res->status != httplib::StatusCode::OK_200) {
throw JFJochException(JFJochExceptionCategory::SimplonError,
"Get key " + key + " failed: " + to_string(res.error()));
}
try {
auto j = nlohmann::json::parse(res->body);
SimplonConfig ret{};
ret.val = j["value"];
if (j.contains("unit"))
ret.unit = j["unit"];
if (j.contains("min"))
ret.min = j["min"];
if (j.contains("max"))
ret.max = j["max"];
return ret;
} catch (const std::exception &e) {
throw JFJochException(JFJochExceptionCategory::JSON, e.what());
}
}
SimplonConfig DectrisSimplonClient::GetConfig(SimplonModule element, const std::string &key) {
return GetConfig(element, SimplonTask::Config, key);
}
SimplonConfig DectrisSimplonClient::GetDetCfg(const std::string &key) {
return GetConfig(SimplonModule::Detector, key);
}
nlohmann::json DectrisSimplonClient::GetStatus(SimplonModule element, const std::string &key) {
return GetConfig(element, SimplonTask::Status, key).val;
}
template <class T>
T float2time(float time_in_seconds) {
return std::chrono::duration_cast<T>(std::chrono::duration<float>(time_in_seconds));
}
void DectrisSimplonClient::ReadDetectorConfig(DetectorSetup &setup) {
// Need to initialize to read configuration information
if ((GetState() == SimplonState::Na)
|| (GetState() == SimplonState::Error))
throw JFJochException(JFJochExceptionCategory::Detector,
"Detector state Na or Error - need to reinitialize before reading configuration");
setup.Description(GetDetCfg( "description").val);
setup.PixelSize_um(std::round(GetDetCfg( "x_pixel_size").val.get<float>() * 1e6f));
setup.SerialNumber(GetDetCfg( "detector_number").val);
setup.SensorMaterial(GetDetCfg( "sensor_material").val);
setup.SensorThickness_um(std::round(GetDetCfg( "sensor_thickness").val.get<float>() * 1e6f));
// Only the image depth is read: it sizes the image buffer before the first frame arrives, and
// is replaced from the stream2 image_dtype when the detector arms. The electronic readout depth
// is deliberately not read - nothing consumes it.
setup.BitDepthImage(GetDetCfg("bit_depth_image").val.get<int64_t>());
setup.MinFrameTime(float2time<std::chrono::nanoseconds>(GetDetCfg("frame_time").min.get<float>()));
setup.MinCountTime(float2time<std::chrono::nanoseconds>(GetDetCfg("count_time").min.get<float>()));
setup.ReadOutTime(float2time<std::chrono::nanoseconds>(GetDetCfg("detector_readout_time").val.get<float>()));
nlohmann::json min_threshold_energy = GetDetCfg("threshold_energy").min;
if (min_threshold_energy.is_number())
setup.MinThreshold_keV(min_threshold_energy.get<float>() / 1000.0f);
else
setup.MinThreshold_keV(2.7f);
int64_t width = GetDetCfg("x_pixels_in_detector").val.get<int64_t>();
int64_t height = GetDetCfg( "y_pixels_in_detector").val.get<int64_t>();
setup.Geometry(DetectorGeometryFixed(width, height));
}
void DectrisSimplonClient::InitializeDetector(DetectorSetup& setup) {
auto state = GetState();
if ((state == SimplonState::Na) || (state == SimplonState::Error))
SendDetectorCommand(SimplonDetectorCommand::Initialize);
if (GetState() != SimplonState::Idle)
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"Initialize unsuccessful");
SetConfig(SimplonModule::Detector, "roi_mode", setup.GetDECTRISROI());
// Configure streaming, disable file writer and monitor
SetConfig(SimplonModule::Stream, "format", "cbor");
SetConfig(SimplonModule::Stream, "mode", "enabled");
SetConfig(SimplonModule::Filewriter, "mode", "disabled");
SetConfig(SimplonModule::Monitor, "mode", "disabled");
ReadDetectorConfig(setup);
}
void DectrisSimplonClient::TriggerAcquisition() {
SendDetectorCommand(SimplonDetectorCommand::Trigger);
}
void DectrisSimplonClient::CancelAcquisition() {
SendDetectorCommand(SimplonDetectorCommand::Cancel);
}
void DectrisSimplonClient::EndAcquisitionFinished() {
SendDetectorCommand(SimplonDetectorCommand::Disarm);
}
void DectrisSimplonClient::StartAcquisition(const DiffractionExperiment& experiment) {
SetConfigIfDifferent(SimplonModule::Detector, "photon_energy", experiment.GetIncidentEnergy_keV() * 1e3f, 0.01f);
auto thr = experiment.GetEigerThreshold_keV();
SetConfigIfDifferent(SimplonModule::Detector, "threshold_energy", thr * 1e3f, 0.01);
SetConfigIfDifferent(SimplonModule::Detector, "count_time", std::chrono::duration<float>(experiment.GetFrameCountTime()).count(), 1e-9);
SetConfigIfDifferent(SimplonModule::Detector, "frame_time", std::chrono::duration<float>(experiment.GetFrameTime()).count(), 1e-9);
SetConfigIfDifferent(SimplonModule::Detector, "nimages", experiment.GetFrameNumPerTrigger());
SetConfigIfDifferent(SimplonModule::Detector, "ntrigger", experiment.GetNumTriggers());
SendDetectorCommand(SimplonDetectorCommand::Arm);
}
void DectrisSimplonClient::ConfigureDetector(const DiffractionExperiment &experiment) {
// TODO: Check if counting_mode retrigger is available
if (experiment.GetDetectorMode() == DetectorMode::DarkMask)
SetConfig(SimplonModule::Detector, "counting_mode", "normal");
else
SetConfig(SimplonModule::Detector, "counting_mode", "retrigger");
SetConfigIfDifferent(SimplonModule::Detector, "photon_energy", experiment.GetIncidentEnergy_keV() * 1e3f, 0.01f);
auto thr = experiment.GetEigerThreshold_keV();
SetConfigIfDifferent(SimplonModule::Detector, "threshold_energy", thr * 1e3f, 0.01);
switch (experiment.GetDetectorTiming()) {
case DetectorTiming::Auto:
SetConfig(SimplonModule::Detector, "trigger_mode", "ints");
break;
case DetectorTiming::Trigger:
SetConfig(SimplonModule::Detector, "trigger_mode", "exts");
break;
case DetectorTiming::Gated:
SetConfig(SimplonModule::Detector, "trigger_mode", "extg");
break;
case DetectorTiming::Burst:
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"Burst mode not supported with DECTRIS sytems");
}
SetConfig(SimplonModule::Detector, "trigger_start_delay", std::chrono::duration<float>(experiment.GetDetectorDelay()).count());
}
SimplonState DectrisSimplonClient::GetState() {
auto c = GetStatus(SimplonModule::Detector, "state");
if (!c.is_string())
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"value must be string");
std::string res = c.get<std::string>();
if (res == "na")
return SimplonState::Na;
else if (res == "ready")
return SimplonState::Ready;
else if (res == "idle")
return SimplonState::Idle;
else if (res == "acquire")
return SimplonState::Acquire;
else if (res == "error")
return SimplonState::Error;
else if (res == "initialize")
return SimplonState::Initialize;
else if (res == "configure")
return SimplonState::Configure;
else if (res == "test")
return SimplonState::Test;
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
"State unknown " + res);
}
float DectrisSimplonClient::GetHumidity() {
return GetStatus(SimplonModule::Detector, "humidity");
}
float DectrisSimplonClient::GetTemperature() {
return GetStatus(SimplonModule::Detector, "temperature");
}
std::vector<uint32_t> DectrisSimplonClient::GetPixelMask() {
auto mask = GetDetCfg("pixel_mask");
if (!mask.val.is_object() || !mask.val.contains("shape") || !mask.val.contains("data"))
throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "No pixel mask");
auto shape = mask.val["shape"];
std::vector<uint32_t> pixel_mask(shape[0].get<int64_t>() * shape[1].get<int64_t>());
macaron::Decode(mask.val["data"], pixel_mask);
return pixel_mask;
}