perl_d
ff00a33806
CI / lint (push) Skipped
CI / test (3.12) (push) Skipped
CI / test (3.13) (push) Skipped
CI / test-with-beamline-plugins (pxi_bec) (push) Skipped
CI / test-with-beamline-plugins (pxii_bec) (push) Skipped
CI / test-with-beamline-plugins (pxiii_bec) (push) Skipped
20 KiB
20 KiB
Inventory of user-configurable values (aaredaq + aarecommon)
Everything that behaves like a setting but is fixed in code, plus a map of the configuration surfaces that already exist and how well the code actually lines up with them. Line numbers are from the working tree on 2026-10-13.
Scope: aaredaq/src, aarecommon/src. Pydantic request/response model defaults
are excluded per the brief; the handful of settings models whose defaults are
the live policy are listed in §6.
1. Configuration surfaces that already exist
| Surface | Mechanism | Where |
|---|---|---|
| Per-beamline YAML | cfg_get("dotted.key", default) → BeamlineYAMLConfig |
aarecommon/config/beamline.py:61, files in aarecommon/config/beamline_configs/{x10sa,x06da,x06sa,simulated}.yaml |
| Local-contact Redis config | LocalContactConfigModel stored at <BL>:local_contact_config |
aaredaq/daq/config_model.py, read config.py:1203 |
| Operator-set Redis state | <BL>:auto_params (SimpleScanParameters), <BL>:settings, <BL>:simple_input_params, <BL>:runtime:simulate:<dev> |
config.py:963, :630, :948, :1050 |
| Environment | BEAMLINE, JWT_AAREDAQ_KEY, AAREDB_SHARED_PASSWORD, AAREDB_USERNAME/AAREDB_PASSWORD, AARE_RECOVERY_CODE, AARE_BEAMLINE, USER |
aarecommon/config/beamline.py:30; beamline_dispatch/default/beamline_dispatch.py:20; aaredb.py:65,132,152,331,335,407,411; server.py:179; gui/threads/daq_worker.py:1249; aarecommon/models/auth.py:67 |
| Beam-centre calibration data | beam_centres.json |
aarecommon/config/beamline_configs/beam_centres.json |
1.1 LocalContactConfigModel — the only Redis-backed, UI-editable policy today
| Field | Default | Bounds |
|---|---|---|
mount_to_center_sleep_s |
0.0 | 0–120 |
line_scan_loop_face_y_padding_fraction_each_side |
0.5 | 0–5 |
line_scan_loop_all_y_padding_fraction_each_side |
0.5 | 0–5 |
Three fields. Almost everything in §3–§5 is a candidate to join them.
1.2 Keys the YAMLs define
x10sa.yaml / x06da.yaml share a nested schema (gui.*, shared.*,
daq.hardware.*, daq.data_collection_settings.*, daq.auto_raster.*).
simulated.yaml is a flat, unrelated schema (redis_host, smargon_base,
tell_url, …) — see §2.3.
2. Config-integrity findings (code vs YAML)
Produced by diffing every cfg_get("…") key in code against the flattened YAMLs.
2.1 Keys read in code that no YAML defines — the code default always wins
| Key read | Where | Falls back to | Likely intended key |
|---|---|---|---|
daq.data_collection_settings.default_rotation_settings.incr_omega_deg |
daq.py:2334 |
0.2 | YAML has increment_omega_deg |
daq.data_collection_settings.default_raster_settings.{dtz,exp_time_s,transmission} |
gui/scan_logic/raster_grid_manager.py:143-149 |
200.0 / 0.01 / 1.0 | YAML has default_raster_scan_settings |
daq.data_collection_settings.default_screening_settings.exp_time_s |
gui/panels/rotation_data_collection.py:116 |
0.1 | no screening block exists in any YAML |
daq.auto_raster.exclude_pin_x_overlap |
operations/common/ml_bounding_box.py:475 |
True | — |
daq.auto_raster.exclude_pin_max_overlap_fraction |
operations/common/ml_bounding_box.py:480 |
0.75 | — |
daq.hardware.jfjoch_url |
aarecommon/config/beamline.py:45-49 |
per-beamline host | YAML has shared.jfjoch.jfjoch_url, which beamline.py:77 reads — two readers, two keys |
endpoints.smargon_base, epics.pv_prefix |
aarecommon/config/beamline.py:64-65 |
None | legacy |
2.2 Keys defined in a YAML that nothing reads — dead config
| File | Key | Note |
|---|---|---|
| x06da | daq.detector_limit_modifier = 2.0 |
Code reads daq.detector_distance_limit_modifier (daq/devices.py:51). x10sa spells it right. x06da silently runs with modifier 1.0. |
| x06da | daq.auto_raster.line_scan_y_padding_fraction = 0.15 |
Superseded by the two *_each_side fields in LocalContactConfigModel |
| x06da | daq.hardware.zoom_min = 1, zoom_max = 1000 |
never read |
| x06da | daq.hardware.sam_cam = 'white_balance_ratio = 1.233' |
never read |
| both | daq.db.aaredb_url |
aaredb.py:44 hard-codes the host as a default arg instead |
| both | daq.maximum_flux |
never read |
| both | beamline_id |
never read |
2.3 Per-beamline gaps
- x10sa has no
daq.auto_raster.*block, so grid padding (0.15/0.15/=top) andinclude_crystal=Falsecome from code (ml_bounding_box.py:277-279, 455). May be intentional; it is invisible. gui.cameras.secondary_beamline_camera_urlexists only in x06da and its only reader is commented out (gui/main_window.py:520).- Under
BEAMLINE=SIMULATEDeverycfg_getmisses becausesimulated.yamluses a different schema, so the simulated beamline runs entirely on the in-code fallbacks in §3.
2.4 Same concept, different fallbacks
| Concept | Values | Where |
|---|---|---|
| Default raster dtz | 200 · 250 · 200.0 | daq.py:2093, daq.py:2318, raster_grid_manager.py:149 |
| Default rotation dtz | 250 (daq) · 110 (dispatch) · 150 (SimpleScanParameters) |
daq.py:2330, beamline_dispatch/default:60, aarecommon/models/models.py:700 |
| Face-detection sweep | model default steps=14, step_size=15 vs inline steps=7, step_size=30 |
operations/face_detection/models.py:14-16 vs daq.py:2501 |
3. Hard-coded values — automation & beamline policy
These are the strongest candidates for LocalContactConfigModel or YAML.
3.1 Automation sequence (daq.py)
| Value | Location | Note |
|---|---|---|
MIN_SPOTS_LOW_RES_THRESHOLD = 10.0 |
daq.py:280 |
crystal-present threshold |
AUTOMATION_BUSY_TIMEOUT_S = 600 |
daq.py:281 |
|
MANUAL_RASTER_BUSY_TIMEOUT_S = 600 |
daq.py:282 |
|
AUTO_RASTER_MAX_IMAGES = 4500 |
daq.py:283 |
grid is scaled down to fit (ml_bounding_box.py:132) |
AUTO_RASTER_MIN_CELL_SIZE_MM = 0.005 |
daq.py:284 |
|
AUTO_RASTER_SKIP_IF_EXCEED_MAX_IMAGE_THRESHOLD = True |
daq.py:285 |
behaviour flag |
_sample_sync_min_interval_s = 2.0 |
daq.py:305 |
TELL sync rate limit |
smargon trace path /sls/mx/applications/logs/smargon_trace.csv |
daq.py:301 (also gui/panels/smargon_trace_panel.py:520) |
|
rotation min_exp_time = 0.0011 |
daq.py:1446 |
silently clamps the request |
resolution fudge new_res = 1/((1/res) + 0.01) |
daq.py:1666 |
spreadsheet target-resolution → dtz |
omega sanity bound -2000 < val < 2000 |
daq.py:1702 |
|
face detection steps=7, step_size=30, face_min_ratio=0.3 |
daq.py:2501 |
overrides the model defaults |
run token secrets.token_hex(3) |
daq.py:2518 |
6 hex chars shared by raster+rotation |
auto-raster seed grid = beam_size × 0.5 |
daq.py:2534 |
|
autofocus settle_time_s = 1.0 |
daq.py:1901 |
|
screenshot settle_time_s = 0.2 |
daq.py:2262, :2272; operations/screenshot/service.py:61, :67 |
|
mount-to-centre, unmount waits sleep(2), sleep(0.2) |
daq.py:1980, :2574, :1740, :1749 |
|
smargon_wait(timeout=180) |
daq.py:2016; raster/service.py:534 |
|
default dtz min/max fallback 100 / 1000 |
daq.py:3028-3029 |
YAML overrides exist |
| raster/rotation default fallbacks | daq.py:2312-2337 |
see §2.4 |
3.2 Mounting (operations/mounting/service.py)
| Value | Location |
|---|---|
DRY_AFTER_FAIL_COUNT = 3 |
:14 |
STOP_AFTER_FAIL_COUNT = 5 |
:15 |
mount timeout=360.0 |
:83 |
unmount timeout=60.0 |
:65, :154, :173, :186 |
magnet-sensor check timeout=10.0 (default 1.0) |
:146 (def :33) |
TELL post-mount sleep(0.1) |
:41 |
3.3 Raster (operations/raster/service.py, models.py)
| Value | Location | Note |
|---|---|---|
Y re-target threshold beam_size.y × 2.0 |
:302; and max(beam.y×2.0, cell.y×4.0) :736 |
|
line-scan cell height beam_size.y × 0.25 |
:730 |
|
per-row overhead n_y × 0.3 s |
:441 |
time estimate |
total-time cap > 1200 s |
:446 |
refuses long rasters |
detector timeout expected × 1.1 + 60 |
:428 |
|
wait_till_running(timeout=60.0) |
:506; rotation/service.py:35 |
|
aerotech wait_till_done(timeout=60) |
:438 |
|
wait_for_screenshot_s = 0.3 |
raster/models.py:26 |
|
sleep(0.2) settle |
:644 |
3.4 Auto-raster bounding box (operations/common/ml_bounding_box.py)
| Value | Location | Note |
|---|---|---|
grid padding x / y / y_bottom 0.15 / 0.15 / =y |
:277-279 |
YAML-overridable (x06da only) |
include_crystal = False |
:455 |
YAML-overridable (x06da only) |
exclude_pin_x_overlap = True |
:475 |
no YAML key anywhere |
exclude_pin_max_overlap_fraction = 0.75 |
:480 |
no YAML key anywhere |
line-scan cell = beam_size × 0.8 |
:258 |
|
line-scan exp_time_s=0.01, transmission=1.0 |
:378-379 |
fixed regardless of beamline defaults |
image-count scale (count/max)**0.5 |
:132 |
|
sleep(0.2) |
:163, :414 |
3.5 Loop centering & face detection
| Value | Location | Note |
|---|---|---|
overlap_with_pin = 0.5 |
operations/loop_centering/models.py:36 |
settings-model default |
zoom_value = 200.0 |
:38 |
|
max_attempts = 2 |
:39 |
|
target_tolerance_um = 500.0 |
:40 |
|
sleep(0.2) |
loop_centering/service.py:100 |
|
face detection steps=14, step_size=15, face_min_ratio=0.3 |
face_detection/models.py:14-16 |
overridden at daq.py:2501 |
angle wrap < 720 |
face_detection/service.py:91 |
|
centre-correction tolerance=0.2 |
:129 (def :51) |
|
MAD outlier k=3.5 |
face_detection/utils.py:39, :69 |
|
enlargement_factor = 2 |
beamcenterfit.py:44 |
comment says 50% |
ML inference conf=0.4 |
aarecommon/lc_infer_wrapper.py:60 |
detection confidence floor |
RETRY_COUNT = 3, RETRY_SLEEP_S = 0.1 |
mlbox.py:46-47 |
3.6 Per-beamline dispatch defaults (aaredaq/beamline_dispatch/)
| Value | Location |
|---|---|
rotation {exp_time_s: 0.04, dtz: 110, incr_omega_deg: 0.2} |
default/beamline_dispatch.py:60 |
auto-raster {exp_time_s: 0.04, transmission: 1.0, dtz: 200} |
default/beamline_dispatch.py:65 |
x10sa rotation transmission: 0.1; raster {0.02, 0.1, 200} |
x10sa/beamline_dispatch.py:129, :134 |
x06da rotation transmission: 1.0; raster {0.04, 1.0, 150} |
x06da/beamline_dispatch.py:126, :130 |
x06da detector tilt poni_rot1 = -0.001396263, poni_rot2 = -0.003839724 |
x06da/beamline_dispatch.py:109-110 (also aarecommon/math/diffraction_geometry.py:83-84) |
| x06da flux-vs-energy tables | x06da/beamline_dispatch.py:10, :25 |
These duplicate the YAML default_*_settings blocks with different numbers (§2.4).
4. Hard-coded values — hardware, calibration, network
4.1 Lamp brightness — three disagreeing encodings
| Encoding | Front light | Back light | Location |
|---|---|---|---|
| EPICS PV predefs | off 1.49 · half 2.0 · max 3.0 | off 0 · half 0.98 · max 1.2 | daq/devices.py:57-70 |
| % ↔ raw conversion | 0 % at ≤1.0, 100 % at ≥2.5, linear /1.5 |
100 % at 3.0, linear /3.0 |
daq.py:1767-1792 |
BEC device YAML userParameter |
on 3.0 · off 0.0 | on 1.3 · off 0.0 | pxii_bec/…/pxii-devices.yaml (out of scope, for comparison) |
Back light "max" is 1.2 in one place, 3.0 in another, 1.3 in the third.
4.2 Other hardware constants
| Value | Location | Note |
|---|---|---|
CHI_ZERO_TOLERANCE_DEG = 1e-3 |
daq/devices.py:31 |
|
dtz_mod default 1.0 |
daq/devices.py:51 |
x06da YAML key misspelled (§2.2) |
PV timeout=10.0 ×4 |
daq/devices.py:60, :67, :77, :90 |
|
det_z(..., timeout=60) |
daq/devices.py:305 |
|
smargon_wait(timeout=10.0) |
daq/devices.py:364, :371 |
|
DEFAULT_LENS_MAGNIFICATION = 10.0 |
config.py:53 |
YAML-overridable |
ABR_POS_MOUNT, ABR_OMEGA_MOUNT = 0.0 |
config.py:47-52 |
|
SMARGON_HOME (z=18), AERO_HOME |
devices/smargon.py:19-20; devices/aerotech.py:10; gui/panels/smargon_trace_panel.py:33-35 |
three copies |
SAFE_POSITION = 600 |
daq/workflows.py:13 |
detector safe dtz |
dtz_safe_position fallback 300.0 |
config.py:1246 |
x06da YAML has None |
area detector gain=15, expo=0.025 |
devices/area_detector.py:113 |
|
MIN_DTZ = 108.0 # beamline dependent |
gui/panels/scan_settings_panel.py:122 |
its own comment says so |
MAX_OMEGA_SPEED_DEG_S = 500.0 |
gui/panels/rotation_data_collection.py:19 |
|
MIN_EXP_TIME_S = {X06DA: 1/900, X10SA: 1/120} |
rotation_data_collection.py:21 |
detector frame-rate limits |
STRATEGY_MIN_EXP_TIME_S = {X06DA: 0.002}, DEFAULT_MIN_EXP_TIME_S = 0.0005, FIXED_FRAME_RATE_HZ = 100.0 |
gui/panels/smart_rotation_panel.py:41-48 |
4.3 Hosts, URLs, paths (code fallbacks)
| Value | Location | Note |
|---|---|---|
Redis port=6379, db=0 |
config.py:105 |
host is YAML-overridable, port/db are not |
{bl}-redis.psi.ch, {bl}-bec-001.psi.ch |
config.py:104; devices/bec_worker.py:97 |
|
smargon http://{x06da,x10sa}-smargopolo.psi.ch:3000 |
devices/smargon.py:25, :28 |
|
aerotech http://mx-{bl}-queue-01.psi.ch:5234 |
devices/aerotech.py:20-26; config.py:1178 |
|
TELL http://{bl}-tell.psi.ch:22222 / http://PC17488:22222 |
devices/tell_backend.py:90, :92 |
x10sa points at a PC hostname |
JFJoch sls-gpu-001/002:8080, frontends |
aarecommon/config/beamline.py:45-49; config.py:1184-1186 |
|
aaredb https://mx-aaredb-dmz-01.psi.ch/dispatcher |
aaredb.py:44; spreadsheetupdater.py:144, :168; tellupdater.py:382 |
YAML key exists but is unread (§2.2) |
data root /sls/mx/data |
server.py:2366; gui/panels/file_path_panel.py:121; gui/threads/jfjoch_viewer.py:69; operations/screenshot/service.py:41 |
four copies |
logs /sls/mx/applications/logs |
operations/screenshot/service.py:40; daq.py:301 |
|
BEC log path /tmp/logs |
devices/bec_worker.py:99 |
|
| BEC device-config path template | devices/bec_worker.py:346 |
|
uvicorn host="127.0.0.1", port=5210, workers=4 |
server.py:2625-2629 |
workers matters for the queue-loop design |
| GUI per-beamline fallbacks (DAQ URL, cert path, camera hosts, gonio id) | gui/gui.py:51-97 |
all YAML-overridable |
axis camera MJPEG params 1280x720&fps=20 |
gui/threads/axis_video_thread.py:34 |
4.4 JFJoch dataset settings (devices/jfjoch.py)
| Value | Location | Note |
|---|---|---|
pgroup fallback "p16371", sample "unknown_sample" |
:142-143 |
writes into a real pgroup when sample is unknown |
data_folder routing raw/raster, raw/screening, raw |
:152-169 |
see docs/rotation_filename_path.md §6.1 |
max_spot_count = 1000 |
:198 |
|
detect_ice_rings = True |
:199 |
|
polarization_factor = 1.0 |
:204 |
|
grid snake = True, vertical = False |
:211, :214 |
|
omega axis vector = [-1, 0, 0] |
:226 |
|
wait_till_running(timeout=60) |
:293 |
5. Timeouts, polling, sessions, auth
| Value | Location | Note |
|---|---|---|
ACCESS_TOKEN_EXPIRE_MINUTES = 24*60*7 |
daq/auth.py:30 |
JWT lifetime 1 week |
SESSION_EXPIRE_SECONDS = 600 |
auth.py:31; config.py:96 (GUI_SESSION_EXPIRE_SECONDS) |
duplicated |
BATON_REQUEST_TIMEOUT_SECONDS = 30; refusal re-arm timeout_sec=10 |
auth.py:32, :423; config.py:431 default 30, +5 grace :439 |
|
grace_seconds = 60 |
config.py:208 |
|
STAFF_GROUP = "unx-sls_mx_bs", SUPER_USERS = [...] |
auth.py:34-35 |
identity policy in code |
ALGORITHM = "HS256" |
auth.py:29 |
|
Redis lock timeout=10 |
config.py:349, :361, :383, :393, :479, :640, :645 |
hardware lock has no timeout (config.py:106) |
_ALL_PGROUPS_TTL_S = 60.0 |
server.py:76 |
|
SSE poll 0.15 s; spreadsheet/ref-tools SSE 10 s; baton 0.5 s; 0.2 s |
server.py:283, :303; :1295, :1313; :2262; :2534, :2537 |
|
SSE_RECONNECT_DELAY_S = 5 |
daq/tellupdater.py:64 |
|
spreadsheet updater sleep(5) |
spreadsheetupdater.py:187 |
|
aaredb HTTP timeout=30 |
aaredb.py:338, :414 |
|
TELL command timeout=360.0; ready 30.0; probe 1.0; poll 0.01; lazy retry 2.0 |
devices/tell_client.py:82, :87, :202, :367; :386; tell_backend.py:65; :172; :336 |
|
smargon REST timeout=2.0; initialize(360.0) / (30.0); wait(60, tol 0.01, poll 0.01) |
devices/smargon.py:43, :77; :111, :235; :214, :222 |
|
aerotech wait_till_done(60) |
devices/aerotech.py:114 |
|
fluorimeter timeout_s=30.0 |
devices/fluorimeter.py:65, :91, :98, :114 |
|
ZMQ timeout_ms=2000 |
devices/zmq_client.py:27 |
|
motor timeout=5.0 / 300.0, poll_rate=0.01 |
devices/my_motor.py:9, :63, :84, :94 |
|
filter transmission timeout=60.0, fail_on_timeout=True |
devices/filter_transmission.py:8 |
|
wait_position(timeout=60.0) |
devices/workflow_tools.py:6 |
|
workflow waits 60, sleep(10.0) |
daq/workflows.py:163, :175, :289 |
|
BEC sleep(1) |
devices/bec_worker.py:566 |
6. Analysis thresholds (aarecommon)
| Value | Location | Note |
|---|---|---|
autofocus threshold median × 3.0 |
math/autofocus.py:20 |
|
ice veto spots_ice > spots × 0.8 |
math/find_xtal.py:147 |
comment says 50 % |
min_efficiency = 1.0 |
find_xtal.py:178 |
|
min_low_res_spots = 10.0 |
find_xtal.py:344 |
same number as MIN_SPOTS_LOW_RES_THRESHOLD |
mask n_sigma = 0.5; n_clusters = 3; percentile = 60.0 |
find_xtal.py:507, :568, :587-590 |
|
coordinate equality tol = 0.1 / 0.01 |
math/coordinate.py:91, :115 |
|
exception DEFAULT_CRITICALITY flags |
errors/exception_handler.py:14, 77, 82, 90, 133, 146, 159, 168, 177, 182, 249, 275, 299, 349, 370, 375 |
decide 503 vs 422 per exception class |
logger levels (all DEBUG) |
config/logger.py:63-88 |
no env/YAML override |
7. GUI behaviour and UX policy
| Value | Location | Note |
|---|---|---|
LOW_CURRENT_THRESHOLD = 360.0 mA |
gui/widgets/message_box.py:11; used panels/sample_queue_panel.py:226 |
gates automation start/pause |
SNOOZE_SECONDS = 3600.0 |
message_box.py:12 |
|
TELL-warning recovery pause 600 × 1000 ms |
sample_queue_panel.py:59 |
|
DEFAULT_SAMPLE_ESTIMATE_S = 150.0 |
panels/automation_panel.py:43 |
ETA before data exists |
dose model KDOSE_1A_PH_UM2_GY = 2000.0, GLASS_TRANSITION_K = 200.0 |
panels/smart_rotation_panel.py:29, :36 |
|
resolution UI VISIBLE_RES_{MIN,MAX,DEFAULT}_A = 0.8 / 10.0 / 2.0, DAQ_RES_EPS_A = 0.001, DTZ_STEP_MM = 10.0, EXP_TIME_STEP_S = 0.001 |
smart_rotation_panel.py:51-59 |
|
DEFAULT_ABR_STEP_UM = 5 |
panels/abr_tweak_panel.py:12 |
|
READBACK_GRACE_S = 1.5 |
panels/illumination_panel.py:11 |
|
ENERGY_AT_TARGET_TOL_KEV = 0.001 |
panels/monochromator_panel.py:19 |
|
filename templates "{date}/{puck}/{pos}", "{sample}"; run range 1-999 |
panels/file_path_panel.py:64, :77, :92 |
{pos} bug, see rotation_filename_path.md §6.6 |
status poll 500 ms; SPREADHSEET_FREQUENCY = 25 ticks (≈5 s); baton tick 1000 ms; stream restarts 1000 ms |
gui/threads/daq_worker.py:158, :58, :217, :1897, :2077, :2351 |
|
idle timer 60_000 ms; remote-close 1000 ms |
gui/main_window.py:880, :885 |
|
status message clear 5000 ms |
gui/widgets/status_bar.py:155 |
|
automation refresh 1000 ms |
panels/automation_panel.py:53 |
|
MAX_FRAMES_IN_FLIGHT = 2 |
gui/threads/prediction_subscriber.py:28 |
|
TABLE_MAX_ROWS = 20 |
panels/smargon_trace_panel.py:36 |
8. Settings-model defaults (pydantic, listed for completeness)
Excluded from the inventory proper, but these are the live policy when nothing overrides them:
SimpleScanParameters(aarecommon/models/models.py:698-716) — schema of<BL>:auto_params;dtz=150, exp_time_s=0.02, incr_omega_deg=0.2, steps=1800, transmission=1.0.LoopCenteringSettings(operations/loop_centering/models.py:36-40) — §3.5.FaceDetectionSettings(operations/face_detection/models.py:14-16) — §3.5, overridden inline.RasterSettings.wait_for_screenshot_s = 0.3(operations/raster/models.py:26).DiffractionGeometryfield bounds (aarecommon/math/diffraction_geometry.py:12-14) — validation limits on energy/dtz/pixel size.