debug: analyse the scan grid, the real motor, and where faults cluster
CI for csaxs_bec / test (push) Successful in 1m53s

The first run against real files produced numbers that were entirely wrong,
in three separate ways. The metadata now reads correctly (num_lines,
frames_per_trigger), which is what exposed them.

**Wrong arrays.** These files hold two populations: the MCS channels, which
are the per-point scan data, and the monitored devices, sampled opportunistically
while a line runs. The monitored set has more signals, so "the length shared by
the most signals" landed on it -- and it is not on the scan grid at all. S00473
was analysed as 881x17 when the metadata says 203 x 428 = 86884, exactly the
MCS length. The grid length is now taken from the metadata when it matches an
array present in the file.

**Wrong axis.** The fast axis was chosen by how monotonically a signal ramps,
and picked `curr` -- the SLS ring current, which decays smoothly through any
window and beats the motor at its own metric. Positioners are now identified
structurally, by having a `<dev>_user_setpoint` sibling, which is what actually
makes a device a motor.

**Wrong assumption that both checks share one array.** They do not. Intensity
lives on the scan grid; motor position lives in the monitored stream at its own
rate and with its own line boundaries. The two checks now read their own data
and say so. This also surfaces a real limitation: cont_grid records roughly 8
position samples per line in S00254 against 205 data points, so a partial stall
can hide between samples. The script warns when sampling is that sparse rather
than reporting a confident negative.

**Truncated reporting hid the reported symptom.** The artefacts are said to
appear towards the end of scans, and the flagged-line list was cut after 25
entries with "... and 221 more" -- which is precisely where an end-of-scan
cluster would be lost. Flagged lines are now summarised by tenth of the scan,
with an explicit verdict on whether they cluster at the start, the end, or not
at all, and both ends of the list are printed. Bug 1 gains a matching
across-scan trend, since end-of-scan degradation is a different question from
end-of-line.

Verified against a file shaped like the real ones: MCS grid plus a sparse motor
stream, an end-of-line intensity drop, and stalls confined to the last tenth.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
menzelandClaude Opus 5 committed 2026-08-25 12:09:32 +02:00
1 parent 52fed5a008
commit 1b99ffeee9
1 file changed
+146 -4
+146 -4
View File
@@ -127,6 +127,23 @@ def read_metadata(h5: h5py.File) -> dict:
return {k: (v.decode() if isinstance(v, bytes) else v) for k, v in meta.items()}
def positioner_devices(h5: h5py.File) -> set[str]:
"""Devices that are motors, identified by having a `<dev>_user_setpoint`.
Needed because judging "which signal is the scanned axis" by shape alone
picks nonsense: the SLS ring current decays monotonically through a line and
scores better than the motor does. A setpoint sibling is what actually makes
a device a positioner.
"""
out: set[str] = set()
if DEVICES_PATH not in h5:
return out
for dev_name, dev in h5[DEVICES_PATH].items():
if isinstance(dev, h5py.Group) and any("user_setpoint" in k for k in dev):
out.add(dev_name)
return out
def monitored_names(h5: h5py.File) -> set[str]:
grp = f"{READOUT_PATH}/monitored"
return set(h5[grp].keys()) if grp in h5 else set()
@@ -274,6 +291,115 @@ def check_end_of_line(signals, grid, monitored, fast_axis) -> None:
print(" last/bulk = mean of final point / mean of interior points, over all lines.")
print(" A value near 1.00 is healthy; Bug 1 predicts a systematic shortfall.")
# End-of-*scan* behaviour is a separate question from end-of-*line*.
print(f"\n Trend across the scan (last tenth of lines vs first tenth)")
for name, arr in sorted(signals.items()):
if arr.size != lines * per_line:
continue
dev = name.split("/")[0]
if monitored and dev not in monitored:
continue
block = arr.reshape(lines, per_line)
if monotonicity(block) > 0.9:
continue
k = max(1, lines // 10)
head, tail = np.mean(block[:k]), np.mean(block[-k:])
if not np.isfinite(head) or abs(head) < 1e-30:
continue
rel = tail / head
flag = " <-- degrades towards the end" if rel < 0.95 else ""
if abs(rel - 1) > 0.02:
print(f" {name:<44} {rel:>7.3f}{flag}")
def report_distribution(flagged: np.ndarray, total: int, label: str) -> None:
"""Where in the scan the flagged lines fall.
Printing only the first N hides a cluster at the end -- and the reported
artefacts are said to appear towards the end of scans, which is exactly the
case a head-truncated list cannot show. So summarise position, then print
both ends of the list rather than just the top.
"""
if flagged.size == 0 or total <= 0:
return
tenths = np.zeros(10, dtype=int)
for i in flagged:
tenths[min(9, int(10 * i / total))] += 1
bar = " ".join(f"{c:>3}" for c in tenths)
print(f" {label} by tenth of scan (start -> end):")
print(f" {bar}")
last_tenth = tenths[-1] / max(1, flagged.size)
first_tenth = tenths[0] / max(1, flagged.size)
if last_tenth > 0.3:
print(f" -> {last_tenth:.0%} of them are in the final tenth: this clusters"
" at the END of the scan.")
elif first_tenth > 0.3:
print(f" -> {first_tenth:.0%} are in the first tenth: clusters at the START.")
else:
print(" -> spread through the scan, no obvious end-of-scan clustering.")
def check_line_motion_from_motor(signals, motors, meta) -> None:
"""Bug 2, using a real positioner trace rather than whatever looks rampy."""
print("\n BUG 2 -- did the fast axis move on every line?")
cands = {n: a for n, a in signals.items()
if n.split("/")[0] in motors and "setpoint" not in n}
if not cands:
print(" no positioner readback in this file, so this cannot be answered here.")
print(" (cont_grid records motor position only in the monitored stream;")
print(" if that is absent the file cannot show whether a line moved.)")
return
expected = int(meta["num_lines"]) if meta.get("num_lines") else None
best = None
for name, arr in sorted(cands.items()):
found = detect_lines(arr)
if found and (best is None or (expected and abs(found - expected) < abs(best[1] - expected))):
best = (name, found, arr)
if best is None:
print(f" positioner traces found ({', '.join(sorted(cands))}) but none shows")
print(" a line structure -- the axis may not have been the scanned one.")
return
name, found, arr = best
per_line = arr.size // found
print(f" fast axis : {name} ({arr.size} samples, {found} lines detected"
+ (f", metadata says {expected}" if expected else "") + ")")
if expected and found != expected:
print(f" NOTE: detected {found} lines but metadata says {expected}."
" Treat the per-line numbers below with care.")
block = arr[: found * per_line].reshape(found, per_line)
spans = np.ptp(block, axis=1)
median = float(np.median(spans))
if median <= 0:
print(" the axis does not move -- check that this is the scanned motor")
return
ratio = spans / median
print(f" samples per line : ~{per_line} median line span : {median:.6g}")
if per_line < 5:
print(f" WARNING: only ~{per_line} position samples per line. The motor trace")
print(" is far sparser than the data, so a partial stall can hide between")
print(" samples. Absence of evidence here is weak evidence of absence.")
dead = np.where(ratio < 0.1)[0]
low = np.where(ratio < 0.9)[0]
print(f" lines below 90% of median span : {len(low)}")
print(f" lines below 10% (motion absent) : {len(dead)}")
report_distribution(low, found, "lines below 90%")
report_distribution(dead, found, "lines with motion absent")
for i in list(dead[:10]) + ([-1] if len(dead) > 20 else []) + list(dead[-10:] if len(dead) > 20 else []):
if i == -1:
print(" ...")
continue
print(f" line {i:>5} span {spans[i]:.6g} ({ratio[i]:.2f}x)")
if len(dead):
print(" Collapsed spans are motion that did not happen while the scan still")
print(" reported success -- the signature we are looking for.")
elif len(low):
print(" No line is missing outright; the spread is a velocity/settling")
print(" variation rather than dropped motion.")
else:
print(" every line travelled a consistent distance.")
def check_line_motion(signals, grid, fast_axis) -> None:
lines, per_line = grid
@@ -360,7 +486,17 @@ def analyse(path: str, do_list: bool, force_lines=None, force_points=None) -> No
# Prefer the length shared by the most signals: per-point data outnumbers
# whatever else is in the file. Taking the longest signal is wrong when a
# monitor is sampled on its own clock.
n = max(lengths, key=lambda k: len(lengths[k]))
# Prefer the length the metadata says is the grid. These files hold two
# populations: the MCS channels, which are the per-point scan data, and the
# monitored devices, sampled opportunistically while the line runs. The
# monitored set is the larger of the two by signal count, so picking the
# most common length lands on the wrong data.
grid_n = None
if meta.get("num_lines") and meta.get("frames_per_trigger"):
want = int(meta["num_lines"]) * int(meta["frames_per_trigger"])
if want in lengths:
grid_n = want
n = grid_n if grid_n else max(lengths, key=lambda k: len(lengths[k]))
how = "metadata"
if force_lines and force_points:
grid, n, how = (force_lines, force_points), force_lines * force_points, "--lines/--points"
@@ -395,9 +531,15 @@ def analyse(path: str, do_list: bool, force_lines=None, force_points=None) -> No
print(" (the scan command that produced it gives both).")
return
mon = monitored_names(h5)
fast = pick_fast_axis(signals, grid)
check_line_motion(signals, grid, fast)
check_end_of_line(signals, grid, mon, fast)
motors = positioner_devices(h5)
print(f" positioners in file: {', '.join(sorted(motors)) or '(none found)'}")
# The two checks read different arrays. Intensity lives on the scan grid;
# motor position does not -- it is sampled while the line runs, at whatever
# rate the readout loop managed, so it has its own length and its own
# line boundaries.
check_line_motion_from_motor(signals, motors, meta)
check_end_of_line(signals, grid, mon, None)
def main(argv=None) -> int: