GridBased

Grid-based X-ray reconstruction for MÖNCH. A fully-convolutional network reads a calibrated frame and predicts, for every pixel cell, whether a photon converted there, where inside the cell, and with what energy. The number of photons in a frame is an output of the model, so one set of weights covers any multiplicity and any flux, and those same weights run on a 32×32 training frame or on a full detector frame.

Design rationale, work-package split and open questions: docs/WORKFLOW_DESIGN.md.

Stage 1: 12 keV monochromatic, MOENCH040 @150 V — the point where this workflow can be compared head-to-head with the published cluster-based results (σ = 0.8 µm on MC, MTF50 = 160 lp/mm at DanMAX). Done: gate G2 on MC, knife edge at four flux points.

Stage 2 (current): polychromatic 3–30 keV, MOENCH014 @150 V — per-photon energy from the same network. Done: α(E,t)/β(E,t) charge LUT, MC training and evaluation, XRF lines at 2603 MaxIV (energy resolution and efficiency against the classical 3×3 sum), and a scan of network sizes. Default model: Results/260919_poly3to30keV_Moench014_c96_00, [32, 64, 96, 96].

---

Pipeline

McGeneration single-photon stamps          (12 keV: existing, 10 M events, noiseless 5×5)
  or α(E,t)/β(E,t) charge-cloud LUT        (3–30 keV: scripts/01_build_charge_lut.py)
        │  scripts/02_generate_frames.py   ← superpose stamps / sample clouds + noise
        ▼
MC frames + truth photon list  (HDF5)
        │  scripts/03_train.py             ← fully-convolutional GridNet
        ▼
model checkpoint
        │  scripts/04_evaluate.py          ← efficiency/purity/RMS vs occupancy
        │  scripts/07_energy_response.py   ← MC line position g(E) and width (poly)
        │  scripts/05_infer_measurement.py ← raw ADU → photon list → image
        ▼
benchmark plots, reconstructed images

Conventions

• Pixel (row r, col c) covers x ∈ [c, c+1), y ∈ [r, r+1). A photon position is a float in [0, S); its cell is (⌊y⌋, ⌊x⌋) and its in-cell offset is (x−c−0.5, y−r−0.5) ∈ [−0.5, 0.5].
• Energies are keV throughout; positions are pixels (25 µm) unless a name says _um.
• Duplicate suppression works in position space. Every cell predicts a physical position, so two cells that point at the same place — within eval.merge_distance (0.1 px in the current configs) — are merged and the stronger one kept, while two cells that point at different places both survive. Predictions that merge are always less than 1 px apart and therefore in adjacent cells, so comparing the 8 neighbours is enough and the decode costs the same as a max-pool.
• Receptive-field radius R = number of 3×3 conv layers = the halo. Cells within R of a frame border are supervised nowhere and reported nowhere, so the zero padding stays invisible to everything that is used. scripts/00_selftest.py checks that a 32×32 frame and the same content embedded in a larger frame give bit-identical predictions inside the halo — that is what makes train-small / infer-large exact — and that R is tight, the ring at R−1 already differing.
• One photon per cell sets a known ceiling. For a Poisson field of occupancy λ, a single slot per 25 µm cell can represent a fraction (1−e^−λ)/λ ≈ 1 − λ/2 of the photons: 99.5 % at 1 %, 97.5 % at 5 %, 95.2 % at 10 %, 90.6 % at 20 %, 78.7 % at 50 %. The generator reports the loss per shard as n_same_cell, and it follows the formula to a few per mille. The ceiling is not a hard bound on the measured efficiency: the offset channels are unbounded, so a cell can report a position inside its neighbour.

Data layout

One HDF5 file per shard:

dataset	shape	meaning
frames	(N, S, S) f4	calibrated frame, keV
photons	(M, 4) f4	x, y, E[keV], z[µm], frame coordinates
frame_ptr	(N+1,) i8	photons of frame i = photons[ptr[i]:ptr[i+1]]
occupancy	(N,) f4	occupancy each frame was thrown at

Splits draw from disjoint stamp files (train 0–11, val 12–13, test/eval_scan 14–15), so every single-photon event belongs to exactly one split.

Running

Training, evaluation and inference need an environment with PyTorch (conda activate pytorch_nightly); generation needs numpy/h5py/omegaconf.

cd /home/xie_x1/MLXID/GridBased

# 0. self-tests (coordinates, D4, translation equivariance, position-space merging)
python scripts/00_selftest.py

# 1. generate MC frames
python scripts/02_generate_frames.py --config Configs/generate_frames_12keV.yaml
#    subsets / quick check:
python scripts/02_generate_frames.py --config Configs/generate_frames_12keV.yaml --splits eval_scan
python scripts/02_generate_frames.py --config Configs/generate_frames_12keV.yaml --smoke

# 2. train   (-u keeps progress live when the output is redirected)
python -u scripts/03_train.py --config Configs/train_grid_12keV.yaml
python scripts/03_train.py --config Configs/train_grid_12keV.yaml --epochs 3 --tag smoke

# 3. evaluate the occupancy scan (0.5 / 1 / 2 / 5 / 10 / 20 / 50 %)
python scripts/04_evaluate.py --checkpoint Results/<exp>/Models/best.pth

# 4. run over measured data
python scripts/05_infer_measurement.py --config Configs/infer_measurement_12keV.yaml \
    --run Edge_3filters_pos0_12keV

# polychromatic 3-30 keV, MOENCH014: charge LUT (+ its validation), frames, training
python scripts/01_build_charge_lut.py --config Configs/charge_lut_Moench014_3to30keV.yaml
python scripts/01b_validate_charge_lut.py --lut data/charge_lut_Moench014_150V_3to30keV.npz \
    --outdir Results/charge_lut_3to30keV
python scripts/02_generate_frames.py --config Configs/generate_frames_poly3to30keV_Moench014.yaml
python -u scripts/03_train.py --config Configs/train_grid_poly3to30keV_Moench014_c96.yaml
python scripts/04_evaluate.py --checkpoint Results/<poly exp>/Models/best.pth
# MC energy response (line position g(E), core width, efficiency) -> energy_response.npz
python scripts/07_energy_response.py --checkpoint Results/<poly exp>/Models/best.pth
# an XRF run, with its photon list for Notebooks/XRF_*.ipynb
python scripts/05_infer_measurement.py --config Configs/infer_measurement_poly_Moench014.yaml \
    --run AgMo_1_factor_10 --save-photons
# the whole chain for several network sizes, unattended, with a comparison report
# (resumable: finished stages are skipped); see Results/overnight_poly3to30keV/
setsid nohup bash scripts/overnight_poly3to30keV.sh \
    > Results/overnight_poly3to30keV/driver.log 2>&1 < /dev/null &
python scripts/08_compare_models.py --out <dir> --model <label> <experiment> <xrf tag> ...

The previous 5–30 keV chain is Configs/charge_lut_Moench014.yaml, generate_frames_poly_Moench014.yaml and train_grid_poly_Moench014.yaml.

Training outputs land in Results/<YYMMDD>_<name>_<nn>/: config.yaml, history.json, Models/{best,last,E*}.pth, Plots/training_curves.png, Plots/eval_scan.png, Plots/eval_differential.png, eval_summary.json.

Knobs that matter

where	key	why
Configs/generate_frames_12keV.yaml	splits.train.occupancy	log-uniform 0.2 – 10 % gives one model for all fluxes; {mode: fixed, value: …} specialises it to one.
	frame.size	32 px training frame; the valid region is (32−2R)² = 56 % of it.
	noise.pixel_keV	0.13 keV, which is what the measured runs deliver once the column common mode is removed.
	noise.common_mode_*	available for studying an uncorrected readout; the measurement pipeline removes common mode from the data directly.
	edge_augmentation.fraction	frames containing a sensor boundary, which shrinks the discarded detector border from R to 1 px.
Configs/train_grid_12keV.yaml	model.channels	length sets R (and the halo); width sets capacity.
	model.norm	none keeps each cell's prediction a function of its own receptive field, which is what makes train-small / infer-large exact; group is there if training ever needs stabilising.
	loss.w_ene	0.1 — at 12 keV the energy head is nearly constant; it earns its weight in the polychromatic extension.
	eval.threshold	the efficiency/purity working point; 04_evaluate.py sweeps it.
Configs/generate_frames_poly3to30keV_Moench014.yaml	spectrum.train	flat 3–30 keV. The energy head is biased within ~0.75 keV of both edges, so the range should reach ~1 keV past the lines of interest; below 3 keV the MC also lacks the entrance-window physics.
Configs/train_grid_poly3to30keV_Moench014_*.yaml	model.channels	[32, 64, 96, 96] is the default; see "Choosing the size".
	loss.w_ene, data.energy_scale	1.0 and 15 keV: here the energy head carries the spectrum.
Configs/infer_measurement_poly_Moench014.yaml	calibration.common_mode	column_masked for full-chip or higher-occupancy runs (see Measurement pipeline).

Results — Results/260831_12keV_Moench040_01

2.4 M training frames (61.5 M photons, occupancy log-uniform 0.2–10 %), GridNet with 92 932 parameters, 150 epochs in ~105 min on one RTX 4090; best.pth is epoch 135. This is the checkpoint every measurement run in Configs/infer_measurement_12keV.yaml is reconstructed with. Evaluated on the fixed-occupancy scan at objectness threshold 0.3, decoding in position space with a 0.1 px merge distance:

occupancy	efficiency	purity	RMS	σ68	sub-pixel non-unif.
0.5 %	99.66 %	99.90 %	0.798 µm	0.59 µm	0.0 %
1 %	99.44 %	99.86 %	0.851 µm	0.61 µm	2.8 %
2 %	99.04 %	99.76 %	0.899 µm	0.63 µm	3.6 %
5 %	97.98 %	99.41 %	1.145 µm	0.71 µm	3.5 %
10 %	96.39 %	98.27 %	1.757 µm	0.94 µm	6.6 %
20 % †	93.62 %	94.86 %	3.352 µm	1.98 µm	22.5 %
50 % †	86.74 %	89.93 %	6.469 µm	6.03 µm	89.7 %

† outside the 0.2–10 % training range; the model is extrapolating. Both points were added to eval_scan to find where the chain breaks, and it breaks in the sub-pixel phase long before it breaks in efficiency: the excess rms of the phase histogram goes 6.6 % → 22.5 % → 89.7 % at 10 / 20 / 50 %, and the shape is the R1 failure mode the metric was written for. Binned 10x10 over the cell, the centre 2x2 bins sit at 1.20 / 1.43 / 3.52 times flat while the border ring sits at 0.92 / 0.71 / 0.21, so by 50 % the predictions have collapsed onto the cell centres and the sub-pixel information is gone even though 86.7 % of the photons are still "found". See the G2 section for how to read that efficiency.

For reference, the cluster-based single-photon model reaches σ = 0.8 µm on 12 keV MC and 96–98 % efficiency at 0.7–1.7 % occupancy, with the photon count supplied by the cluster energy. Position resolution matches on the first attempt.

The predecessor Results/260830_12keV_Moench040_00 — 120 epochs on a 10x smaller training set, decoded with a 3×3 local maximum — reached 95.5 / 80.8 / 66.5 % efficiency at 1 / 5 / 10 % occupancy. Nearly all of that deficit was the local maximum itself, which silently imposes a 2 px minimum separation between reported photons; src/decode.py documents why it was replaced. The offset channels are unbounded, so a cell may report a position outside its own footprint and the position-space decode is not held to the one-photon-per-cell ceiling (~95 % at 10 % occupancy) that bounded the old one. A finer output grid — cells of 25/s µm — remains the lever if the grid itself ever becomes the limit.

Baseline comparison — gate G2

scripts/06_baseline_comparison.py runs three chains over the same simulated frames and scores them against the same truth with the same matcher (grid model 260831_12keV_Moench040_01/Models/best.pth, epoch 135, 2000 frames per point, valid region = frame minus the halo). The scan runs to 50 % occupancy, five times past the top of the 0.2–10 % training range, to find where each chain stops working:

occupancy	chain	efficiency	purity	RMS	σ68
0.5 %	grid	99.61 %	99.90 %	0.805 µm	0.593 µm
	cluster + 1/2/3-photon CNNs	99.64 %	99.73 %	0.891 µm	0.595 µm
	cluster + eta interpolation	87.15 %	99.86 %	1.430 µm	1.161 µm
1 %	grid	99.44 %	99.89 %	0.860 µm	0.610 µm
	cluster + CNNs	98.26 %	99.37 %	1.264 µm	0.620 µm
	cluster + eta	75.49 %	99.90 %	1.391 µm	1.078 µm
2 %	grid	99.01 %	99.79 %	0.895 µm	0.624 µm
	cluster + CNNs	91.79 %	98.00 %	1.886 µm	0.675 µm
	cluster + eta	58.70 %	99.84 %	1.374 µm	1.038 µm
5 %	grid	97.96 %	99.41 %	1.141 µm	0.710 µm
	cluster + CNNs	55.76 %	92.58 %	3.522 µm	0.926 µm
	cluster + eta	26.27 %	99.87 %	1.608 µm	1.082 µm
10 %	grid	96.37 %	98.29 %	1.771 µm	0.945 µm
	cluster + CNNs	16.12 %	85.83 %	5.083 µm	1.643 µm
	cluster + eta	6.99 %	99.88 %	2.001 µm	1.279 µm
20 %	grid	93.56 %	94.84 %	3.340 µm	1.979 µm
	cluster + CNNs	0.98 %	(84.85 %)	—	—
	cluster + eta	0.57 %	(100 %)	—	—
50 %	grid	86.77 %	89.92 %	6.470 µm	6.025 µm
	cluster + CNNs	0.00 %	—	—	—
	cluster + eta	0.00 %	—	—	—

At 0.5 % the grid chain and the cluster chain agree on efficiency and the grid chain is slightly sharper; from 1 % upwards the gap opens, because merged clusters fall outside all three energy windows and are dropped. The published reference points sit where they should: 0.8 µm for the cluster-based CNN and ~1.1 µm for eta at 12 keV.

Above 10 % the cluster chains are not merely worse, they are gone, and it happens twice over. On 200 of these frames the finder itself returns 3446 raw clusters at 2 %, 5229 at 10 %, but only 1449 at 20 % and 213 at 50 %: the charge has merged into a few large connected objects rather than many small ones. Of what it does return, the fraction that falls inside one of the three single-/double-/triple-photon energy windows goes 76.5 % → 41.7 % → 22.2 % → 0.9 %. The two effects multiply, leaving 10 / 0 / 0 usable clusters per 2000 frames at 50 % against 21 348 / 3974 / 850 at 2 %. The purity and resolution of those rows are computed over a handful of matches and mean nothing, which is why they are bracketed or left out.

The two cluster chains do not depend on the grid checkpoint, so the five rows up to 10 % are byte-identical to the earlier comparison run at epoch 100 of 260831_12keV_Moench040_00; the grid rows there moved by at most +0.7 points of efficiency and −0.9 points of purity at 10 %.

The grid chain keeps producing an image at 20 % and 50 %, but its own numbers have to be read carefully there:

• the model is extrapolating — it never saw anything above 10 % occupancy in training;
• the matcher accepts a prediction within 1 px (25 µm) of a true photon, and at 50 % occupancy that circle contains ~1.6 photons on average, so a prediction that is merely near a photon still counts. 86.77 % is therefore an upper bound on "found the right photon", not a resolution claim;
• the honest signal is σ68: 0.593 µm at 0.5 %, then 0.945 / 1.979 / 6.025 µm at 10 / 20 / 50 %. At 50 % the matched positions are spread over a quarter of the 25 µm matching window, so the chain detects that photons are there without locating them.

The generator reports how many photons share a cell with another, which is the ceiling a one-photon-per-cell output grid could reach: 95.20 % at 10 %, 90.62 % at 20 %, 78.69 % at 50 %. The measured efficiency sits above that ceiling at every point (96.37 / 93.56 / 86.77 %) because the offset channels are unbounded and a cell may report a position inside its neighbour — the same effect noted in the Results section. A finer output grid is what would turn that from an artefact of the matching radius into real resolution at these fluxes.

Gate G2 asked for σ ≤ 0.8 µm on 12 keV MC and ≥ 96 % efficiency at 1.7 % occupancy with purity at least as good as the cluster chain. The scan has no 1.7 % point, so the claim is read off the harder 2 % one: 0.805 µm at 0.5 %, 99.01 % efficiency and 99.79 % purity at 2 %, against 91.79 % and 98.00 % for the cluster chain there.

The comparison needs both torch and aare, so it runs in the mlxid_demo environment. On these 32×32 frames a VarClusterFinder is built per frame: reusing one, as _processFrames does, is exact at its 101×101 ROI (verified on real measurement frames) but overruns the internal hit buffer on frames this small.

Measurement pipeline

src/measurement.py + scripts/05_infer_measurement.py take a run from raw ADU to a photon list in a single pass, with pedestal subtraction, common-mode removal and the 3D-LUT all running on the GPU.

python scripts/05_infer_measurement.py --config Configs/infer_measurement_12keV.yaml \
    --run Edge_3filters_pos0_12keV
python scripts/05_infer_measurement.py --config ... --run Flat_3filters_pos0_12keV
python scripts/05_infer_measurement.py --config ... --run Pedestal_3filters_pos0_12keV  # fake-rate test

Each run writes, as .npy and .png: binned_image (reconstruction at pixel_size / binning_factor), native_image (the same photons rebinned to the 25 µm pitch), analog_image (mean calibrated energy per frame — the classical native readout), plus comparison.png with the last two side by side, the energy spectrum, the per-frame photon count and summary.json.

summary.json reports the occupancy twice: averaged over the valid region, and inside the illuminated area (pixels above half the 99th-percentile density). For a knife-edge or slit run the second number is the one that describes the flux the model sees.

Measured on Edge_3filters_pos0_12keV (MOENCH040 @150 V, 12 keV, 140 files, 14 M frames, 287 GB):

GPU calibration vs UsefulFuncs.convertAduFrameToEnergyFrame	agrees to 2.4e-6 keV (float32 rounding)
pixel noise, raw	0.1401 keV
pixel noise, column common mode removed	0.1291 keV — the 0.13 keV the model was trained with
frame-mean fluctuation	28.5 eV → 1.0 eV after the correction
pedestal stability over the whole run	−2 ADU = 0.015 keV, a tenth of the noise
photon occupancy	0.66 % inside the illuminated area (0.16 % averaged over a ROI that is mostly knife-edge shadow). Matches the 0.69 % of the published DanMAX working point and sits inside the 0.2–10 % training range.
fake rate on dark frames (cross-run pedestal, p>0.3)	0 photons in 1000 frames
throughput	5200–5900 frame/s while sharing the GPU with a training job → ~45 min for the full 14 M-frame run

The MOENCH040 LUT is (1740, 400, 400), so convertAduFrameToEnergyFrame takes its else branch (ADU from −1000 in steps of 10, signed natively). The MOENCH014 LUT is the 1640-bin variant that starts at 0 ADU and handles negative values through |adu| and the sign; GpuCalibrator implements both (agreement with the numpy reference: 2e-6 keV).

Common mode column_masked takes the column median over the pixels without charge. The plain median is pulled up by the photons in the column, which on the full-chip XRF runs (0.7–1.3 % occupancy) leaves every pixel 4–7 meV low — tens of meV on a 3×3 sum, growing with the flux. At the 101×101 knife-edge ROI (~0.8 hit pixels per column) the effect is negligible and column is kept.

Inference speed (RTX 4090)

model	frame	precision	model only	end-to-end run
12 keV [32, 64, 64, 64]	101×101	FP32 (TF32)	11 665 frame/s (119 Mpx/s)	7 805 frame/s (80 Mpx/s)
		FP16 + channels-last	28 991 frame/s (296 Mpx/s)	12 265 frame/s (125 Mpx/s)
5–30 keV [32, 64, 128, 128]	400×400	FP32 (TF32)	428 frame/s (68 Mpx/s)	360 frame/s (58 Mpx/s)
		FP16 + channels-last	973 frame/s (156 Mpx/s)	694 frame/s (111 Mpx/s)
3–30 keV [32, 64, 96, 96] (default)	400×400	FP16 + channels-last	1 172 frame/s (188 Mpx/s)	777 frame/s (124 Mpx/s)

At the same frame size the 12 keV model runs 1.7x more pixels per second than the 5-30 keV one. End-to-end keeps a smaller share of the model speed on 101×101 frames (42-67 %) than on 400×400 frames (71-84 %). Per frame the loop adds ~0.05 ms of work that does not scale with the frame size: decode splits the batch frame by frame, and the energy histogram and image accumulation run once per frame. In FP16 that is more than the 0.04 ms the 12 keV model itself needs per 101×101 frame.

inference.precision: fp16 runs the network under FP16 autocast in channels-last layout (predict_fp16). An FP16 head alone would round every energy to the FP16 grid: 7.3 meV steps below 15 keV, 14.6 meV above. That is harmless for the resolution, but a 50 eV spectrum then shows a comb. So the offsets and energy of the cells above threshold, about 1 % of the cells, are recomputed in FP32 from the same features. That costs nothing measurable; converting the whole feature map to FP32 for the head would cost 15 %. Both models are trained with AMP, and the FP16 photon list matches the FP32 one:

• on MC, efficiency, purity, position RMS/σ68 and the energy core width agree to the last digit quoted;
• on AgMo/CuCrCo, 99.997 % of the photons are identical, with positions within ~20 nm. The only differences are the 0.03 % of photons at a cell border, where the two cells claiming the photon swap.

The poly config uses FP16; the 12 keV config still runs FP32, as its published numbers were made that way. In the FP16 poly loop the model takes 67 % of the time, decode 12 % and calibration 10 %. Reading the raw file runs in a prefetch thread. TF32 and batches above 32 bring nothing, because these convolutions are bandwidth-bound.

Full-size MÖNCH, 800×1200 (virtual frames tiled 2×3 from real AgMo raw frames, ~6500 photons per frame; the Moench014 LUT tiled to 1640×800×1200 = 6.3 GB on the GPU; frames served from RAM, so file reading is not included):

model	model only, FP32	model only, FP16	end-to-end, FP16
poly [32, 64, 96, 96] (default)	90 frame/s	198 frame/s	144 frame/s
poly [32, 64, 128, 128]	72 frame/s	157 frame/s	—
poly [32, 64, 64, 64]	125 frame/s	287 frame/s	—
12 keV [32, 64, 64, 64]	124 frame/s	302 frame/s	—

Per 800×1200 frame, end to end in FP16 with the default model:

• upload 0.25 ms;
• calibration 0.54 ms;
• model 4.95 ms;
• decode 1.0 ms;
• image + histogram 0.19 ms.

The pixel rate is the same as at 400×400 (≈ 190 Mpx/s for the model), and batch size (4–16) makes no difference. The FP32 path needs ~15–22 GB at batch 16 next to the LUT; FP16 needs 12 GB. Reading the raw file needs 1.9 MB per frame, 280 MB/s at 144 frame/s.

The last two 128-channel layers carry 92 % of the MACs of [32, 64, 128, 128]. Narrower variants were trained and compared (see "Choosing the size"), and [32, 64, 96, 96] replaced it as the default at no measurable cost.

Knife-edge test — Notebooks/KnifeEdge_*.ipynb

Four flux points of the same DanMAX knife-edge scan, one notebook each, all with the same ROI and the same ESF and fit as Plotting/fig_knife_edge.py --fit odr (generate_presampled_esf + get_odr_fitting of DeepLearning/src/funcs_postprocess.py, ESF bins 0.3 µm, symmetric ±4σ window), so the only thing that differs between the rows is the reconstruction. The edge sits in the same place at every flux point, so one ROI serves all four and the notebooks are directly comparable:

notebook	measurement	frames	occupancy (illuminated)	classical chains available
KnifeEdge_0filters	{Edge,Flat}_0filters_12keV	3.5 M	9.42 %	none — this point was never clustered
KnifeEdge_1filters	{Edge,Flat}_1filters_12keV	7 M	3.90 %	eta only (the CNN images are older model versions)
KnifeEdge_2filters	{Edge,Flat}_2filters_12keV	7 M	1.59 %	eta + the 1/2/3-photon CNNs
KnifeEdge_3filters_pos0	{Edge,Flat}_3filters_pos0_12keV	14 M	0.66 %	eta + the 1/2/3-photon CNNs

LSF width σ [µm], with MTF50 [lp/mm] of the fitted Gaussian in brackets:

reconstruction	0 filters	1 filter	2 filters	3 filters
eta interpolation	—	1.506 ± 0.023 (124)	1.483 ± 0.024 (126)	1.536 ± 0.019 (122)
cluster + CNN, 1 photon	—	—	1.352 ± 0.022 (139)	1.355 ± 0.016 (138)
cluster + CNN, ≤2 photons	—	—	1.372 ± 0.022 (137)	1.380 ± 0.016 (136)
cluster + CNN, ≤3 photons	—	—	1.386 ± 0.018 (135)	1.381 ± 0.017 (136)
GridBased 12 keV	1.473 ± 0.017 (127)	1.388 ± 0.020 (135)	1.346 ± 0.019 (139)	1.360 ± 0.019 (138)
GridBased poly 2–20 keV	1.856 ± 0.037 (101)	1.551 ± 0.028 (121)	1.391 ± 0.018 (135)	1.420 ± 0.021 (132)

The ODR model is A/2 (1 + erf((x - c)/(sqrt(2) sigma))) + B, so σ is the width of the Gaussian LSF and MTF50 = 1000 sqrt(ln 2 / 2) / (pi sigma). The GridBased rows are the self-flattened reconstructions. (Until 2026-09-28 this table took the fitted σ for √2 σ, so its MTF50 was √2 too high, and binned the ESF with an older copy of generate_presampled_esf that made σ ~3 % too small.) Photon counts are compared within a flux point only; every chain there sees the same beam, so their ratio is the ratio of quantum efficiencies and the relative DQE ∝ QE·MTF² is well defined without the absolute flux. MTF(200 lp/mm) is measured from the LSF (Plotting/knife_edge_numbers.py):

flux point	GridBased vs ≤3-photon chain	GridBased vs eta	sub-pixel excess rms
0 filters, 9.42 %	—	—	18.3 % → 1.6 %
1 filter, 3.90 %	—	2.46x photons, 3.24x DQE	19.8 % → 3.0 %
2 filters, 1.59 %	1.044x photons, 1.17x DQE	1.47x photons, 2.82x DQE	21.6 % → 3.8 %
3 filters, 0.66 %	1.014x photons, 1.23x DQE	1.19x photons, 2.60x DQE	22.6 % → 4.3 %

At the two low-flux points, where the cluster chain is in its comfort zone, the grid chain matches the sharpest cluster chain to within the errors and collects as many photons as the ≤3-photon chain or slightly more, which puts its DQE at 200 lp/mm 17–23 % above that chain and 2.6–2.8x above eta. The interesting part is what happens above it: there is no comparable cluster + CNN image beyond 1.59 % — at 3.90 % only older model versions were ever run, at 9.42 % the measurement was never clustered at all — while the grid chain keeps resolving the edge: σ stays at 1.35–1.36 µm up to 1.59 % and grows to 1.39 and 1.47 µm at 3.90 and 9.42 %.

The polychromatic 2–20 keV MOENCH040 model (KnifeEdge_poly2to20keV_Moench040.ipynb, same runs and processing) is 3–4 % wider at the two low-flux points, on par with the ≤3-photon chain, and falls behind above 4 %, where its fit also degrades (χ²/NDF 3.4 and 7.4). Its 12 keV line moves from 12.05 to 11.64 keV with flux; the plain column common mode is the suspect (on the Siemens star the masked one moved the 15 keV line from 14.45 to 15.01 keV), not yet tested on the knife edge.

Sub-pixel flattening is about image uniformity rather than resolution: it takes the sub-pixel phase from ~20 % to ~2–4 % excess rms.

The ESF width depends strongly on the ROI: a band of 500 rows across this strongly slanted edge gives a width 1.6x that of the published 230-row ROI. Cross-method comparisons are only meaningful at a fixed ROI.

Results — polychromatic, default Results/260919_poly3to30keV_Moench014_c96_00

1.2 M training frames, flat 3–30 keV, occupancy log-uniform 0.2–10 %. Every photon gets its own energy, conversion depth and charge cloud from the α(E,t)/β(E,t) table data/charge_lut_Moench014_150V_3to30keV.npz (WP1.1, validated by 01b_validate_charge_lut.py). GridNet channels [32, 64, 96, 96] (R = 4, 157 636 parameters), 300 epochs, best.pth = epoch 224. It has been the default in infer_measurement_poly_Moench014.yaml since 2026-09-19. Fixed-occupancy scan at threshold 0.3:

occupancy	efficiency	purity	RMS	σ68	energy σ68	sub-pixel non-unif.
0.5 %	99.67 %	99.73 %	1.70 µm	0.99 µm	0.28 keV	22.2 %
1 %	99.42 %	99.54 %	1.78 µm	1.04 µm	0.29 keV	22.2 %
2 %	98.88 %	99.19 %	1.89 µm	1.08 µm	0.31 keV	21.8 %
5 %	97.25 %	98.07 %	2.29 µm	1.28 µm	0.42 keV	18.3 %
10 %	94.37 %	96.22 %	2.96 µm	1.70 µm	0.89 keV	16.3 %

Efficiencies include the 3–5 keV photons (97–98 % found), which the 5–30 keV model never saw. Against energy (scripts/07_energy_response.py, occupancy ≤ 2.5 %, 4.3e5 matched photons, 0.25 keV bins):

E_true	core σ	line position g(E) − E	tails, |ΔE| > 1 keV	efficiency
3.1 keV	0.13 keV *	+125 meV *	2.8 %	97.5 %
3.6 keV	0.23 keV	+16 meV	2.3 %	97.6 %
4.1 keV	0.26 keV	+4 meV	2.3 %	98.0 %
5.4 keV	0.26 keV	−12 meV	3.0 %	98.3 %
7.9 keV	0.26 keV	0 meV	4.0 %	98.8 %
12.1 keV	0.28 keV	−7 meV	4.3 %	99.2 %
17.4 keV	0.27 keV	−11 meV	4.4 %	99.5 %
22.1 keV	0.30 keV	0 meV	4.7 %	99.5 %
27.6 keV	0.31 keV	−5 meV	4.9 %	99.7 %
29.6 keV	0.24 keV *	−83 meV *	7.0 %	99.6 %

* within ~0.75 keV of a training edge, where the line is shifted and the energy scale compressed. Between 4 and 28 keV the line position stays within 25 meV.

The energy_bias_keV of eval_summary.json is the mean per 2.5 keV bin. It is set by the 3–7 % of photons in the tails, which the bounded range cuts from one side, so it is pulled towards the centre (~+0.1 keV at 5–12 keV) even where the line does not move. Correcting per photon with the mean is therefore wrong; the notebooks invert the core position g(E) instead. With the 3–30 keV model that correction only matters below ~4 keV.

Choosing the size — Results/overnight_poly3to30keV/report

Four networks trained overnight on the same 3–30 keV frames (scripts/overnight_poly3to30keV.sh, report by scripts/08_compare_models.py):

channels	params	FP16 frame/s (400×400)	MC 1 %: RMS / E σ68 / purity	MC 10 %: E σ68 / purity	XRF FWHM Co / Cu / Mo / Ag [keV]	dark-run photons / 1000 frames
[32, 64, 128, 128]	240 772	965	1.772 µm / 0.290 keV / 99.55 %	0.872 keV / 96.28 %	0.62 / 0.72 / 0.81 / 0.94	0
[32, 64, 96, 96]	157 636	1 185	1.777 µm / 0.291 keV / 99.54 %	0.889 keV / 96.22 %	0.62 / 0.72 / 0.82 / 0.95	0
[32, 64, 64, 64]	92 932	1 761	1.785 µm / 0.292 keV / 99.52 %	0.919 keV / 96.17 %	0.63 / 0.72 / 0.82 / 0.95	15 (all 2.3–3.2 keV)
[32, 64, 128] (R = 3)	93 188	1 712	1.790 µm / 0.297 keV / 99.37 %	1.040 keV / 95.64 %	0.63 / 0.73 / 0.82 / 0.93	8 (2.4–3.7 keV)

[32, 64, 96, 96] matches the widest network on every number and is 23 % faster. (The XRF runs of this comparison were redone with clean FP16 energies, see Inference speed.) [32, 64, 64, 64] loses ~5 % of energy resolution at 10 % occupancy and reports photons on dark frames below 3.2 keV. Three layers (R = 3) cost the most at high occupancy.

Previous default — 5–30 keV [32, 64, 128, 128], Results/260901_poly5to30keV_Moench014_00

Same architecture and data volume, flat 5–30 keV, best.pth = epoch 290. MC at 1 % occupancy: 99.52 % efficiency, 99.59 % purity, 1.76 µm RMS, 0.29 keV energy σ68. Its 5 keV edge shifted Cr Kα (5.41 keV) by +52 meV on MC and compressed the scale there (g′ = 0.82). On data that meant +33 meV with a 0.18 keV core, which is why the range was extended. Its XRF reconstructions carry the tag _poly5to30.

XRF test — Notebooks/XRF_*.ipynb

Two flood-field runs from the 2603 MaxIV XRF series (MOENCH014 @150 V, full 400×400 chip, slit factor 10), reconstructed with the default 3–30 keV [32, 64, 96, 96] model and Configs/infer_measurement_poly_Moench014.yaml:

• AgMo_1_factor_10: 0.69 % occupancy, 1.05e8 photons;
• CuCrCo_1_factor_10: 1.29 % occupancy, 1.99e8 photons, 12 keV beam.

The reference is the classical 3×3 sum around a local maximum on the same calibrated frames (first 10 k frames), with no model. The yaml carries more samples and the full flux series (slit factor 1–20). The reconstructions of the previous 5–30 keV model are kept as <run>_poly5to30, and those of all four sizes as <run>_c128/_c96/_c64/_r3.

Energy resolution. Core σ of the fitted line in keV, FWHM of the whole line shape in brackets, straight from the model (no correction):

line	E [keV]	GridBased	classical 3×3	FWHM grid / 3×3	MC core σ
Cr Kα	5.41	0.245 (0.58)	0.398 (0.94)	0.62	0.256
Co Kα	6.93	0.265 (0.62)	0.408 (0.96)	0.65	0.264
Cu Kα	8.04	0.305 (0.72)	0.441 (1.04)	0.69	0.267
elastic	12.0	0.311 (0.73) ‡	0.470 (1.11) ‡	0.66	0.279
Mo Kα	17.44	0.327 (0.82)	0.464 (1.09)	0.75	0.275
Mo Kβ	19.64	0.353 (0.83)	0.492 (1.16)	0.72	0.290
Ag Kα	22.10	0.378 (0.95)	0.489 (1.15)	0.82	0.301

‡ FWHM = 2.355 σ.

• The grid lines are 18–38 % narrower than the 3×3 sum at every energy. The core σ stays below the 0.38 keV noise floor of a fixed 3×3 sum (√9 × 0.126 keV) at every line, the gain that design decision D5 anticipated. In CuCrCo that separates Cu Kα from Co Kβ; the 3×3 sum shows Cu only as a shoulder.
• Cr and Co match the MC within ±5 %. Cu, the elastic line and the lines at 17–22 keV are 12–26 % wider, an extra term that grows with E (~1 % of E) and that the MC noise model (0.13 keV per pixel) does not contain. Pixel-to-pixel gain residuals of the LUT would do this (WP1.3).
• The previous 5–30 keV model put Cr Kα 0.4 keV above its training edge and compressed it (0.182 keV core, 0.48 keV FWHM, +33 meV); the MC correction restored the width but overshot the position. With the 3 keV edge the line needs no correction.

Line positions. The grid puts every line within 0.04 keV of the tables (Cr Kα +0.01 keV). The classical 3×3 sum reads 0.05–0.10 keV high: the MOENCH014 LUT (2025-03-05) is 0.3–1 % high (~0.9 % at 5–8 keV, 0.3–0.5 % at 17–22 keV). The MC correction changes no line by more than 12 meV.

Efficiency, without truth, against the classical finder:

	AgMo	CuCrCo	MC
isolated photons ≥ 5 keV found	99.97 %	99.99 %	99.96 % (no neighbour within 4 px, 1 %)
isolated photons 3–5 keV found	92.9 %	100.0 %	97–98 %
grid / classical photon count	1.091	1.067	—
classical clusters holding ≥ 2 grid photons	8.5 %	10.2 %	Poisson: 8.3 % / 15.0 %
reconstructed / deposited energy	0.997	0.996	—

The extra grid photons are pile-up that the 3×3 finder merges; in CuCrCo the pile-up continuum at 13–25 keV is 5–10× lower than in the classical spectrum. On a dark run (Pedestal_CuCrCo_1, 1000 full frames) the model reports 0 photons (< 2e-8 per pixel per frame).

The sub-pixel phase of these flood fields has 48 % (AgMo) and 20 % (CuCrCo) excess rms against 22 % on MC, so R1 is worse on data at high energies. Self-flattening takes both to 0.5 % and 3.2 %.

Status

• WP0 data contract, self-tests
• WP1.1 α(E,t) / β(E,t) charge-cloud LUT, MOENCH014 @150 V, with leave-one-out validation
• WP1.2 frame generator: stamp reuse at 12 keV, per-photon LUT sampling at 3–30 keV
• WP1.3 noise / physics completion (escape peak, Compton, LUT gain residuals) and gate G1 (simulated vs measured spectra); the XRF widths at 17–22 keV ask for the gain term
• WP2 halo / valid-region handling, verified exact; edge augmentation
• WP2 ablation of R and training-frame size
• WP3 model, loss, decode, training loop; 12 keV, 5–30 keV and 3–30 keV models
• WP3 network size scan on 3–30 keV (four widths / depths); [32, 64, 96, 96] is the default
• WP4 metrics: efficiency/purity, RMS/σ68, energy, pair resolution, sub-pixel uniformity
• WP4 baseline comparison against eta interpolation and the 1/2/3-photon models (gate G2 passed)
• WP4 gate G3: efficiency, purity and energy response tabulated over 3–30 keV; self-flattening available
• WP5.1/5.2 measurement pipeline: GPU calibration (both LUT variants, masked common mode), inference, images
• WP5.3 XRF lines (AgMo, CuCrCo): positions, energy resolution, efficiency against the classical sum
• WP5.3 energy-resolved imaging (the XRF runs are flood fields; needs a structured sample)
• WP5.4 ESF / MTF / relative DQE from the knife edge at four flux points (0.66–9.42 % occupancy), against every published chain
• WP5.4 SOLEIL 15 keV Siemens star
• WP5.5 fake rate on dark runs (12 keV: 0 / 1000 frames of 101×101; poly: 0 / 1000 full frames)
• gate G4: knife-edge σ = 1.360 µm (MTF50 138 lp/mm) at 0.66 %, equal within the errors to the best published chain (CNN 1 photon, 1.355 µm, 138 lp/mm); XRF lines resolved and in place
• WP6 extensions (full-size MÖNCH, ONNX/TensorRT, online), WP7 write-up