GridBased/README.md

# GridBased

Grid-based X-ray reconstruction for MÖNCH.  A fully-convolutional network reads a
calibrated frame and predicts, for **every pixel cell**, whether a photon
converted there, where inside the cell, and with what energy.  The number of
photons in a frame is an output of the model, so one set of weights covers any
multiplicity and any flux, and those same weights run on a 32×32 training frame
or on a full detector frame.

Design rationale, work-package split and open questions:
[`docs/WORKFLOW_DESIGN.md`](docs/WORKFLOW_DESIGN.md).

**Stage 1: 12 keV monochromatic, MOENCH040 @150 V** — the point where this
workflow can be compared head-to-head with the published cluster-based results
(σ = 0.8 µm on MC, MTF50 = 160 lp/mm at DanMAX).  Done: gate G2 on MC, knife
edge at four flux points.

**Stage 2 (current): polychromatic 3–30 keV, MOENCH014 @150 V** — per-photon
energy from the same network.  Done: α(E,t)/β(E,t) charge LUT, MC training and
evaluation, XRF lines at 2603 MaxIV (energy resolution and efficiency against the
classical 3×3 sum), and a scan of network sizes.  Default model:
`Results/260919_poly3to30keV_Moench014_c96_00`, `[32, 64, 96, 96]`.

---

## Pipeline

```
McGeneration single-photon stamps          (12 keV: existing, 10 M events, noiseless 5×5)
  or α(E,t)/β(E,t) charge-cloud LUT        (3–30 keV: scripts/01_build_charge_lut.py)
        │  scripts/02_generate_frames.py   ← superpose stamps / sample clouds + noise
        ▼
MC frames + truth photon list  (HDF5)
        │  scripts/03_train.py             ← fully-convolutional GridNet
        ▼
model checkpoint
        │  scripts/04_evaluate.py          ← efficiency/purity/RMS vs occupancy
        │  scripts/07_energy_response.py   ← MC line position g(E) and width (poly)
        │  scripts/05_infer_measurement.py ← raw ADU → photon list → image
        ▼
benchmark plots, reconstructed images
```

## Conventions

* Pixel `(row r, col c)` covers `x ∈ [c, c+1)`, `y ∈ [r, r+1)`.  A photon position
  is a float in `[0, S)`; its cell is `(⌊y⌋, ⌊x⌋)` and its in-cell offset is
  `(x−c−0.5, y−r−0.5) ∈ [−0.5, 0.5]`.
* Energies are keV throughout; positions are pixels (25 µm) unless a name says `_um`.
* **Duplicate suppression works in position space.**  Every cell predicts a
  physical position, so two cells that point at the same place — within
  `eval.merge_distance` (0.1 px in the current configs) — are merged and the
  stronger one kept, while two
  cells that point at different places both survive.  Predictions that merge are
  always less than 1 px apart and therefore in adjacent cells, so comparing the 8
  neighbours is enough and the decode costs the same as a max-pool.
* **Receptive-field radius R = number of 3×3 conv layers = the halo.**  Cells
  within R of a frame border are supervised nowhere and reported nowhere, so the
  zero padding stays invisible to everything that is used.
  `scripts/00_selftest.py` checks that a 32×32 frame and the same content embedded
  in a larger frame give bit-identical predictions inside the halo — that is what
  makes train-small / infer-large exact — and that R is tight, the ring at R−1
  already differing.
* **One photon per cell sets a known ceiling.**  For a Poisson field of occupancy
  λ, a single slot per 25 µm cell can represent a fraction
  `(1−e^−λ)/λ ≈ 1 − λ/2` of the photons: 99.5 % at 1 %, 97.5 % at 5 %, 95.2 % at
  10 %, 90.6 % at 20 %, 78.7 % at 50 %.  The generator reports the loss per shard
  as `n_same_cell`, and it follows the formula to a few per mille.  The ceiling is
  not a hard bound on the *measured* efficiency: the offset channels are unbounded,
  so a cell can report a position inside its neighbour.

## Data layout

One HDF5 file per shard:

| dataset | shape | meaning |
|---|---|---|
| `frames` | (N, S, S) f4 | calibrated frame, keV |
| `photons` | (M, 4) f4 | `x, y, E[keV], z[µm]`, frame coordinates |
| `frame_ptr` | (N+1,) i8 | photons of frame *i* = `photons[ptr[i]:ptr[i+1]]` |
| `occupancy` | (N,) f4 | occupancy each frame was thrown at |

Splits draw from **disjoint stamp files** (train 0–11, val 12–13,
test/eval_scan 14–15), so every single-photon event belongs to exactly one split.

## Running

Training, evaluation and inference need an environment with PyTorch
(`conda activate pytorch_nightly`); generation needs numpy/h5py/omegaconf.

```bash
cd /home/xie_x1/MLXID/GridBased

# 0. self-tests (coordinates, D4, translation equivariance, position-space merging)
python scripts/00_selftest.py

# 1. generate MC frames
python scripts/02_generate_frames.py --config Configs/generate_frames_12keV.yaml
#    subsets / quick check:
python scripts/02_generate_frames.py --config Configs/generate_frames_12keV.yaml --splits eval_scan
python scripts/02_generate_frames.py --config Configs/generate_frames_12keV.yaml --smoke

# 2. train   (-u keeps progress live when the output is redirected)
python -u scripts/03_train.py --config Configs/train_grid_12keV.yaml
python scripts/03_train.py --config Configs/train_grid_12keV.yaml --epochs 3 --tag smoke

# 3. evaluate the occupancy scan (0.5 / 1 / 2 / 5 / 10 / 20 / 50 %)
python scripts/04_evaluate.py --checkpoint Results/<exp>/Models/best.pth

# 4. run over measured data
python scripts/05_infer_measurement.py --config Configs/infer_measurement_12keV.yaml \
    --run Edge_3filters_pos0_12keV

# polychromatic 3-30 keV, MOENCH014: charge LUT (+ its validation), frames, training
python scripts/01_build_charge_lut.py --config Configs/charge_lut_Moench014_3to30keV.yaml
python scripts/01b_validate_charge_lut.py --lut data/charge_lut_Moench014_150V_3to30keV.npz \
    --outdir Results/charge_lut_3to30keV
python scripts/02_generate_frames.py --config Configs/generate_frames_poly3to30keV_Moench014.yaml
python -u scripts/03_train.py --config Configs/train_grid_poly3to30keV_Moench014_c96.yaml
python scripts/04_evaluate.py --checkpoint Results/<poly exp>/Models/best.pth
# MC energy response (line position g(E), core width, efficiency) -> energy_response.npz
python scripts/07_energy_response.py --checkpoint Results/<poly exp>/Models/best.pth
# an XRF run, with its photon list for Notebooks/XRF_*.ipynb
python scripts/05_infer_measurement.py --config Configs/infer_measurement_poly_Moench014.yaml \
    --run AgMo_1_factor_10 --save-photons
# the whole chain for several network sizes, unattended, with a comparison report
# (resumable: finished stages are skipped); see Results/overnight_poly3to30keV/
setsid nohup bash scripts/overnight_poly3to30keV.sh \
    > Results/overnight_poly3to30keV/driver.log 2>&1 < /dev/null &
python scripts/08_compare_models.py --out <dir> --model <label> <experiment> <xrf tag> ...
```

The previous 5–30 keV chain is `Configs/charge_lut_Moench014.yaml`,
`generate_frames_poly_Moench014.yaml` and `train_grid_poly_Moench014.yaml`.

Training outputs land in `Results/<YYMMDD>_<name>_<nn>/`: `config.yaml`,
`history.json`, `Models/{best,last,E*}.pth`, `Plots/training_curves.png`,
`Plots/eval_scan.png`, `Plots/eval_differential.png`, `eval_summary.json`.

## Knobs that matter

| where | key | why |
|---|---|---|
| `Configs/generate_frames_12keV.yaml` | `splits.train.occupancy` | log-uniform 0.2 – 10 % gives one model for all fluxes; `{mode: fixed, value: …}` specialises it to one. |
| | `frame.size` | 32 px training frame; the valid region is `(32−2R)²` = 56 % of it. |
| | `noise.pixel_keV` | 0.13 keV, which is what the measured runs deliver once the column common mode is removed. |
| | `noise.common_mode_*` | available for studying an uncorrected readout; the measurement pipeline removes common mode from the data directly. |
| | `edge_augmentation.fraction` | frames containing a sensor boundary, which shrinks the discarded detector border from R to 1 px. |
| `Configs/train_grid_12keV.yaml` | `model.channels` | length sets R (and the halo); width sets capacity. |
| | `model.norm` | `none` keeps each cell's prediction a function of its own receptive field, which is what makes train-small / infer-large exact; `group` is there if training ever needs stabilising. |
| | `loss.w_ene` | 0.1 — at 12 keV the energy head is nearly constant; it earns its weight in the polychromatic extension. |
| | `eval.threshold` | the efficiency/purity working point; `04_evaluate.py` sweeps it. |
| `Configs/generate_frames_poly3to30keV_Moench014.yaml` | `spectrum.train` | flat 3–30 keV.  The energy head is biased within ~0.75 keV of both edges, so the range should reach ~1 keV past the lines of interest; below 3 keV the MC also lacks the entrance-window physics. |
| `Configs/train_grid_poly3to30keV_Moench014_*.yaml` | `model.channels` | `[32, 64, 96, 96]` is the default; see "Choosing the size". |
| | `loss.w_ene`, `data.energy_scale` | 1.0 and 15 keV: here the energy head carries the spectrum. |
| `Configs/infer_measurement_poly_Moench014.yaml` | `calibration.common_mode` | `column_masked` for full-chip or higher-occupancy runs (see Measurement pipeline). |

## Results — `Results/260831_12keV_Moench040_01`

2.4 M training frames (61.5 M photons, occupancy log-uniform 0.2–10 %), GridNet
with 92 932 parameters, 150 epochs in ~105 min on one RTX 4090; `best.pth` is
epoch 135.  This is the checkpoint every measurement run in
`Configs/infer_measurement_12keV.yaml` is reconstructed with.  Evaluated on the
fixed-occupancy scan at objectness threshold 0.3, decoding in position space
with a 0.1 px merge distance:

| occupancy | efficiency | purity | RMS | σ68 | sub-pixel non-unif. |
|---|---|---|---|---|---|
| 0.5 % | 99.66 % | 99.90 % | **0.798 µm** | 0.59 µm | 0.0 % |
| 1 % | 99.44 % | 99.86 % | 0.851 µm | 0.61 µm | 2.8 % |
| 2 % | 99.04 % | 99.76 % | 0.899 µm | 0.63 µm | 3.6 % |
| 5 % | 97.98 % | 99.41 % | 1.145 µm | 0.71 µm | 3.5 % |
| 10 % | 96.39 % | 98.27 % | 1.757 µm | 0.94 µm | 6.6 % |
| 20 % † | 93.62 % | 94.86 % | 3.352 µm | 1.98 µm | 22.5 % |
| 50 % † | 86.74 % | 89.93 % | 6.469 µm | 6.03 µm | 89.7 % |

† outside the 0.2–10 % training range; the model is extrapolating.  Both points
were added to `eval_scan` to find where the chain breaks, and it breaks in the
sub-pixel phase long before it breaks in efficiency: the excess rms of the phase
histogram goes 6.6 % → 22.5 % → 89.7 % at 10 / 20 / 50 %, and the shape is the
R1 failure mode the metric was written for.  Binned 10x10 over the cell, the
centre 2x2 bins sit at 1.20 / 1.43 / 3.52 times flat while the border ring sits
at 0.92 / 0.71 / 0.21, so by 50 % the predictions have collapsed onto the cell
centres and the sub-pixel information is gone even though 86.7 % of the photons
are still "found".  See the G2 section for how to read that efficiency.

For reference, the cluster-based single-photon model reaches σ = 0.8 µm on 12 keV
MC and 96–98 % efficiency at 0.7–1.7 % occupancy, with the photon count supplied
by the cluster energy.  Position resolution matches on the first attempt.

The predecessor `Results/260830_12keV_Moench040_00` — 120 epochs on a 10x smaller
training set, decoded with a 3×3 local maximum — reached 95.5 / 80.8 / 66.5 %
efficiency at 1 / 5 / 10 % occupancy.  Nearly all of that deficit was the local
maximum itself, which silently imposes a 2 px minimum separation between
reported photons; `src/decode.py` documents why it was replaced.  The offset
channels are unbounded, so a cell may report a position outside its own
footprint and the position-space decode is not held to the one-photon-per-cell
ceiling (~95 % at 10 % occupancy) that bounded the old one.  A finer output grid
— cells of 25/s µm — remains the lever if the grid itself ever becomes the limit.

## Baseline comparison — gate G2

`scripts/06_baseline_comparison.py` runs three chains over the *same* simulated
frames and scores them against the same truth with the same matcher (grid model
`260831_12keV_Moench040_01/Models/best.pth`, epoch 135, 2000 frames per point,
valid region = frame minus the halo).  The scan runs to 50 % occupancy, five
times past the top of the 0.2–10 % training range, to find where each chain
stops working:

| occupancy | chain | efficiency | purity | RMS | σ68 |
|---|---|---|---|---|---|
| 0.5 % | grid | 99.61 % | 99.90 % | **0.805 µm** | 0.593 µm |
| | cluster + 1/2/3-photon CNNs | 99.64 % | 99.73 % | 0.891 µm | 0.595 µm |
| | cluster + eta interpolation | 87.15 % | 99.86 % | 1.430 µm | 1.161 µm |
| 1 % | grid | 99.44 % | 99.89 % | 0.860 µm | 0.610 µm |
| | cluster + CNNs | 98.26 % | 99.37 % | 1.264 µm | 0.620 µm |
| | cluster + eta | 75.49 % | 99.90 % | 1.391 µm | 1.078 µm |
| 2 % | grid | 99.01 % | 99.79 % | 0.895 µm | 0.624 µm |
| | cluster + CNNs | 91.79 % | 98.00 % | 1.886 µm | 0.675 µm |
| | cluster + eta | 58.70 % | 99.84 % | 1.374 µm | 1.038 µm |
| 5 % | grid | 97.96 % | 99.41 % | 1.141 µm | 0.710 µm |
| | cluster + CNNs | 55.76 % | 92.58 % | 3.522 µm | 0.926 µm |
| | cluster + eta | 26.27 % | 99.87 % | 1.608 µm | 1.082 µm |
| 10 % | grid | 96.37 % | 98.29 % | 1.771 µm | 0.945 µm |
| | cluster + CNNs | 16.12 % | 85.83 % | 5.083 µm | 1.643 µm |
| | cluster + eta | 6.99 % | 99.88 % | 2.001 µm | 1.279 µm |
| 20 % | grid | 93.56 % | 94.84 % | 3.340 µm | 1.979 µm |
| | cluster + CNNs | 0.98 % | (84.85 %) | — | — |
| | cluster + eta | 0.57 % | (100 %) | — | — |
| 50 % | grid | 86.77 % | 89.92 % | 6.470 µm | 6.025 µm |
| | cluster + CNNs | 0.00 % | — | — | — |
| | cluster + eta | 0.00 % | — | — | — |

At 0.5 % the grid chain and the cluster chain agree on efficiency and the grid
chain is slightly sharper; from 1 % upwards the gap opens, because merged
clusters fall outside all three energy windows and are dropped.  The published
reference points sit where they should: 0.8 µm for the cluster-based CNN and
~1.1 µm for eta at 12 keV.

Above 10 % the cluster chains are not merely worse, they are gone, and it
happens twice over.  On 200 of these frames the finder itself returns 3446 raw
clusters at 2 %, 5229 at 10 %, but only 1449 at 20 % and 213 at 50 %: the charge
has merged into a few large connected objects rather than many small ones.  Of
what it does return, the fraction that falls inside one of the three
single-/double-/triple-photon energy windows goes 76.5 % → 41.7 % → 22.2 % →
0.9 %.  The two effects multiply, leaving 10 / 0 / 0 usable clusters per 2000
frames at 50 % against 21 348 / 3974 / 850 at 2 %.  The purity and resolution of
those rows are computed over a handful of matches and mean nothing, which is why
they are bracketed or left out.

The two cluster chains do not depend on the grid checkpoint, so the five rows up
to 10 % are byte-identical to the earlier comparison run at epoch 100 of
`260831_12keV_Moench040_00`; the grid rows there moved by at most +0.7 points of
efficiency and −0.9 points of purity at 10 %.

The grid chain keeps producing an image at 20 % and 50 %, but its own numbers
have to be read carefully there:

* the model is extrapolating — it never saw anything above 10 % occupancy in
  training;
* the matcher accepts a prediction within 1 px (25 µm) of a true photon, and at
  50 % occupancy that circle contains ~1.6 photons on average, so a prediction
  that is merely *near* a photon still counts.  86.77 % is therefore an upper
  bound on "found the right photon", not a resolution claim;
* the honest signal is σ68: 0.593 µm at 0.5 %, then 0.945 / 1.979 / 6.025 µm at
  10 / 20 / 50 %.  At 50 % the matched positions are spread over a quarter of
  the 25 µm matching window, so the chain detects that photons are there without
  locating them.

The generator reports how many photons share a cell with another, which is the
ceiling a one-photon-per-cell output grid could reach: 95.20 % at 10 %, 90.62 %
at 20 %, 78.69 % at 50 %.  The measured efficiency sits above that ceiling at
every point (96.37 / 93.56 / 86.77 %) because the offset channels are unbounded
and a cell may report a position inside its neighbour — the same effect noted in
the Results section.  A finer output grid is what would turn that from an
artefact of the matching radius into real resolution at these fluxes.

Gate G2 asked for σ ≤ 0.8 µm on 12 keV MC and ≥ 96 % efficiency at 1.7 %
occupancy with purity at least as good as the cluster chain.  The scan has no
1.7 % point, so the claim is read off the harder 2 % one: 0.805 µm at 0.5 %,
99.01 % efficiency and 99.79 % purity at 2 %, against 91.79 % and 98.00 % for
the cluster chain there.

The comparison needs both torch and `aare`, so it runs in the `mlxid_demo`
environment.  On these 32×32 frames a `VarClusterFinder` is built per frame:
reusing one, as `_processFrames` does, is exact at its 101×101 ROI (verified on
real measurement frames) but overruns the internal hit buffer on frames this
small.

## Measurement pipeline

`src/measurement.py` + `scripts/05_infer_measurement.py` take a run from raw ADU
to a photon list in a single pass, with pedestal subtraction, common-mode removal
and the 3D-LUT all running on the GPU.

```bash
python scripts/05_infer_measurement.py --config Configs/infer_measurement_12keV.yaml \
    --run Edge_3filters_pos0_12keV
python scripts/05_infer_measurement.py --config ... --run Flat_3filters_pos0_12keV
python scripts/05_infer_measurement.py --config ... --run Pedestal_3filters_pos0_12keV  # fake-rate test
```

Each run writes, as `.npy` and `.png`: `binned_image` (reconstruction at
`pixel_size / binning_factor`), `native_image` (the same photons rebinned to the
25 µm pitch), `analog_image` (mean calibrated energy per frame — the classical
native readout), plus `comparison.png` with the last two side by side, the energy
spectrum, the per-frame photon count and `summary.json`.

`summary.json` reports the occupancy twice: averaged over the valid region, and
**inside the illuminated area** (pixels above half the 99th-percentile density).
For a knife-edge or slit run the second number is the one that describes the flux
the model sees.

Measured on `Edge_3filters_pos0_12keV` (MOENCH040 @150 V, 12 keV, 140 files,
14 M frames, 287 GB):

| | |
|---|---|
| GPU calibration vs `UsefulFuncs.convertAduFrameToEnergyFrame` | agrees to 2.4e-6 keV (float32 rounding) |
| pixel noise, raw | 0.1401 keV |
| pixel noise, column common mode removed | **0.1291 keV** — the 0.13 keV the model was trained with |
| frame-mean fluctuation | 28.5 eV → 1.0 eV after the correction |
| pedestal stability over the whole run | −2 ADU = 0.015 keV, a tenth of the noise |
| photon occupancy | **0.66 % inside the illuminated area** (0.16 % averaged over a ROI that is mostly knife-edge shadow).  Matches the 0.69 % of the published DanMAX working point and sits inside the 0.2–10 % training range. |
| fake rate on dark frames (cross-run pedestal, p>0.3) | **0 photons in 1000 frames** |
| throughput | 5200–5900 frame/s while sharing the GPU with a training job → ~45 min for the full 14 M-frame run |

The MOENCH040 LUT is `(1740, 400, 400)`, so `convertAduFrameToEnergyFrame` takes
its `else` branch (ADU from −1000 in steps of 10, signed natively).  The MOENCH014
LUT is the 1640-bin variant that starts at 0 ADU and handles negative values
through `|adu|` and the sign; `GpuCalibrator` implements both (agreement with the
numpy reference: 2e-6 keV).

Common mode `column_masked` takes the column median over the pixels without
charge.  The plain median is pulled up by the photons in the column, which on
the full-chip XRF runs (0.7–1.3 % occupancy) leaves every pixel 4–7 meV low —
tens of meV on a 3×3 sum, growing with the flux.  At the 101×101 knife-edge ROI
(~0.8 hit pixels per column) the effect is negligible and `column` is kept.

### Inference speed (RTX 4090)

| model | frame | precision | model only | end-to-end run |
|---|---|---|---|---|
| 12 keV `[32, 64, 64, 64]` | 101×101 | FP32 (TF32) | 11 665 frame/s (119 Mpx/s) | 7 805 frame/s (80 Mpx/s) |
| | | FP16 + channels-last | 28 991 frame/s (296 Mpx/s) | 12 265 frame/s (125 Mpx/s) |
| 5–30 keV `[32, 64, 128, 128]` | 400×400 | FP32 (TF32) | 428 frame/s (68 Mpx/s) | 360 frame/s (58 Mpx/s) |
| | | FP16 + channels-last | 973 frame/s (156 Mpx/s) | 694 frame/s (111 Mpx/s) |
| 3–30 keV `[32, 64, 96, 96]` (default) | 400×400 | FP16 + channels-last | 1 172 frame/s (188 Mpx/s) | **777 frame/s** (124 Mpx/s) |

At the same frame size the 12 keV model runs 1.7x more pixels per second than
the 5-30 keV one.  End-to-end keeps a smaller share of the model speed on
101×101 frames (42-67 %) than on 400×400 frames (71-84 %).  Per frame the loop
adds ~0.05 ms of work that does not scale with the frame size: decode splits
the batch frame by frame, and the energy histogram and image accumulation run
once per frame.  In FP16 that is more than the 0.04 ms the 12 keV model itself
needs per 101×101 frame.

`inference.precision: fp16` runs the network under FP16 autocast in
channels-last layout (`predict_fp16`).  An FP16 head alone would round every
energy to the FP16 grid: 7.3 meV steps below 15 keV, 14.6 meV above.  That is
harmless for the resolution, but a 50 eV spectrum then shows a comb.  So the
offsets and energy of the cells above threshold, about 1 % of the cells, are
recomputed in FP32 from the same features.  That costs nothing measurable;
converting the whole feature map to FP32 for the head would cost 15 %.  Both
models are trained with AMP, and the FP16 photon list matches the FP32 one:
- on MC, efficiency, purity, position RMS/σ68 and the energy core width agree
  to the last digit quoted;
- on AgMo/CuCrCo, 99.997 % of the photons are identical, with positions within
  ~20 nm.  The only differences are the 0.03 % of photons at a cell border,
  where the two cells claiming the photon swap.

The poly config uses FP16; the 12 keV config still runs FP32, as its published
numbers were made that way.  In the FP16 poly loop the model takes 67 % of the
time, decode 12 % and calibration 10 %.  Reading the raw file runs in a prefetch
thread.  TF32 and batches above 32 bring nothing, because these convolutions are
bandwidth-bound.

**Full-size MÖNCH, 800×1200** (virtual frames tiled 2×3 from real AgMo raw
frames, ~6500 photons per frame; the Moench014 LUT tiled to 1640×800×1200 =
6.3 GB on the GPU; frames served from RAM, so file reading is not included):

| model | model only, FP32 | model only, FP16 | end-to-end, FP16 |
|---|---|---|---|
| poly `[32, 64, 96, 96]` (default) | 90 frame/s | 198 frame/s | **144 frame/s** |
| poly `[32, 64, 128, 128]` | 72 frame/s | 157 frame/s | — |
| poly `[32, 64, 64, 64]` | 125 frame/s | 287 frame/s | — |
| 12 keV `[32, 64, 64, 64]` | 124 frame/s | 302 frame/s | — |

Per 800×1200 frame, end to end in FP16 with the default model:
- upload 0.25 ms;
- calibration 0.54 ms;
- model 4.95 ms;
- decode 1.0 ms;
- image + histogram 0.19 ms.

The pixel rate is the same as at 400×400 (≈ 190 Mpx/s for the model), and batch
size (4–16) makes no difference.  The FP32 path needs ~15–22 GB at batch 16
next to the LUT; FP16 needs 12 GB.  Reading the raw file needs 1.9 MB per
frame, 280 MB/s at 144 frame/s.

The last two 128-channel layers carry 92 % of the MACs of `[32, 64, 128, 128]`.
Narrower variants were trained and compared (see "Choosing the size"), and
`[32, 64, 96, 96]` replaced it as the default at no measurable cost.

## Knife-edge test — `Notebooks/KnifeEdge_*.ipynb`

Four flux points of the same DanMAX knife-edge scan, one notebook each, all with
the same ROI and the same ESF and fit as `Plotting/fig_knife_edge.py --fit odr`
(`generate_presampled_esf` + `get_odr_fitting` of
`DeepLearning/src/funcs_postprocess.py`, ESF bins 0.3 µm, symmetric ±4σ window),
so the only thing that differs between the rows is the reconstruction.  The edge
sits in the same place at every flux point, so one ROI serves all four and the
notebooks are directly comparable:

| notebook | measurement | frames | occupancy (illuminated) | classical chains available |
|---|---|---|---|---|
| `KnifeEdge_0filters` | `{Edge,Flat}_0filters_12keV` | 3.5 M | 9.42 % | none — this point was never clustered |
| `KnifeEdge_1filters` | `{Edge,Flat}_1filters_12keV` | 7 M | 3.90 % | eta only (the CNN images are older model versions) |
| `KnifeEdge_2filters` | `{Edge,Flat}_2filters_12keV` | 7 M | 1.59 % | eta + the 1/2/3-photon CNNs |
| `KnifeEdge_3filters_pos0` | `{Edge,Flat}_3filters_pos0_12keV` | 14 M | 0.66 % | eta + the 1/2/3-photon CNNs |

LSF width σ [µm], with MTF50 [lp/mm] of the fitted Gaussian in brackets:

| reconstruction | 0 filters | 1 filter | 2 filters | 3 filters |
|---|---|---|---|---|
| eta interpolation | — | 1.506 ± 0.023 (124) | 1.483 ± 0.024 (126) | 1.536 ± 0.019 (122) |
| cluster + CNN, 1 photon | — | — | 1.352 ± 0.022 (139) | 1.355 ± 0.016 (138) |
| cluster + CNN, ≤2 photons | — | — | 1.372 ± 0.022 (137) | 1.380 ± 0.016 (136) |
| cluster + CNN, ≤3 photons | — | — | 1.386 ± 0.018 (135) | 1.381 ± 0.017 (136) |
| **GridBased 12 keV** | **1.473 ± 0.017 (127)** | **1.388 ± 0.020 (135)** | **1.346 ± 0.019 (139)** | **1.360 ± 0.019 (138)** |
| GridBased poly 2–20 keV | 1.856 ± 0.037 (101) | 1.551 ± 0.028 (121) | 1.391 ± 0.018 (135) | 1.420 ± 0.021 (132) |

The ODR model is `A/2 (1 + erf((x - c)/(sqrt(2) sigma))) + B`, so σ is the width
of the Gaussian LSF and `MTF50 = 1000 sqrt(ln 2 / 2) / (pi sigma)`.  The GridBased
rows are the self-flattened reconstructions.  (Until 2026-09-28 this table took
the fitted σ for √2 σ, so its MTF50 was √2 too high, and binned the ESF with an
older copy of `generate_presampled_esf` that made σ ~3 % too small.)  Photon
counts are compared within a flux point only; every chain there sees the same
beam, so their ratio is the ratio of quantum efficiencies and the relative
DQE ∝ QE·MTF² is well defined without the absolute flux.  MTF(200 lp/mm) is
measured from the LSF (`Plotting/knife_edge_numbers.py`):

| flux point | GridBased vs ≤3-photon chain | GridBased vs eta | sub-pixel excess rms |
|---|---|---|---|
| 0 filters, 9.42 % | — | — | 18.3 % → 1.6 % |
| 1 filter, 3.90 % | — | 2.46x photons, 3.24x DQE | 19.8 % → 3.0 % |
| 2 filters, 1.59 % | 1.044x photons, 1.17x DQE | 1.47x photons, 2.82x DQE | 21.6 % → 3.8 % |
| 3 filters, 0.66 % | 1.014x photons, 1.23x DQE | 1.19x photons, 2.60x DQE | 22.6 % → 4.3 % |

At the two low-flux points, where the cluster chain is in its comfort zone, the
grid chain matches the sharpest cluster chain to within the errors and collects
as many photons as the ≤3-photon chain or slightly more, which puts its DQE at
200 lp/mm 17–23 % above that chain and 2.6–2.8x above eta.  The interesting part
is what happens above it: there is no comparable cluster + CNN image beyond
1.59 % — at 3.90 % only older model versions were ever run, at 9.42 % the
measurement was never clustered at all — while the grid chain keeps resolving
the edge: σ stays at 1.35–1.36 µm up to 1.59 % and grows to 1.39 and 1.47 µm at
3.90 and 9.42 %.

The polychromatic 2–20 keV MOENCH040 model (`KnifeEdge_poly2to20keV_Moench040.ipynb`,
same runs and processing) is 3–4 % wider at the two low-flux points, on par with
the ≤3-photon chain, and falls behind above 4 %, where its fit also degrades
(χ²/NDF 3.4 and 7.4).  Its 12 keV line moves from 12.05 to 11.64 keV with flux;
the plain column common mode is the suspect (on the Siemens star the masked one
moved the 15 keV line from 14.45 to 15.01 keV), not yet tested on the knife edge.

Sub-pixel flattening is about image uniformity rather than resolution: it takes
the sub-pixel phase from ~20 % to ~2–4 % excess rms.

The ESF width depends strongly on the ROI: a band of 500 rows across this
strongly slanted edge gives a width 1.6x that of the published 230-row ROI.
Cross-method comparisons are only meaningful at a fixed ROI.

## Results — polychromatic, default `Results/260919_poly3to30keV_Moench014_c96_00`

1.2 M training frames, flat **3–30 keV**, occupancy log-uniform 0.2–10 %.  Every
photon gets its own energy, conversion depth and charge cloud from the
α(E,t)/β(E,t) table `data/charge_lut_Moench014_150V_3to30keV.npz` (WP1.1,
validated by `01b_validate_charge_lut.py`).  GridNet channels
**[32, 64, 96, 96]** (R = 4, 157 636 parameters), 300 epochs, `best.pth` =
epoch 224.  It has been the default in `infer_measurement_poly_Moench014.yaml`
since 2026-09-19.  Fixed-occupancy scan at threshold 0.3:

| occupancy | efficiency | purity | RMS | σ68 | energy σ68 | sub-pixel non-unif. |
|---|---|---|---|---|---|---|
| 0.5 % | 99.67 % | 99.73 % | 1.70 µm | 0.99 µm | 0.28 keV | 22.2 % |
| 1 % | 99.42 % | 99.54 % | 1.78 µm | 1.04 µm | 0.29 keV | 22.2 % |
| 2 % | 98.88 % | 99.19 % | 1.89 µm | 1.08 µm | 0.31 keV | 21.8 % |
| 5 % | 97.25 % | 98.07 % | 2.29 µm | 1.28 µm | 0.42 keV | 18.3 % |
| 10 % | 94.37 % | 96.22 % | 2.96 µm | 1.70 µm | 0.89 keV | 16.3 % |

Efficiencies include the 3–5 keV photons (97–98 % found), which the 5–30 keV
model never saw.  Against energy (`scripts/07_energy_response.py`, occupancy
≤ 2.5 %, 4.3e5 matched photons, 0.25 keV bins):

| E_true | core σ | line position g(E) − E | tails, \|ΔE\| > 1 keV | efficiency |
|---|---|---|---|---|
| 3.1 keV | 0.13 keV * | +125 meV * | 2.8 % | 97.5 % |
| 3.6 keV | 0.23 keV | +16 meV | 2.3 % | 97.6 % |
| 4.1 keV | 0.26 keV | +4 meV | 2.3 % | 98.0 % |
| 5.4 keV | 0.26 keV | −12 meV | 3.0 % | 98.3 % |
| 7.9 keV | 0.26 keV | 0 meV | 4.0 % | 98.8 % |
| 12.1 keV | 0.28 keV | −7 meV | 4.3 % | 99.2 % |
| 17.4 keV | 0.27 keV | −11 meV | 4.4 % | 99.5 % |
| 22.1 keV | 0.30 keV | 0 meV | 4.7 % | 99.5 % |
| 27.6 keV | 0.31 keV | −5 meV | 4.9 % | 99.7 % |
| 29.6 keV | 0.24 keV * | −83 meV * | 7.0 % | 99.6 % |

\* within ~0.75 keV of a training edge, where the line is shifted and the
energy scale compressed.  Between 4 and 28 keV the line position stays within
25 meV.

The `energy_bias_keV` of `eval_summary.json` is the *mean* per 2.5 keV bin.  It
is set by the 3–7 % of photons in the tails, which the bounded range cuts from
one side, so it is pulled towards the centre (~+0.1 keV at 5–12 keV) even where
the line does not move.  Correcting per photon with the mean is therefore wrong;
the notebooks invert the core position g(E) instead.  With the 3–30 keV model
that correction only matters below ~4 keV.

### Choosing the size — `Results/overnight_poly3to30keV/report`

Four networks trained overnight on the same 3–30 keV frames
(`scripts/overnight_poly3to30keV.sh`, report by `scripts/08_compare_models.py`):

| channels | params | FP16 frame/s (400×400) | MC 1 %: RMS / E σ68 / purity | MC 10 %: E σ68 / purity | XRF FWHM Co / Cu / Mo / Ag [keV] | dark-run photons / 1000 frames |
|---|---|---|---|---|---|---|
| [32, 64, 128, 128] | 240 772 | 965 | 1.772 µm / 0.290 keV / 99.55 % | 0.872 keV / 96.28 % | 0.62 / 0.72 / 0.81 / 0.94 | 0 |
| **[32, 64, 96, 96]** | 157 636 | **1 185** | 1.777 µm / 0.291 keV / 99.54 % | 0.889 keV / 96.22 % | 0.62 / 0.72 / 0.82 / 0.95 | **0** |
| [32, 64, 64, 64] | 92 932 | 1 761 | 1.785 µm / 0.292 keV / 99.52 % | 0.919 keV / 96.17 % | 0.63 / 0.72 / 0.82 / 0.95 | 15 (all 2.3–3.2 keV) |
| [32, 64, 128] (R = 3) | 93 188 | 1 712 | 1.790 µm / 0.297 keV / 99.37 % | 1.040 keV / 95.64 % | 0.63 / 0.73 / 0.82 / 0.93 | 8 (2.4–3.7 keV) |

`[32, 64, 96, 96]` matches the widest network on every number and is 23 %
faster.  (The XRF runs of this comparison were redone with clean FP16 energies,
see Inference speed.)  `[32, 64, 64, 64]` loses ~5 % of energy resolution at 10 % occupancy
and reports photons on dark frames below 3.2 keV.  Three layers (R = 3) cost
the most at high occupancy.

### Previous default — 5–30 keV `[32, 64, 128, 128]`, `Results/260901_poly5to30keV_Moench014_00`

Same architecture and data volume, flat 5–30 keV, `best.pth` = epoch 290.  MC at
1 % occupancy: 99.52 % efficiency, 99.59 % purity, 1.76 µm RMS, 0.29 keV energy
σ68.  Its 5 keV edge shifted Cr Kα (5.41 keV) by +52 meV on MC and compressed
the scale there (g′ = 0.82).  On data that meant +33 meV with a 0.18 keV core,
which is why the range was extended.  Its XRF reconstructions carry the tag
`_poly5to30`.

## XRF test — `Notebooks/XRF_*.ipynb`

Two flood-field runs from the 2603 MaxIV XRF series (MOENCH014 @150 V, full
400×400 chip, slit factor 10), reconstructed with the default 3–30 keV
`[32, 64, 96, 96]` model and `Configs/infer_measurement_poly_Moench014.yaml`:

* `AgMo_1_factor_10`: 0.69 % occupancy, 1.05e8 photons;
* `CuCrCo_1_factor_10`: 1.29 % occupancy, 1.99e8 photons, 12 keV beam.

The reference is the classical 3×3 sum around a local maximum on the *same*
calibrated frames (first 10 k frames), with no model.  The yaml carries more
samples and the full flux series (slit factor 1–20).  The reconstructions of the
previous 5–30 keV model are kept as `<run>_poly5to30`, and those of all four
sizes as `<run>_c128/_c96/_c64/_r3`.

**Energy resolution.**  Core σ of the fitted line in keV, FWHM of the whole
line shape in brackets, straight from the model (no correction):

| line | E [keV] | GridBased | classical 3×3 | FWHM grid / 3×3 | MC core σ |
|---|---|---|---|---|---|
| Cr Kα | 5.41 | 0.245 (0.58) | 0.398 (0.94) | 0.62 | 0.256 |
| Co Kα | 6.93 | 0.265 (0.62) | 0.408 (0.96) | 0.65 | 0.264 |
| Cu Kα | 8.04 | 0.305 (0.72) | 0.441 (1.04) | 0.69 | 0.267 |
| elastic | 12.0 | 0.311 (0.73) ‡ | 0.470 (1.11) ‡ | 0.66 | 0.279 |
| Mo Kα | 17.44 | 0.327 (0.82) | 0.464 (1.09) | 0.75 | 0.275 |
| Mo Kβ | 19.64 | 0.353 (0.83) | 0.492 (1.16) | 0.72 | 0.290 |
| Ag Kα | 22.10 | 0.378 (0.95) | 0.489 (1.15) | 0.82 | 0.301 |

‡ FWHM = 2.355 σ.

* The grid lines are 18–38 % narrower than the 3×3 sum at every energy.  The
  core σ stays below the 0.38 keV noise floor of a fixed 3×3 sum
  (√9 × 0.126 keV) at every line, the gain that design decision D5 anticipated.
  In CuCrCo that separates Cu Kα from Co Kβ; the 3×3 sum shows Cu only as a
  shoulder.
* Cr and Co match the MC within ±5 %.  Cu, the elastic line and the lines at
  17–22 keV are 12–26 % wider, an extra term that grows with E (~1 % of E) and
  that the MC noise model (0.13 keV per pixel) does not contain.
  Pixel-to-pixel gain residuals of the LUT would do this (WP1.3).
* The previous 5–30 keV model put Cr Kα 0.4 keV above its training edge and
  compressed it (0.182 keV core, 0.48 keV FWHM, +33 meV); the MC correction
  restored the width but overshot the position.  With the 3 keV edge the line
  needs no correction.

**Line positions.**  The grid puts every line within 0.04 keV of the tables (Cr Kα
+0.01 keV).  The classical 3×3 sum reads 0.05–0.10 keV high: the MOENCH014 LUT
(2025-03-05) is 0.3–1 % high (~0.9 % at 5–8 keV, 0.3–0.5 % at 17–22 keV).  The
MC correction changes no line by more than 12 meV.

**Efficiency**, without truth, against the classical finder:

| | AgMo | CuCrCo | MC |
|---|---|---|---|
| isolated photons ≥ 5 keV found | 99.97 % | 99.99 % | 99.96 % (no neighbour within 4 px, 1 %) |
| isolated photons 3–5 keV found | 92.9 % | 100.0 % | 97–98 % |
| grid / classical photon count | 1.091 | 1.067 | — |
| classical clusters holding ≥ 2 grid photons | 8.5 % | 10.2 % | Poisson: 8.3 % / 15.0 % |
| reconstructed / deposited energy | 0.997 | 0.996 | — |

The extra grid photons are pile-up that the 3×3 finder merges; in CuCrCo the
pile-up continuum at 13–25 keV is 5–10× lower than in the classical spectrum.
On a dark run (`Pedestal_CuCrCo_1`, 1000 full frames) the model reports
**0 photons** (< 2e-8 per pixel per frame).

The sub-pixel phase of these flood fields has 48 % (AgMo) and 20 % (CuCrCo)
excess rms against 22 % on MC, so R1 is worse on data at high energies.
Self-flattening takes both to 0.5 % and 3.2 %.

## Status

- [x] WP0 data contract, self-tests
- [x] WP1.1 α(E,t) / β(E,t) charge-cloud LUT, MOENCH014 @150 V, with leave-one-out validation
- [x] WP1.2 frame generator: stamp reuse at 12 keV, per-photon LUT sampling at 3–30 keV
- [ ] WP1.3 noise / physics completion (escape peak, Compton, LUT gain residuals) and gate G1 (simulated vs measured spectra); the XRF widths at 17–22 keV ask for the gain term
- [x] WP2 halo / valid-region handling, verified exact; edge augmentation
- [ ] WP2 ablation of R and training-frame size
- [x] WP3 model, loss, decode, training loop; 12 keV, 5–30 keV and 3–30 keV models
- [x] WP3 network size scan on 3–30 keV (four widths / depths); `[32, 64, 96, 96]` is the default
- [x] WP4 metrics: efficiency/purity, RMS/σ68, energy, pair resolution, sub-pixel uniformity
- [x] WP4 baseline comparison against eta interpolation and the 1/2/3-photon models (gate G2 passed)
- [x] WP4 gate G3: efficiency, purity and energy response tabulated over 3–30 keV; self-flattening available
- [x] WP5.1/5.2 measurement pipeline: GPU calibration (both LUT variants, masked common mode), inference, images
- [x] WP5.3 XRF lines (AgMo, CuCrCo): positions, energy resolution, efficiency against the classical sum
- [ ] WP5.3 energy-resolved imaging (the XRF runs are flood fields; needs a structured sample)
- [x] WP5.4 ESF / MTF / relative DQE from the knife edge at four flux points (0.66–9.42 % occupancy), against every published chain
- [ ] WP5.4 SOLEIL 15 keV Siemens star
- [x] WP5.5 fake rate on dark runs (12 keV: 0 / 1000 frames of 101×101; poly: 0 / 1000 full frames)
- [x] gate G4: knife-edge σ = 1.360 µm (MTF50 138 lp/mm) at 0.66 %, equal within the errors to the best published chain (CNN 1 photon, 1.355 µm, 138 lp/mm); XRF lines resolved and in place
- [ ] WP6 extensions (full-size MÖNCH, ONNX/TensorRT, online), WP7 write-up