A walkthrough of the MLXID pipeline: Monte Carlo generation of single-photon detector clusters -> assembling multi-photon pile-up samples -> trained deep learning models that reconstructs sub-pixel photon position -> for single photon, comparison against the classical eta-interpolation baseline, scored directly against known MC ground truth -> optional training and inference on real measurement data.

See the sibling repos for the full picture this is distilled from: DataProcess, DeepLearning, EtaInterpolation, McGeneration. The charge-transport simulation that produced data/mc_raw_simulation/ (the basis for data/mc_params/) is open-sourced at https://github.com/slsdetectorgroup/ChargeTransportSimulation.

12 keV working point

12 keV is the MaxIV 2603 beamtime working point: it has knife-edge measurements (for a quantitative MTF benchmark) with flat-field measurements, and already has trained 1/2/3-photon models -- which is what this demo's pretrained checkpoints are.

What's in here vs. what's left out

Everything in data/ and models/ is chosen or generated specifically to be shippable:

Folder	Contents	Size	Provenance
`data/mc_raw_simulation/`	raw per-depth-bin charge-transport simulation output for this one sensor/voltage/energy point (256 files)	~34 MB	pre-generated
`data/mc_params/`	charge-cloud shape parameters alpha(t)/beta(t), fit fresh from the above each time `01` runs (+ a plot of the fit)	~200 B	generated by `scripts/01_...`
`data/mc_samples/`	~40k single-photon MC clusters + exact ground-truth (x,y,z,E)	~18 MB	generated by `scripts/01_...`
`data/pileup_samples/`	~40k 2-photon + ~40k 3-photon MC pile-up clusters, ground truth	~40 MB	generated by `scripts/02_...`
`data/measurement_samples/`	small fallback slice of real measurement clusters (both targets, 1/2/3-photon), for `05` when group storage isn't reachable	~27 MB	generated by `scripts/prepare_measurement_fallback.py`
`models/`	one pretrained checkpoint per photon count (1/2/3), the same ones `DeepLearning/Configs/infer_*photon.yaml` point at for 12 keV	~13 MB	copied from `DeepLearning/Results/**/Models/`

Left out on purpose: the full real measurement data isn't shipped -- it's tens of GB per target (see DataProcess/Samples/), so 05 reads a small slice of it directly from group storage at run time by default. Only a much smaller (~30k-65k event) fallback slice lives in data/measurement_samples/, for when that storage isn't reachable (see "What 05 does"). The full production MC/pile-up training sets (tens of GB) are also not shipped. The charge-transport simulation code that produced data/mc_raw_simulation/ is a separate, open-sourced repo: ChargeTransportSimulation.

Layout

data/            small datasets (see table above)
models/          three pretrained 12 keV checkpoints (1/2/3-photon)
src/             shared code: CNN architectures, datasets, eta-interpolation
                 functions, and a trimmed MC generator (copied/adapted from
                 the four source repos, kept here so this folder is self-
                 contained)
scripts/         01-05, run in order (see below); plus
                 prepare_measurement_fallback.py, a separate one-time
                 utility (needs group storage access) that (re)builds
                 data/measurement_samples/ -- not part of the numbered
                 sequence

Environment

All five scripts run in one conda env, mlxid_demo, which has everything (numpy, scipy, ROOT, h5py, omegaconf, torch+CUDA, and aare) together -- unlike the four source repos, which switch between several partial envs per pipeline stage. It doesn't depend on any machine-specific build: aare comes straight from PyPI, so a colleague can build the same env on their own machine (with a CUDA GPU) with:

conda create -n mlxid_demo -c conda-forge python=3.12 numpy scipy h5py omegaconf root -y
conda activate mlxid_demo
pip install aare
pip install torch --index-url https://download.pytorch.org/whl/cu124   # match your CUDA version

ROOT has to come from conda (conda-forge; no usable pip wheel); everything else here is pip-installable, matching how the scripts actually import it. environment.yml in this folder is the exact conda env export of the env these scripts were tested against, for when you want a precise reproduction rather than the recipe above (conda env create -f environment.yml). If disk space is tight, conda clean --all is safe to run first -- it only clears the package cache, not any env.

Running it

Scripts 01-02 regenerate data/; skip them and just use the shipped files unless you want to change the sample sizes.

conda activate mlxid_demo

# regenerate demo data (optional -- already shipped in data/)
python scripts/01_generate_mc_singlephoton.py
python scripts/02_assemble_pileup_samples.py

# score the shipped pretrained models (and, for 1-photon, classical
# eta-interpolation) against MC ground truth
python scripts/03_eval_mc_truth.py

# train a model from scratch on the small demo dataset (sanity check, not
# a reproduction of the shipped checkpoint's accuracy)
python scripts/04_train_demo.py --photons 1
python scripts/04_train_demo.py --photons 2
python scripts/04_train_demo.py --photons 3

# qualitative inference on real measurement data (needs group storage access,
# see "What 05 does" below) -- defaults to a small, quick slice
python scripts/05_infer_measurement.py --photons 1 --target Edge2Filters_12keV
python scripts/05_infer_measurement.py --photons 2 --target Flat2Filters_12keV

The double/triple-photon model classes (src/models.py) hard-code .to('cuda') for their coordinate grids, inherited from the source repo -- a CUDA device is required for --photons 2 and --photons 3 (and for 03 and 05 where they load those checkpoints).

What `03`'s eta-vs-CNN comparison shows

Classical eta interpolation (the Rosenblatt-LUT method in EtaInterpolation/EtaInterpolationFunctions.py) only reconstructs one photon per cluster by construction, so this comparison is single-photon only (2/3-photon pile-up has no classical baseline to compare against -- that's the reason a CNN is used there at all). It's deliberately not the same computation as DeepLearning/Infer_1Photon.py's interpolate_eta_from_points(predictions): that applies eta interpolation as a post-hoc smoothing step on top of the CNN's own output. Here, the two methods run independently on the same raw clusters, both scored against the same MC truth.

The LUT is fit on the same full dataset it then interpolates (no train/test split) -- that matches real usage: there's no separate "training set" for eta interpolation, a measurement interpolates itself from its own charge-sharing statistics.

eval_single_photon() reports RMS vs. truth three ways: the raw eta value used directly as a position (before interpolation), the LUT-interpolated position (after), and the CNN. The before/after split is the point of including it at all -- it shows why eta interpolation exists, not just how it compares to the CNN. Raw eta values are pulled toward pixel centers by the charge-sharing response (the classic detector "S-curve" bias); the LUT interpolation removes most of that.

Reference numbers from a normal run (regenerating data/ via 01/02 first): eta interpolation ~0.048 px RMS, CNN ~0.033 px; pile-up per-photon RMS ~0.051 px (2ph) / ~0.067 px (3ph).

What `05` does

Runs the pretrained models on real measurement data that normally lives on sls group storage (/mnt/sls_det_storage/moench_data/MLXID/Samples/Measurement, not part of this repo -- each target is tens of GB across 16 chunk files per photon count). By default it reads just one chunk, capped at 50,000 events, so it finishes in a few seconds; widen with --n-chunks / --max-events-per-chunk for a fuller (slower) reconstruction.

If that group-storage folder isn't reachable, 05 automatically falls back to the small sample shipped in data/measurement_samples/ instead -- same code path, just a much smaller input (one pseudo-chunk of 30k-65k events instead of up to 16 chunks of millions). --n-chunks is capped to 1 in that case. To (re)build that fallback sample, run scripts/prepare_measurement_fallback.py (needs group-storage access itself).

There's no ground truth for real data, so this is qualitative, and the output format follows DeepLearning/Infer_{1,2,3}Photon.py's accumulate_hits() / save_results(): a super-resolution reconstructed frame, a per-pixel hit-count frame, and a sub-pixel position histogram, each saved as .npy and .png under InferenceResults/<target>/<n>ph/. The knife-edge target's super-resolution frame should show a sharp edge; the flat-field target's sub-pixel histogram should look uniform (though with the fallback sample's much smaller statistic). Unlike the Infer scripts, 05 does not also apply the post-hoc eta-interpolation smoothing step to the predictions -- just the CNN's own output.

README.md

MLXIDDemo