This repo contains shared MATLAB tools and example scripts to analyse XPS and NEXAFS data from PSI beamlines using a common table‑based data structure. Everyone is encouraged to use it, break it (gently), fix it, and extend it. 😄🚀

Table of contents 🗺️

What this is 🧪
Core ideas 💭
Keep an Overview — FAIR and open research 🧭
Prerequisites 🧰
Installation 💻
The shared data table 📊
MATLAB help & function call conventions 📚
- Basic in‑place call
- Getting help from within MATLAB
Core MATLAB data types 🧠
Python / pandas side quests 🐍🐼
Contributing 🤝
License / usage 📄

What this is

This toolbox provides XPS & NEXAFS functions to import, bookkeeping, spectra generation, background handling, peak fitting, plotting, and data export.

All nappMATLAB functions operate on a single MATLAB table, where each row is one data file (SES or PShell measurement) and columns store image data, spectra, meta‑data, fit results, etc.

Core ideas

One table to rule them all
Data from various beamtimes and instruments get imported into the same type of MATLAB table.
Scripts call functions, functions operate on tables
Scripts are your personal playground for "what do I want to see today?", while shared napp* functions offer the tools for your analysis. 🧰
Input → script → output
To keep the overview, it is suggested to split your analysis topics into individual script files: Each of your scripts takes an existing table as input, applies a series of operations, and saves an updated table with the same name as the script file.
Raw data stays raw
Data columns (image, spectrum_countsPerSecond, etc.) are overwritten by processing, so that there is only one recent and up‑to‑date version. The advantage is that the processing function can be called in (almost) any order or sequence. But each has a corresponding *_Roh column to go back to the original state.
Collaborative by design
Everyone in the group is more than welcome to contribute new functions and examples so that the next beamtime is easier for everyone. 🤝

🧭 Keep an Overview — FAIR and open research

💡 The Big Idea

Use scripts (or live scripts) that call nappMATLAB functions to process data step by step. Each script documents what was done, why it was done, and what the outcome looks like (including plots). This turns everyday analysis into a lightweight data‑management practice that supports open and FAIR research from day one. 🙂 It helps you quickly find and share data in typical situations:

A team member might ask: Can you send me the figure from your last presentation so that I can edit it?
You might ask: Can I compare this NEXAFS spectrum to one from my first beamtime 2 years ago?
A referee for your publication might ask: Have you used all data, or were some discarded?

Concretely, concepts like the ACTRIS data levels (Level 0 raw, Level 1 calibrated, Level 2 processed, Level 3 publication‑ready) can be mirrored in how scripts and data tables are named and stored, and aligned with the SNSF Open Research Data policy and the ACTRIS Data Policy.

📊 Data Levels

Level	Description
0 – Raw data	Native resolution with acquisition metadata (instrumental, beamline, sample info).
1 – Calibrated	Basic QC — excludes faulty or aborted spectra.
2 – Processed	Fully QC-checked, calibrated, normalized, and outlier-filtered (for scientific, not technical, reasons).
3 – Final	Publication-ready results for figures, tables, and discussions. 🎯

🧪 Processing Workflow

Beamtime data are processed in two main phases:

Levels 0–1: during the beamtime 🕰️ — ensures data quality and guides ongoing experiments. Scripts live in the beamtime folder and are usually shared.
Levels 2–3: after the beamtime 🧠 — selects and combines results from multiple sources (e.g. lab experiments, simulations) and beamtimes. Scripts live in your personal analysis folder on the shared drive.

⚙️ Script Overview

`importData.m`

Level 1 data
Shared script in beamtime folder
Imports all data from one beamtime
- Load data into MATLAB tables
- Add X-ray energy and sample descriptions
- Identify and document faulty data
Saves output as FileList.mat (name has legacy reasons) 🧓

`generateSpectra.m`

Level 1 data
Shared script in beamtime folder
Processes all spectra
- Generate spectra (dimensional reduction)
- Plot & QC XPS/NEXAFS spectra
- Compare key edges/regions
- Identify outliers
- Calculate binding energies & peak widths
Saves output as SpectrumList.mat (tradition beats convention, again!) 😄

`specificScienceQuestion.m`

Level 2 or level 3 data
Your personal data-analysis script 👩‍🔬
Combines selected datasets to answer one question
Examples:
- AnalysisStep3DepthProfiles.m
- AnalysisStep4SurfaceConcentration.m
- BeamTimeNov2025isItChlorineFits.m
- GaussFitHClOnIce.m
- FigureHexylamineN1sLjIce.m
Saves output as a .mat file with the same name 🧾

🔁 Typical Processing Chain

beamtime raw files
↓importData.m → FileList.mat
↓generateSpectra.m → SpectrumList.mat
↓AnalysisStep4SurfaceConcentration.m → AnalysisStep4SurfaceConcentration.mat
↓GaussFitHClOnIce.m → GaussFitHClOnIce.mat

SpectrumList.mat
↓FigureHexylamineN1sLjIce.m → FigureHexylamineN1sLjIce.mat

Reuse .mat outputs to skip re-running previous steps 🧠💤

Prerequisites

MATLAB (reasonably recent; minimum R2018; R2025b+ recommended).

To get it, create a MathWorks account using your PSI email. Your PSI license will then automatically be assigned to your account (assuming you have an older version installed via the software kiosk), and you can go ahead and download the latest version from the MathWorks homepage, even if it is not yet available through the software kiosk.
Access to the Napp data folders (Windows network share).
Git (if you want to clone and contribute instead of downloading zips).

Optional but handy:

A Python environment if you want to poke at the tables with pandas later. 🐼

1. Get the code

Option A – clone via git (recommended)

WIP: No idea yet what it takes to set up gitea git on a PSI computer. Can one of the Windows users update this section here, so that the others know? 🙂

Open a Terminal (Windows Power Shell)
Go to the folder where you want to store the git repository folder using the cd command, e.g. cd .\Documents\Gitea\
Open the repository in your browser. You need the whole functionsMatlab repo: https://gitea.psi.ch/5505/functionsMatlab.git not only the beamtime subfolder.
Click on the blue <> Code button & copy the HTTPS link
Type git clone into the terminal and copy the HTTPS link after it (git clone https://gitea.psi.ch/5505/functionsMatlab.git). Press enter to run the command.
If you get an authentication error (happens quite often with PSI gitea): Run the command again (press the up arrow on your keyboard and press enter again).
It should now start copying the contents of the repository to the folder.
Once it is done, you can go into the folder with cd .\functionsMatlab\ and do all the git things.

Option B – download zip (for non‑git days)

Open the repository in your browser. You need the whole functionsMatlab repo: https://gitea.psi.ch/5505/functionsMatlab.git not only the beamtime subfolder.
Click <> Code and then Download ZIP.
Unzip to your favorite folder to store shared functions such as D:\MATLAB\napp-matlab or /Users/you/Code/napp-matlab.

Add to the MATLAB path

You need the repository (and its subfolders) on the MATLAB path.

In MATLAB:

% Adapt this to where you cloned/unzipped the repo
toolboxPath = 'C:\Users\you\Code\napp-matlab';
addpath(genpath(toolboxPath));
% Optionally make it persistent
savepath;

Data folders and naming

The functions require a specific file organisation and folder naming for the beamtime data to locate and importe SES, NEXAFS, photo, and pressure data consistently across beamtimes. A single beamtime is organised in a top‑level "beamtime folder" that contains four data subfolders and no further nesting inside them:

202202_Feb_IssIceNapp/
├─ SES/       # SES XPS/PE images in IgorPro .ibw or .xy files
├─ NEXAFS/    # NEXAFS (XAS) data in .h5 files
├─ Photos/    # Experimental photographs in .bmp format
└─ Pressure/  # LabView pressure logs in ASCII .dat or .txt files

Additional folders might be added to the beamtime folder to hold the scripts, reports, lab notes. You usually start by scanning such a folder into a table (see MATLAB example files):

FileList = nappFileList('2025/20251119_NappIss', ...
    'mode','local', ...
    'importLabView','off');
save('FileList.mat',"FileList");

The first argument is a relative path under your data root.
mode selects where to look for data (local / network), i.e. it defines the data root.
The result is a MATLAB table (by convention often called FileList because it lists all the files acquired during a beamtime).

Naming conventions

Folders: YYYYMMDD_BeamlineName (e.g. 20251119_NappIss).
Scripts: descriptive names like generateSpectra.m, PlottingForAnand.m.
Output: typically script‑matching names, saved into .mat files.

Data files date and time

Copying files while preserving their original timestamps (Creation Date and Last Modification Date) is a common challenge because standard "drag-and-drop" operations often treat the copy as a "new" file created at the moment of the transfer.

To do this safely and reliably, you need to use specific commands or utilities designed for archival integrity.

🪟 Windows 11: Using Robocopy

The most robust tool built into Windows is Robocopy (Robust File Copy). Unlike the standard copy command, it has specific flags for metadata.

Open Terminal or Command Prompt.
Use the following syntax:

robocopy "C:\Source\Folder" "D:\Destination\Folder" /E /DCOPY:DAT /COPY:DAT

Breakdown of Flags

/E: Copies all subdirectories, including empty ones.
/DCOPY:DAT: This is the crucial part. It tells Windows to copy the Data, Attributes, and Timestamps for directories.
/COPY:DAT: Tells Windows to copy the Data, Attributes, and Timestamps for files.

🍎 macOS: Using `cp` or `rsync`

On macOS, the standard "Finder" copy usually preserves the Modification Date but often resets the Creation Date to the time of the copy. To preserve everything, use the Terminal.

Option A: The `cp` command (Simplest)

Open Terminal.
Run:

cp -pPR /Source/Path /Destination/Path

-p: Preserves modification time, access time, file flags, and permissions.
-R: Recursive (copies folders and their contents).

Option B: `rsync` (Best for large transfers)

Rsync is the professional standard for syncing data while maintaining metadata.

rsync -avz --times /Source/Path/ /Destination/Path/

-a (Archive mode): This is a "bundle" flag that preserves permissions, symlinks, and—crucially—timestamps.
--times: Explicitly ensures modification times are transferred.

The shared data table

All nappMATLAB functions operate on a MATLAB table with one row per file and a set of standard variables (columns). Using the same variable (column) names across users and software helps with data sharing and is a prerequisite for robust scripting. The key point of this whole table exercise is that data can be filtered (and found) across several beamtimes or experimental campaigns. Filtering makes most sense using metadata such as temperature and pressure, but the month of the beamtime or, importantly, the sample description added during the beamtime can also be very useful. Filtering uses standard MATLAB code. The example below filters for all XPS spectra from 3rd of September that are not labeled as N2O5, bad data, excluded data, gas-phase, Survey, or Auger, and that were recorded at 720 eV X-ray energy on the N1s region.

tmpDataSelection =  contains(FileList.name,'.ibw') ...
    & day(FileList.lastModifiedDatenum) == 3 ...
    & month(FileList.lastModifiedDatenum) == 9 ...
    & ~contains(FileList.sample,'N2O5') ...
    & ~contains(FileList.sample,'(bad data)') & ~contains(FileList.sample,'(excluded data)')...
    & ~contains(FileList.sample,'gas-phase')...
    & ~contains(FileList.name,'Survey')...
    & ~contains(FileList.name,'Auger')...
    & FileList.xRayEkinRange_eV(:,1) == 720 ...
    &  contains(FileList.name,'N1s');

Table Variables: Identification & meta

name
Filename and ID as given by the instrument software during acquisition. Typically includes run number, edge, and photon energy.
sample
Free‑text description of the sample or experiment. Often used to mark data quality, e.g. including (bad data) for files that should be excluded from analysis.
lastModifiedDatenum
Timestamp (numeric datenum) from the data file metadata indicating when the acquisition was finished.
lastModifiedDatestr
Human‑readable string version of the lastModifiedDatenum.
creationDate
File creation date as reported by the operating system (this logs when the measurement was started).
importDate
Date/time when the file was imported into the MATLAB table (can differ from acquisition/creation time, useful for tracking when data entered the workflow).
analysisDir
Path to the analysis directory where the current table and related scripts live.

Table Variables: Raw and processed data

image
Primary result from SES software; multidimensional data (typically counts per second vs kinetic energy vs sequence iteration). This is the original measurement data.
imageRoh
Snapshot of image at import time. Remains unchanged so you can always go back to the original image regardless of later processing steps.
spectrum_countsPerSecond
Spectrum derived from image (counts per second vs energy), usually by averaging or summing over sequence iterations and normalising to 1 s.
spectrum_counts
Same as above but in raw counts (not normalised per second), depending on context and instrument output.
spectrumRoh_countsPerSecond
Snapshot of spectrum_countsPerSecond at generation time. Used to restore or compare against the original spectrum.
spectrumRoh_counts
Snapshot of spectrum_counts at generation time. Preserves the initial integrated spectrum even if later processing modifies the live columns.

Table Variables: Analyzer and X‑ray settings

scientaEkin_eV
Array of kinetic energies (in eV) corresponding to the analyser energy axis for the spectrum or image.
scientaEkinRange_eV
Two‑element range of the kinetic energy window in a human‑readable form.
scientaPassEnergy_eV
Pass energy (in eV) used by the Scienta analyser; controls resolution vs line shapes.
scientaDwellTime_ms
Dwell time per channel/step (in ms); influences counting statistics and total acquisition time.
regionName
Name of the acquisition region as defined in the SES software (e.g. "C1s", "Survey", "O_Kedge_Resonant").
scientaAcquisitionMode
Acquisition mode selected in the analyser, sweep or fixed (snapshot).
beamlineInt
X‑ray flux, typically the output from a diode.
scientaLensMode
Lens mode of the analyser such as HiPPHAXPES.
scientaRegionIterations
Number of region iterations.
scientaSequenceIterations
Number of sequence iterations.
xRayEkin_eV
X‑ray photon energy (in eV); essential for converting to binding energy in XPS and for obvious reasons for NEXAFS. It is automatically recorded for NEXAFS and needs to be added manually for XPS.
xRayEkinRange_eV
Human‑readable X‑ray energy range in eV.
(SLITS – WIP)
Slit settings (horizontal/vertical or monochromator slits). Once implemented, these should store the X‑ray beamline slit openings used during acquisition.

Table Variables: Sample settings

temperature_dC
Sample temperature during the measurement (units °C).
pressure_mbar
Pressure of the sample environment.
x-y-z position_mm
Sample position coordinates (stage x/y/z).
gas flows
Gas flow rates (e.g. in sccm) for reactants or carrier gases in the experimental cell.

Table Variables: Processing outcome

These variables are typically created or updated by fitting/analysis functions.

peakPosition
Fitted peak position(s).

Live vs Roh columns

Columns without Roh (image, spectrum_countsPerSecond, …) are live and will be overwritten by processing functions.
Columns ending in Roh store the state at import / generation time and are never touched again by the common functions.

This allows:

iterative processing during beamtime, and
going back to clean starting points without reimporting everything. 🔁

MATLAB help & function call conventions

Most functions in this toolbox operate on the shared MATLAB data table and follow a few simple calling patterns.

Basic in‑place call

Many functions take a table as input and return the same table with additional variables or modified content:

InputTable = nappFunctionAbc(InputTable);

InputTable is a MATLAB table containing your data.
The function reads from InputTable, writes its results into new or existing columns, and returns the updated table.
Plotting functions are an exception: they often do not modify the table and only create figures.

Copy‑on‑write call (not recommended)

If you want to keep the input table unchanged and work on a copy, use different variable names:

CopyOfTable = nappFunctionAbc(InputTable);

InputTable remains untouched.
CopyOfTable is a full copy of InputTable plus the additional results created by nappFunctionAbc.

Mandatory vs optional inputs

Function calls typically look like this:

OutputTable = nappFunctionAbc(InputTable, x, y, ...
    'tableSelection', tmpDataSelection, ...
    'tableVariable', 'TableVariableName', ...
    'energyScale',  'kinetic', ...
    'mode',         'mean', ...
    'showFigures',  true);

Mandatory inputs (e.g. InputTable, x, y) come first and are positional.
Optional inputs are given as name–value pairs, just like in standard MATLAB functions.

Common options:

'tableSelection': tmpDataSelection
Logical array that selects the rows to operate on. Only rows where tmpDataSelection is true are processed.
'tableVariable': 'TableVariableName'
Name of the column in the table that the function should use as its primary data (e.g. 'image', 'spectrum_countsPerSecond').
'energyScale': 'kinetic' (or 'binding', 'xRay', depending on the function)
Specifies which energy scale to use for plotting or calculations.
'mode': 'mean'
Selects a specific processing mode, for example whether to compute a mean or sum over iterations, or which plotting style to use. The available modes depend on the function.
'showFigures': true (or false)
Controls whether figures and other graphical output are displayed.

Each function may define additional options; always check the help text for the full list.

Getting help from within MATLAB

Each function should ship with a short header and usage examples. 📝✨
You can get a list of all functions and a short description by navigating to the beamtime function folder 🧪 in MATLAB and running:

help beamtime 💡

Contents of beamtime:

                 Napp_RH_p - helper function that calculates the relative humidity for a given temperature and water pressure.
                 Napp_p_RH - helper function that calculates the partial pressure of water for a given temperature and relative humidity.
                Napp_p_ice - marks the vapor pressure at a given temperature in a plot of the vapor pressure of ice between 200-270 K
           addPhotonEnergy - adds the Xray photon energy for selected measurements to the data table
            addSampleNotes - adds a sample description to the data
           nappAlignImages - shifts each sequence iteration to align images with a drifting kinetic energy of the PE features.
        nappAnnotateSurvey - is a small helper function that adds region labels to a survey in the topmost figure.
            nappBackground - Removes background from photoelectron or XAS spectra.
      nappBindingEnergyXps - calculates the binding energy scale for spectra based on the kinetic energy shift.
        nappCombineSpectra - Combine multiple PE or NEXAFS spectra through averaging or summation.
             nappCopyShift - Copy energy shift from one XPS region to spectra of the same sequence but other regions.
     nappDetermineShiftXps - Determine energy shift calibration from XPS images or spectra.
              nappFileList - Import spectroscopy data (XPS, NEXAFS) from a beam time folder into data table.
           nappFitGaussXps - fits photoelectron (PE) spectra with up to 4 or 5 Gaussian peaks.
nappFitGaussXpsDevelopment - Playground to add Shirley background :-).
       nappGenerateSpectra - Generate 1D spectra from image spectra.
                  nappPlot - NAPPLOT Plot photoelectron spectra and NEXAFS images from a data table.
                 nappScale - Normalize XPS and NEXAFS spectra.

Additionally, little helper functions can be found in the utils folder:

functionsMatlab/utils:

          brewermap - The complete selection of ColorBrewer colorschemes/palettes (RGB colormaps).
    brewermap_nodes - Create a figure showing all ColorBrewer 2.0 colorschemes' defining nodes.
     brewermap_view - Create an interactive figure for viewing ColorBrewer colormaps. With demo!
            distFig - Arrange all open figures on the screen.
exportFigureData - Export line and scatter plot data to ASCII files
       getPsiColors - Returns a struct with PSI corporate identity colors.
 plotExportedData - Plot all exported ASCII data files in one figure
outerjoinRowToTable - Adds one row to a table. Variables that are missing in
    preset_colormap - Wrapper for any COLORMAP function, to provide preset parameter values.

You can access a more detailed description of each function via standard MATLAB commands:

help nappFunctionAbc 🔍
doc nappFunctionAbc 📖

The nappMATLAB example scripts in the documentedExampleScripts folder of this repo
are a good starting point for your work with nappMATLAB 🚀:

importData.m shows how to import and annotate SLS data.📈🧪
generateSpectra.m gives examples how to plot data allready during beamtimes 📊
HowDoI.m lists solutions for most common beamtime wishes on data processing 🧠. It includes a description of how to merge data tables with different number and type of colums.
gaussFit.m fitting during beamtimes helps, here is how 🔬

This assumes you are already familiar with the basic MATLAB tools and usage 🧰:

Getting started with MATLAB
https://www.mathworks.com/help/matlab/index.html
Working with data tables
Tables overview (creation, indexing, adding variables, etc.):
https://www.mathworks.com/help/matlab/tables.html

Date handling with datetime functions https://www.mathworks.com/help/matlab/ref/month.html https://www.mathworks.com/help/matlab/ref/year.html

Text matching in table variables with contains:
https://www.mathworks.com/help/matlab/ref/contains.html
Working with functions
Calling functions:
https://www.mathworks.com/help/matlab/learn_matlab/calling-functions.html

Name–value pair arguments:
https://www.mathworks.com/help/matlab/matlab_prog/namevalue-in-function-calls.html

Defining your own functions:
https://www.mathworks.com/help/matlab/ref/function.html

Core MATLAB data types.

MATLAB has a few central container types: numeric arrays (including matrices), cell arrays, character arrays, and string arrays. Arrays/matrices are homogeneous numeric containers, while cell arrays can hold mixed types; text can be stored either as character arrays (' ' syntax) or as string arrays (" " syntax).

Arrays and cell arrays

Numeric array / matrix: Homogeneous numeric data, typically created as A = [1 2; 3 4].
- Index with A(i,j) using (); result is numeric and the same type as A.
Cell array: Container where each element ("cell") can hold anything (numbers, char vectors, strings, structs, other cells, …), e.g. C = {1, 'abc'; [1 2 3], "foo"}.
- C(i,j) with () returns a cell array (wrapper, not the content).
- C{i,j} with {} returns the content stored in that cell.

Characters vs strings

Character array: A vector or matrix of char, created with single quotes, e.g. s = 'abc'.
- Index with s(i) using () to get one or more characters as char.
String array: Newer text type, created with double quotes, e.g. t = "abc", S = ["a" "b" "c"].
- Index with S(i) using () to get a string scalar (still type string).
- Recommended for new code; cell arrays of character vectors ({'a','b'}) are legacy for text storage.

Data table

Tables and timetables are both tabular containers for column-oriented variables that can have different data types, as long as all columns share the same number of rows.

Table indexing (same ideas for timetables, but first dimension is time):
- T(rows, vars) with () returns a subtable.
- T{rows, vars} with {} returns the contents as an array (numeric, cell, etc.), not a table.
- T.VarName uses dot notation to get a single variable as a column vector/array.

Quick syntax summary

()
- Arrays: content, same type as the array (numeric, string, char, logical, …).
- Cell arrays: returns a cell array (one or more cells, still wrapped).
{}
- Only for cell arrays: unwrap and return the stored value (double, char, string, struct, etc.).
' '
- Create a character array (legacy text representation).
" "
- Create a string scalar; multiple elements give a string array (preferred for text).

Python / pandas side quests 🐍

If you or a future group member want to explore data in Python:

The MATLAB table maps naturally to a pandas.DataFrame: one row per experiment, columns identical to MATLAB variable names.
Image and spectrum columns can be stored as arrays (NumPy) in "object" columns.
Keep the same column naming scheme (image, imageRoh, spectrum_countsPerSecond, …) and document any mapping if you deviate.

A simple workflow is:

Save the MATLAB table (FileList) as .mat or HDF5.
Load it in Python and convert to a DataFrame, preserving column names.
Use pandas for tabular manipulations and matplotlib for plotting and processing.

Contributing

This is a group toolbox. Contributions are welcome. 🎯

Guidelines:

Reuse the shared table structure and column naming (especially data vs Roh columns).
Keep functions focused and composable; put experiment‑specific logic in scripts.
Prefer updating existing functions over creating new (personal) versions.
Document new functions with:
- a clear help header,
- input / output description,
- at least one minimal example.
Add or update example scripts (examples/ or scripts/) showing how to use your new features.
If you introduce new table variables, add a short description in the data‑structure docs.

Git hygiene (short version):

Use feature branches for non‑trivial changes.
Write clear commit messages and push changes for every major update or change.
Do not commit raw beamline data; keep those on shared drives and refer to them from scripts.

License / usage

This code is intended for internal use within the research group (and close collaborators). If you plan to use or release it outside the group, please check first.

Happy NAPPing! 🎉🧊
If you are lost, confused, or plotting the wrong edge – reach out.

readme.md Unescape Escape

nappMATLAB toolbox 🧪📈