5505-public/nappMATLAB-toolbox
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bf1eb85f88877faec8fdb63f8cdf6754d0fbf346
nappMATLAB-toolbox/beamtime/nappFileList.m
T

1833 lines
78 KiB
Matlab
Executable File
Raw Blame History

function ListFiles = nappFileList(dataPath,varargin)
%NAPPFILELIST Import spectroscopy data (XPS, NEXAFS) from a beam time folder into data table.
% Photos, analyser and sample settings such as pass energy, lens mode, temperature,
% position, and pressure are automatically extracted and added to the table.
%
% SYNTAX
% ------
% ListFiles = NAPPFILELIST(dataPath)
% ListFiles = NAPPFILELIST(dataPath, DataTable)
% ListFiles = NAPPFILELIST(dataPath, DataTable, 'Name', Value)
%
% DESCRIPTION
% -----------
% NAPPFILELIST scans a beam time folder and imports all experimental data
% files (XPS, NEXAFS, and photos) with their associated
% metadata into a MATLAB table. The function automatically:
% - Reads XPS data and metadata from SES (.ibw) or Specs (.xy) files
% - Reads NEXAFS data and metadata from HDF5 (.h5) files
% - Splits multi-scan HDF5 files into individual single-scan files
% - Extracts instrument settings (pass energy, dwell time, lens mode, etc.)
% - Normalizes count rates (per dwell time) to counts per second (cps)
% - Averages region iterations during import
% - Creates a semicolon-separated CSV file for Excel import
%
% REQUIRED INPUT ARGUMENTS
% ------------------------
% dataPath
% Beam time folder path (string or char vector).
% Format: 'YYYY/BeamtimeName'(Mac,Linux,Unix) or 'YYYY\BeamtimeName' (Windows)
% Examples: '2025/November_NappIss', '2024/March_PolLux'
%
% Note: The folder structure must follow the strict naming convention
% described in the "Folder Structure" section below.
%
% OPTIONAL POSITIONAL INPUT
% -------------------------
% DataTable
% Existing data table to append imported data to (table object).
% If provided, new files are merged with existing table.
% If not provided, a new table is created from scratch.
%
% NAME-VALUE PAIR ARGUMENTS
% -------------------------
% 'mode'
% Import mode (char or string, default: 'local').
% Supported values:
% 'local' - Import from local computer (default)
% 'nappDrive' - Import from mounted NAPP network drive
% 'dataDrive' - Import from mounted group drive (5505)
% 'localNew' - Import only new files not in existing DataTable
% 'nappDrivePick' - Browse and select files from NAPP drive
% 'localPick' - Browse and select files from local folder
% 'dataPick' - Browse and select files from mounted group drive
%
% 'importLabView'
% Import LabView pressure/temperature logs (char or string).
% Default: 'on'
% Supported values:
% 'on' - Import LabView logs from Pressure subfolder
% 'off' - Skip LabView log import (this is much,much faster)
%
% OUTPUT ARGUMENTS
% ----------------
% ListFiles
% MATLAB table with the following columns (variables):
% - name: file name (string)
% - folder: full path to file (string)
% - date: file modification date (datetime)
% - bytes: file size in bytes (numeric)
% - isdir: directory flag (logical)
% - datenum: date as serial number (numeric)
% - creationDate: acquisition timestamp (datevec format)
% - image: 2D spectral map or image (cell array)
% - imageRoh: raw 2D spectral map (cell array)
% - spectrum_counts: integrated spectrum (numeric or cell array)
% - spectrum_countsPerSecond: mean spectrum normalized to cps
% - scientaEkin_eV: kinetic energy axis (cell array, eV)
% - scientaEkinRange_eV: [min max] kinetic energy range
% - scientaPassEnergy_eV: analyzer pass energy (numeric, eV)
% - scientaDwellTime_ms: dwell time per channel (numeric, ms)
% - regionName: region label (e.g., 'C1s', 'O K-edge NEXAFS')
% - scientaAcquisitionMode: 'Swept' or 'Fixed' mode
% - scientaLensMode: lens transmission mode (string)
% - scientaRegionIterations: number of region scans (numeric)
% - scientaSequenceIterations: number of sequence repeats (numeric)
% - xRayEkin_eV: X-ray photon energy for XPS (numeric or cell array, eV)
% - xRayEkinRange_eV: [min max] photon energy range
% - beamlineInt: beamline intensity (I_0 for NEXAFS)
% - importDate: timestamp of table generation (char)
% - analysisDir: working directory at import time (string)
%
% Also creates: YYYYMmm_DataList.csv
% Semicolon-separated ASCII file in beam time folder for Excel import.
%
% FOLDER STRUCTURE & NAMING CONVENTIONS
% =====================================
%
% REQUIRED DIRECTORY STRUCTURE
% ----------------------------
% The beam time folder must contain the following sub-directories:
%
% SES/ - XPS data in Igor (.ibw) or Specs (.xy) format
% NEXAFS/ - NEXAFS maps in HDF5 (.h5) format
% Photos/ - Sample photographs in BMP (.bmp) format
% Pressure/ - LabView pressure and temperature logs
%
% Full path convention:
% LOCAL: ~/Forschung_Einsam/PSI_SLS/NAPP/SC Data/YYYY/BeamtimeName
% NAPP DRIVE: /Volumes/NAPP/SC Data/YYYY/BeamtimeName (Mac)
% \\fs03\NAPP\SC Data\YYYY\BeamtimeName (Windows)
% DATA DRIVE: /Volumes/5505/Data/YYYY/BeamtimeName (Mac)
% \\fs101\5505\Data\YYYY\BeamtimeName (Windows)
%
% EXAMPLE FOLDER LAYOUT
% ---------------------
% 2025/November_NappIss/
%  SES/ [XPS data]
%   0001_C1s_525eV.ibw
%   0002_O1s_850eV.ibw
%   0003_Survey_850eV.ibw
%   ...
%  NEXAFS/ [NEXAFS maps]
%   0010_O_Kedge.h5
%   0011_N_Kedge.h5
%   ...
%  Photos/ [Sample photos]
%   A0001-20251123_102407.bmp
%   A0002-20251123_102408.bmp
%   ...
%  Pressure/ [Environmental logs]
%  LabView_T_RH_20251119.txt
%  logfile-23-11-2025-12h46min.txt
%
% FILE NAMING CONVENTIONS
% -----------------------
%
% XPS FILES (.ibw, .xy)
% Format: NNNN_RegionName_PhotonEnergy.ext
% Example: 0191_C1s_525eV.ibw
% 0192_O1s_850eV.ibw
% 0045_Survey_1000eV.ibw
%
% NNNN = Four-digit SES file number (from instrument)
% RegionName = XPS region: C1s, O1s, N1s, Si2p, Cl2p, Survey, etc.
% PhotonEnergy = X-ray photon energy in eV
% .ibw = Igor file format (Scienta SES analyzer)
% .xy = Specs SpecsLab file format
%
% NEXAFS FILES (.h5)
% Format: NNNN_ElementName_EdgeType.h5
% Example: 0200_O_Kedge.h5
% 0201_N_Kedge.h5
% 0202_C_Kedge.h5
%
% NNNN = Four-digit file number
% ElementName = Element symbol: O, N, C, etc.
% EdgeType = K-edge, L-edge, etc.
%
% Note: Files starting with "N" are recognized. Files starting with
% other characters (e.g., from old naming schemes) will be ignored.
%
% PHOTO FILES (.bmp)
% Format: descriptive_name.bmp
% Example: sample_overview.bmp
% before_dosing.bmp
% after_annealing.bmp
%
% DATA FILE REQUIREMENTS & LIMITATIONS
% ====================================
%
% - At least ONE .ibw or .xy file must exist in SES/ subfolder
% - Do NOT run while SES or NEXAFS data are being actively recorded
% - Multi-scan HDF5 files are automatically split (one scan per file)
% - Region iterations are averaged during import (not preserved individually)
% - Counts are normalized to 1 second dwell time (results in cps)
% - System files starting with '.' or '~' are automatically excluded
%
% HDF5 MULTI-SCAN HANDLING
% -------------------------
% If an HDF5 file contains multiple scans (>3 groups detected):
% 1. Function calls splitH5Scans(filename)
% 2. Each scan is extracted into a separate file
% 3. New files: original_name_scan_1.h5, original_name_scan_2.h5, etc.
% 4. Original multi-scan file is deleted after splitting
%
% EXAMPLES
% ========
%
% % Example 1: Import all data from local folder
% FileList = nappFileList('2025/November_NappIss', 'mode', 'local');
%
% % Example 2: Add new data to existing table
% FileList = nappFileList('2025/November_NappIss', FileList, 'mode', 'local');
%
% % Example 3: Import only new files not yet in table
% FileList = nappFileList('2025/November_NappIss', FileList, ...
% 'mode', 'localNew');
%
% % Example 4: Import from network drive with file browser
% FileList = nappFileList('2025/November_NappIss', ...
% 'mode', 'nappDrivePick');
%
% % Example 5: Skip LabView pressure/temperature logs
% FileList = nappFileList('2025/November_NappIss', ...
% 'mode', 'local', 'importLabView', 'off');
%
% DEPENDENCIES
% ============
% - IBWread function (Igor Pro format reader for MATLAB)
% https://ch.mathworks.com/matlabcentral/fileexchange/42679-igor-pro-file-format-ibw-to-matlab-variable
% - h5info, h5read (HDF5 file support, built-in MATLAB)
% - splitH5Scans function (custom, included in toolkit)
% - LabView_importData function (custom, included in toolkit)
%
% TROUBLESHOOTING
% =======================
%
% - If "path wrong" warning appears: check folder spelling and that
% network drives are mounted
% - If Igor files fail to read: verify files are complete and not corrupted
% (partially recorded files should be renamed with "ignore_" prefix)
% - For older data (pre-2020): HDF5 file structure may differ; function
% attempts multiple path conventions automatically
% - Mac: Creation date for NEXAFS files obtained via 'stat' command
% - Windows: Creation date for NEXAFS files set to NaN
%
% MATLAB VERSION REQUIREMENTS
% ===========================
% - MATLAB 2018a or higher (for '**' notation in dir patterns)
%
% HISTORY & UPDATES
% =================
% - January 2020: Added new h5 file structure and naming convention
% - March 2021: Added local file structure support (not just network)
% - March 2021: Added Lens Mode column support
% - April 2021: X-ray kinetic energy and images now made unique
% - March 2022: Refactored to use varargin instead of positional arguments
% - March 2022: Strict data structure required for local and remote sources
% - November 2025: Added support for PShell h5 file structure (NEXAFS)
% - November 2025: Added creation date extraction for Mac (NEXAFS)
%
% See also: TABLE, IBWread, h5read, h5info, DIR, inputParser
%
% Copyright: Laboratory of Atmospheric Chemistry, Paul Scherrer Institut (PSI)
% thorsten.bartels-rausch@psi.ch
%
% =========================================================================
%% Updated
%
% * January 2020, added new h5 file structure and naming convention.
% * March 2021, added possibility to work on local file structure
% * March 2021, added Lens Mode as column (have not tested this for older
% NEXAFS files).
% * April 2021, X-ray kinetic energy and images are now made unique
% * March 2022, input now using varargin
% * March 2022, strict data structure required for local and remote sources
% * November 2025, PShell h5 files for NEXAFS have a new structure;
% % November 2025, added creation date to mac for NEXAFS.
%% ToDo
%
% * Check if jpg are imported. Before Nov25 standard format was bitmap
% * replace the hard coded table consistency with function.
%% Input
% Define conditions and defaults of input
defaultMode = 'local';
expectedMode = {'local','nappDrive','dataDrive','localNew','nappDrivePick','localPick','dataPick'};
defaultImportLabView = 'on';
expectedImportLabView = {'on','off'};
% Parse input
p = inputParser;
addRequired(p,'dataPath');
addOptional(p,'DataTable',@istable);
addParameter(p,'mode',defaultMode,...
@(x) any(validatestring(x,expectedMode)));
addParameter(p,'importLabView',defaultImportLabView,...
@(x) any(validatestring(x,expectedImportLabView)));
parse(p,dataPath,varargin{:});
%% Data Paths
%
% Mac and Windows use different notations to address network drives. For
% windows, we use the UNC naming convention. This is a more recent way to
% address mounted drives that does not rely on the use of drive letters.
% The letters might be different for the same drive on different computer.
if contains(p.Results.mode,'local')
% Data server address
if ismac
server = '/Users/bartels-rausch/Forschung_Einsam/PSI_SLS/';
elseif ispc
server = userpath;
end
elseif contains(p.Results.mode,'nappDrive')
% Data server address
if ismac
server = '/Volumes/';
elseif ispc
server = '\\fs03\';
end
elseif contains(p.Results.mode,'data')
% Data server address
if ismac
server = '/Volumes/';
elseif ispc
server = '\\fs101\';
end
end
if contains(p.Results.mode,'data')
dataPath = fullfile(server,'5505','Data',dataPath);
else
dataPath = fullfile(server,'NAPP','SC Data',dataPath);
end
% Data Paths to subdirectories for specific files:
dataPath_XPS = fullfile(dataPath, 'SES');
dataPath_Photo = fullfile(dataPath, 'Photos');
dataPath_NEXAFS = fullfile(dataPath, 'NEXAFS');
dataPath_LabView = fullfile(dataPath, 'Pressure');
%
if ~isfolder(dataPath_XPS)
warning('There is something wrong with the path. Check for typo and that the drives are mounted')
disp('There is something wrong with the path. Check for typo and that the drives are mounted')
disp('I quit now')
return
end
%% Naming Convention
%
% See: MATLAB Programming Style Guidelines, by Richard Johnson
%
% * Variable: mixed case words starting with lower case: credibleThreat, qualityOfLife
% * Variables with a large scope should have meaningful names. Variables
% with a small scope can have short names.
% * A convention on pluralization should be followed consistently.
% * Structure names should begin with a capital letter: SegmentData.length
% * The prefix n should be used for variables representing the number of
% objects. A MATLAB-specific addition is the use of m for number of rows
% (based on matrix notation). nFiles, mRows
% * Iterator variables should be named or prefixed with i. for iPosition =
% 1:nPositions, ii, ij, ik....
% * Functions should have meaningful names written in lower case.
% computetotalwidth(), Avoid compwid();
%% Query beam time folder content
%
% We use MATLAB's 'dir' command which writes the content of a folder into a
% structure with the following fields:
%
% * name
% * folder
% * date
% * bytes
% * isdir
% * datenum
%
% XPS data: Create a structure 'TmpFileList{1}' with all Igor files in the
% directory and its subfolders (1st level). The '\N*' at the beginning is
% important. Otherwise hidden files starting with a . will also be listed.
% Obviously, Igor file names from SES software and h5 file names from Epics
% (NEXAFS) need to start with an N, otherwise change the search term below:
%
% THE ** notation needs a recent version of Matlab, 2014 does not work.
if endsWith(p.Results.mode,'Pick')
% select files (Xps or Nexafs, ignoring photos) to be imported from data directory
% (data can be taken from any directory; files need to be in same directory)
[filePick,pathPick] = uigetfile({'*.ibw;*.h5';'*.xy'},...
'Select One or More Files', dataPath,...
'MultiSelect', 'on');
% make lists for Nexafs and for XPS
if ischar(filePick)
filePick = {filePick};
end
filePickXps = filePick(endsWith(filePick,'.ibw'));
filePickNexafs = filePick(endsWith(filePick,'.h5'));
filePickXY = filePick(endsWith(filePick,'.xy'));
if isempty(filePickXps)
TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.trallalla']);
else
for ii = 1:size(filePickXps,2)
TmpFileList{1}(ii,1) = dir([pathPick filePickXps{ii}]);
end
end
if isempty(filePickNexafs)
TmpFileList{2} = dir([dataPath_NEXAFS filesep '**' filesep '*.trallalla']);
else
for ii = 1:size(filePickNexafs,2)
TmpFileList{2}(ii,1) = dir([pathPick filePickNexafs{ii}]);
end
end
if isempty(filePickXY)
TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.trallalla']);
else
for ii = 1:size(filePickXps,2)
TmpFileList{1}(ii,1) = dir([pathPick filePickXps{ii}]);
end
end
% make a empty file list for photos, to be consitent
TmpFileList{3} = dir([dataPath_Photo filesep '**' filesep '*.trallalla']);
else
% SES files in IGOR format or SpecsLab in .xy format: list all files, then get rid of system files
if contains(p.Results.mode,'data')
TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.xy']);
TmpFileList{1} = TmpFileList{1}(find(~cellfun(@isempty,regexp({TmpFileList{1}.name},'^[a-zA-Z0-9]\w*')))); %#ok<FNDSB>
else
TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.ibw']);
TmpFileList{1} = TmpFileList{1}(find(~cellfun(@isempty,regexp({TmpFileList{1}.name},'^[a-zA-Z0-9]\w*')))); %#ok<FNDSB>
end
% EPICS files in h5 format list all files, then get rid of system files
TmpFileList{2} = dir([dataPath_NEXAFS filesep '**' filesep '*.h5']);
TmpFileList{2} = TmpFileList{2}(find(~cellfun(@isempty,regexp({TmpFileList{2}.name},'^[a-zA-Z0-9]\w*')))); %#ok<FNDSB>
% Photos list all files, then get rid of system files
TmpFileList{3} = dir([dataPath_Photo filesep '**' filesep '*.bmp']);
TmpFileList{3} = TmpFileList{3}(find(~cellfun(@isempty,regexp({TmpFileList{3}.name},'^[a-zA-Z0-9]\w*')))); %#ok<FNDSB>
% The second line of code filters all files that start with Numbers or
% letters. All system files starting with a . or ~ are discarded.
% =========================================================================
% SPLIT HDF5 FILES WITH MULTIPLE DATASETS INTO INDIVIDUAL FILES
% =========================================================================
%
% BACKGROUND:
% HDF5 files from the beamline can contain multiple experimental scans
% (e.g., scan_1, scan_2, scan_3) stored as separate groups within a
% single file. However, the NEXAFS and XPS data processing pipelines
% are designed to work with one scan per file.
%
% PURPOSE:
% This script identifies HDF5 files that contain multiple scans and
% splits each scan into a separate file with a descriptive name.
% Each output file contains only one scan (renamed to scan_1) plus
% the logs and metadata groups.
%
% PROCESS:
% 1. Loop through all NEXAFS HDF5 files in the file list
% 2. Check if each file contains more than 3 groups (indicating multiple scans)
% 3. If it does, split it using the splitH5Scans function
% 4. The original file is deleted after splitting
%
% OUTPUT:
% For each multi-scan file, creates separate output files:
% Example: 0002_..._O_Kedge.h5  0002_..._O_Kedge_scan_1.h5
% 0002_..._O_Kedge_scan_2.h5
% =========================================================================
% Loop through each file in the NEXAFS file list
% TmpFileList{2} contains the full paths to all NEXAFS HDF5 files
% ii is the loop counter (1, 2, 3, ... up to the total number of files)
for ii = 1:size(TmpFileList{2}, 1)
% Get the filename of the current file being processed
current_file = [TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name];
% Read information about the structure of the HDF5 file
% This tells us what groups (datasets) are stored in the file
fileinfo = h5info(current_file);
% Extract the names of all groups at the root level
% Example: {'/scan_1', '/scan_2', '/logs', '/__DATA_TYPES__'}
groupnames = {fileinfo.Groups.Name};
% Check if this file has more than 3 groups
% Files with more than 3 groups likely contain multiple scans
% (the extra groups beyond scans are typically '/logs' and '/__DATA_TYPES__')
if numel(groupnames) > 3
% This file contains multiple scans, so split it
% The splitH5Scans function will:
% - Create separate files for each scan
% - Rename each scan to '/scan_1' within its new file
% - Delete the original multi-scan file when done
splitH5Scans(current_file);
end
end
% =========================================================================
% NOTE: After this loop completes, all NEXAFS files in data folder
% will contain exactly one scan per file, ready for processing
% =========================================================================
% EPICS files in h5 format list all files, then get rid of system files
TmpFileList{2} = dir([dataPath_NEXAFS filesep '**' filesep '*.h5']);
TmpFileList{2} = TmpFileList{2}(find(~cellfun(@isempty,regexp({TmpFileList{2}.name},'^[a-zA-Z0-9]\w*')))); %#ok<FNDSB>
disp('*************************************************')
disp('* Voila: Splitting NEXAFS files done *')
disp('*************************************************')
% only process new files
if (istable(p.Results.DataTable) & matches(p.Results.mode,'localNew')) % do this only if previous list was input to function call
tmpXpsFiles = endsWith(p.Results.DataTable.name,'.ibw'); % get Scienta XPS files in current data table
tmpMissingFiles = ~ismember({TmpFileList{1}(:).name},p.Results.DataTable.name(tmpXpsFiles,:)); % find those files that are yet imported
TmpFileList{1} = TmpFileList{1}(tmpMissingFiles);
tmpXpsFiles = endsWith(p.Results.DataTable.name,'.xy'); % get Specs XPS files in current data table
tmpMissingFiles = ~ismember({TmpFileList{1}(:).name},p.Results.DataTable.name(tmpXpsFiles,:)); % find those files that are yet imported
TmpFileList{1} = TmpFileList{1}(tmpMissingFiles);
tmpNexafsFiles = endsWith(p.Results.DataTable.name,'.h5'); % get XPS files in current data table
tmpMissingFiles = ~ismember({TmpFileList{2}(:).name},p.Results.DataTable.name(tmpNexafsFiles,:)); % find those files that are yet imported
TmpFileList{2} = TmpFileList{2}(tmpMissingFiles);
tmpPhotoFiles = endsWith(p.Results.DataTable.name,'.bmp'); % get XPS files in current data table
tmpMissingFiles = ~ismember({TmpFileList{3}(:).name},p.Results.DataTable.name(tmpPhotoFiles,:)); % find those files that are yet imported
TmpFileList{3} = TmpFileList{3}(tmpMissingFiles);
end
end
% get date of first .ibw or .xy file in folder .
if contains(p.Results.mode,'data')
tmp.fileinfo = dir([dataPath_XPS '/*.xy']);
else
tmp.fileinfo = dir([dataPath_XPS '/*.ibw']);
end
if isempty(tmp.fileinfo)
warning('There appears to be no file in the SES folder. Add at least one file first.')
disp('There appears to be no file in the SES folder. Add at least one file first.')
disp('I quit now')
return
end
try
if contains(p.Results.mode,'data')
tmp.fileinfo = dir([dataPath_XPS '/*.xy']);
filedate_first = tmp.fileinfo(1).datenum;
clearvars name tmp
else
tmp.fileinfo = dir([dataPath_XPS '/*.ibw']);
filedate_first = tmp.fileinfo(1).datenum;
clearvars name tmp
end
catch
warning('There appears to be something wrong with the SES or Specs file.')
disp('There appears to be something wrong with the SES or Specs file.')
end
%% Import XPS data to get SES settings
%
% Uses the IgorPro importer for Matlab:
% https://ch.mathworks.com/matlabcentral/fileexchange/42679-igor-pro-file-format-ibw-to-matlab-variable
%
% Normally, Matlab stops when an error occurs. I use 'try' to force Matlab
% to continue even if an error occurs. Matlab will instead print a warning,
% as defined after the 'catch'. Error might occur when single data files
% can't be read. If a warning comes up, check the corresponding data file
% and try to resolve it. I suggest to rename these original data files that
% are damaged, or not complete (because measurement was aborted) to
% "ignore_filename".
if contains(p.Results.mode,'data') % for Specs xy data %TODO this is based on file location, needs work!!!!
for ii = 1:size(TmpFileList{1},1) % cycle through each XPS file
try
%data
[fid,~] = fopen([TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name],'rt');
tmpSpecsData = textscan(fid,'%f%f','Delimiter', '\b', 'CommentStyle','#'); % this reads the data; comment lines are ignored
fclose(fid);
% comments
opts = delimitedTextImportOptions("NumVariables", 1);
% Specify range and delimiter
opts.DataLines = [1, 45];
opts.Delimiter = "\t";
% Specify column names and types
opts.VariableNames = "metaData";
opts.VariableTypes = "char";
% Specify file level properties
opts.ExtraColumnsRule = "ignore";
opts.EmptyLineRule = "skip";
opts.ConsecutiveDelimitersRule = "join";
% Specify variable properties
opts = setvaropts(opts, "metaData", "WhitespaceRule", "preserve");
opts = setvaropts(opts, "metaData", "EmptyFieldRule", "auto");
tmpSpecsComments = readtable([TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name],opts);
catch
warning('Problem reading the Specs file')
disp('Problem reading the Specs file')
fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name])
end
try
%% Convert Specs meta data into variables
% data
tmpEndFirstSpectrum = find(diff(tmpSpecsData{1})<0,1); % FIXME: imports only first spectrum
% counts
PE.y = tmpSpecsData{2}(1:tmpEndFirstSpectrum);
% kinetic energy
PE.x = tmpSpecsData{1}(1:tmpEndFirstSpectrum);
% RayEnergy
PE.xRayEkin_eV{:} = str2double(extractAfter(tmpSpecsComments(29,:).metaData{:},"# Excitation Energy:"));
PE.xRayEkinRange_eV =[ PE.xRayEkin_eV{1} PE.xRayEkin_eV{end}];
% Start date/time of measurement
PE.creationDate = extractBetween(char(tmpSpecsComments(20,:).metaData{:}),"# Acquisition Date:","UTC");
PE.creationDate = datetime(PE.creationDate,'InputFormat','MM/dd/yy HH:mm:ss','PivotYear',2000,'TimeZone','UTC');
% Pass energy
PE.passEnergy = str2double(extractAfter(tmpSpecsComments(31,:).metaData{:},"# Pass Energy:"));
% Dwell time (or step time)
PE.dwellTime = str2double(extractAfter(tmpSpecsComments(28,:).metaData{:},"# Dwell Time: "));
% Energy step
tmpEnergyStep = diff(PE.x);
PE.scientaEkinStep = tmpEnergyStep(1);
if range(tmpEnergyStep) ~= 0
disp("Ups something wrong with the energy steps")
end
% Region name
PE.regionName = strtrim(extractAfter(tmpSpecsComments(18,:).metaData{:},"# Region:"));
% Acquisition Mode
PE.acquisitionMode = strtrim(extractAfter(tmpSpecsComments(25,:).metaData{:},"# Scan Mode:"));
PE.regionIterations = str2double(extractAfter(tmpSpecsComments(39,:).metaData{:},"# Number of Scans:"));
% counts/s
PE.y = PE.y./PE.regionIterations;
% Pass energy
PE.passEnergy = str2double(extractAfter(tmpSpecsComments(31,:).metaData{:},"# Pass Energy:"));
catch
warning('Problem reading the PE variable')
disp('Problem reading the PE variable')
fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name])
end
%% Write SES meta data variables into XPS FileList structure
try
TmpFileList{1}(ii).creationDate = datevec(PE.creationDate);
TmpFileList{1}(ii).image = {NaN};
TmpFileList{1}(ii).spectrum_counts = NaN;
TmpFileList{1}(ii).spectrum_countsPerSecond = PE.y;
TmpFileList{1}(ii).xRayEkin_eV = PE.xRayEkin_eV;
TmpFileList{1}(ii).xRayEkinRange_eV = PE.xRayEkinRange_eV;
% SES pass energy
TmpFileList{1}(ii).scientaPassEnergy_eV = PE.passEnergy; % in eV
% SES dwell time
TmpFileList{1}(ii).scientaDwellTime_ms = PE.dwellTime; % in ms
% SES region Name
TmpFileList{1}(ii).regionName = PE.regionName;
TmpFileList{1}(ii).scientaAcquisitionMode = PE.acquisitionMode;
% SES kinetic energy window
TmpFileList{1}(ii).scientaEkinRange_eV = [PE.x(1) PE.x(end)];
TmpFileList{1}(ii).scientaEkinStep_eV = PE.scientaEkinStep;
TmpFileList{1}(ii).scientaEkin_eV = PE.x;
% Beamline Intensity (I0 during NEXAFS)
TmpFileList{1}(ii).beamlineInt = cell(1);
% SES lens mode
TmpFileList{1}(ii).scientaLensMode = {' '};
TmpFileList{1}(ii).scientaRegionIterations = PE.regionIterations;
TmpFileList{1}(ii).scientaSequenceIterations = NaN;
TmpFileList{1}(ii).imageRoh = {NaN};
TmpFileList{1}(ii).spectrum_countsPerSecondRoh = PE.y;
TmpFileList{1}(ii).importDate = datestr(now);
try
TmpFileList{1}(ii).analysisDir = file.mFile;
catch
TmpFileList{1}(ii).analysisDir = pwd;
end
catch
warning('Problem with a XPS data file for XPS list')
disp('Problem with a XPS data file for XPS list')
fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name])
fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{1}(ii).datenum))
end
end
else % for Scienta igor data
for ii = 1:size(TmpFileList{1},1) % cycle through each XPS file
try
PE = IBWread([TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]); % read XPS file
catch
warning('Problem reading the Igor file')
disp('Problem reading the Igor file')
fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name])
end
fclose('all');
try
%% Convert SES meta data into variables
%
% Upon import, all meta data are written into _WaveNotes_ variable.
% Here, I select the meta data for each setting and write those
% into individual variables that are then added to the list.
% Split the WaveNotes text string into individual strings at each line break.
% (This is a work-around. Did not manage to select values with regexp
% function only.)
tmp = regexp(PE.WaveNotes, '\r', 'split');
% Then, read out the values into variables using a combination of string operations:
% Start date/time of measurement
PE.creationDate = datevec([char(regexp(tmp{29},'((?<=Date=).*)','match')) ...
' ' char(regexp(tmp{30},'((?<=Time=).*)','match'))]);
% Pass energy
PE.passEnergy = str2double(char(regexp(tmp{5},'((?<=Pass Energy=).*)','match')));
% Dwell time (or step time)
PE.dwellTime = str2double(char(regexp(tmp{15},'((?<=Step Time=).*)','match')));
% Energy step
PE.scientaEkinStep = str2double(char(regexp(tmp{14},'((?<=Energy Step=).*)','match')));
% Region name
PE.regionName = char(regexp(tmp{3},'((?<=Region Name=).*)','match'));
% Acquisition Mode
PE.acquisitionMode = char(regexp(tmp{9},'((?<=Acquisition Mode=).*)','match'));
% kinetic energy
for n = 1:size(PE.y,1)
PE.x(n)= PE.x0(1) + (n-1)*PE.dx(1);
end
% lens mode
PE.lensMode = char(regexp(tmp{4},'((?<=Lens Mode=).*)','match'));
% define iterations
%
% There is a BUG in this code. The SES file apparently has the same
% structure for 1 seq. iteration and several reg. iterations and
% several seq. iterations and 1 region iteration.
if PE.Ndim == 3
PE.regionIterations = size(PE.y,2);
PE.sequenceIterations = size(PE.y,3);
elseif PE.Ndim == 2
PE.regionIterations = 1;
PE.sequenceIterations = size(PE.y,2);
else
PE.regionIterations = 1;
PE.sequenceIterations = size(PE.y,2);
end
% average region iterations
if PE.Ndim == 3
PE.y = squeeze(mean(PE.y,2));
elseif PE.Ndim == 2
% PE.y = squeeze(mean(PE.y,2));
end
% normalise data to 1ms
PE.y = PE.y .* 1000 ./ PE.dwellTime;
% Pass energy
PE.passEnergy = str2double(char(regexp(tmp{5},'((?<=Pass Energy=).*)','match')));
catch
warning('Problem reading the PE variable')
disp('Problem reading the PE variable')
fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name])
end
%% Write SES meta data variables into XPS FileList structure
try
TmpFileList{1}(ii).creationDate = PE.creationDate;
TmpFileList{1}(ii).image = PE.y;
TmpFileList{1}(ii).spectrum_counts = NaN;
TmpFileList{1}(ii).spectrum_countsPerSecond = NaN;
TmpFileList{1}(ii).xRayEkin_eV = NaN;
TmpFileList{1}(ii).xRayEkinRange_eV = [NaN NaN];
% SES pass energy
TmpFileList{1}(ii).scientaPassEnergy_eV = PE.passEnergy; % in eV
% SES dwell time
TmpFileList{1}(ii).scientaDwellTime_ms = PE.dwellTime; % in ms
% SES region Name
TmpFileList{1}(ii).regionName = PE.regionName;
TmpFileList{1}(ii).scientaAcquisitionMode = PE.acquisitionMode;
% SES kinetic energy window
TmpFileList{1}(ii).scientaEkinRange_eV = [PE.x(1) PE.x(end)];
TmpFileList{1}(ii).scientaEkinStep_eV = PE.scientaEkinStep;
TmpFileList{1}(ii).scientaEkin_eV = PE.x;
% Beamline Intensity (I0 during NEXAFS)
TmpFileList{1}(ii).beamlineInt = cell(1);
% SES lens mode
TmpFileList{1}(ii).scientaLensMode = PE.lensMode;
TmpFileList{1}(ii).scientaRegionIterations = PE.regionIterations;
TmpFileList{1}(ii).scientaSequenceIterations = PE.sequenceIterations;
TmpFileList{1}(ii).imageRoh = PE.y;
TmpFileList{1}(ii).spectrum_countsPerSecondRoh = NaN;
TmpFileList{1}(ii).importDate = datestr(now);
try
TmpFileList{1}(ii).analysisDir = file.mFile;
catch
TmpFileList{1}(ii).analysisDir = pwd;
end
catch
warning('Problem with a XPS data file for XPS list')
disp('Problem with a XPS data file for XPS list')
fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name])
fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{1}(ii).datenum))
end
end
end
disp('*************************************************')
disp('* Voila: Done reading & writing XPS data! *')
disp('*************************************************')
%% Read and write NEXAFS meta data
%
% The h5 file is much nicer organised, so we do not need string operations
% to select individual meta data. The data can be directly addressed and
% written into the list structure:
% if datenum(filedate_first) >= datenum('31-Dec-2017 23:59:00') %('31-Dec-2019 23:59:00')
for ii = 1:size(TmpFileList{2},1) % do this for each list element
%if ~isempty(h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/energy'))
% creation date of file
if ismac
tmpCommand = sprintf('stat -f "%%SB" "%s"', [TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name]);
[~,creation] = system(tmpCommand);
TmpFileList{2}(ii).creationDate = datevec(creation);
else
TmpFileList{2}(ii).creationDate = datevec(NaN);
end
try
% read data
%if datenum(filedate_first) <= datenum('21-Nov-2020 00:01:00') % Date not exaclty certain.
try
TmpImage= h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/spectrum');
catch
try
TmpImage= h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/spectrum');
%else
catch
try
TmpImage = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/scienta spectrum');
catch
TmpImage = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/detector/data'); % pre-2020 (?) h5 structure
end
end
end
% 21.8.20 changed from spectrum to scienta spectrum, later
% switched back to spectrum.
% Spectrum
TmpFileList{2}(ii).spectrum_counts = NaN;
TmpFileList{2}(ii).spectrum_countsPerSecond = NaN;
% import Photon Energy (cols)
try
TmpXrayEkin = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/energy'); % X-ray excitation energy; Photon energy in eV
TmpXrayEkin(TmpXrayEkin==0)=[];
catch
try
TmpXrayEkin = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/energy'); % X-ray excitation energy; Photon energy in eV
TmpXrayEkin(TmpXrayEkin==0)=[];
catch
TmpXrayEkin = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/PhotonEnergy'); % X-ray excitation energy; Photon energy in eV
TmpXrayEkin(TmpXrayEkin==0)=[];
end
end
if isempty(TmpXrayEkin)
TmpFileList{2}(ii).xRayEkinRange_eV = [NaN NaN];
else
TmpFileList{2}(ii).xRayEkinRange_eV = [TmpXrayEkin(1) TmpXrayEkin(end)];
end
% Make unique
[TmpFileList{2}(ii).xRayEkin_eV, ind] = unique(TmpXrayEkin);
TmpFileList{2}(ii).image = TmpImage(:,ind);
TmpFileList{2}(ii).image = TmpImage(:,ind);
% NEXAFS pass energy
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/PassEnergy'); % in eV
TmpFileList{2}(ii).scientaPassEnergy_eV = str2double(tmp(1,:));
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/PassEnergy'); % in eV
TmpFileList{2}(ii).scientaPassEnergy_eV = str2double(tmp(1));
catch
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/PassEnergy'); % in eV
if tmp(1) == 1
TmpFileList{2}(ii).scientaPassEnergy_eV = 20;
elseif tmp(1) == 2
TmpFileList{2}(ii).scientaPassEnergy_eV = 50;
elseif tmp(1) == 3
TmpFileList{2}(ii).scientaPassEnergy_eV = 100;
elseif tmp(1) == 4
TmpFileList{2}(ii).scientaPassEnergy_eV = 200;
end
end
end
% NEXAFS dwell time
% if datenum(filedate_first) >= datenum('1-Feb-2020 23:59:00')
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/ScientaStepTime')*1000; % in ms
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaStepTime')*1000; % in ms
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/StepTime')*1000; % in ms
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/StepTime')*1000; % in ms
catch
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/ScientaStepTime')*1000;
end
end
end
end
TmpFileList{2}(ii).scientaDwellTime_ms = tmp(1);
% NEXAFS region Name
if TmpFileList{2}(ii).xRayEkinRange_eV(:,1) <= 530 && TmpFileList{2}(ii).xRayEkinRange_eV(:,2) >= 530
TmpFileList{2}(ii).regionName = 'O K-edge NEXAFS';
else
TmpFileList{2}(ii).regionName = 'NEXAFS';
end
% Acquisition mode
try
TmpFileList{2}(ii).scientaAcquisitionMode = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/AcquisitionMode');
catch
try
TmpFileList{2}(ii).scientaAcquisitionMode = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/AcquisitionMode');
catch
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/AcquireMode');
if tmp(1) == 0
TmpFileList{2}(ii).scientaAcquisitionMode = "Swept";
elseif tmp(1) == 1
TmpFileList{2}(ii).scientaAcquisitionMode = "Fixed";
end
clear tmp
end
end
if strcmp(TmpFileList{2}(ii).scientaAcquisitionMode,'Fixed')
% NEXAFS kinetic energy window - fixed mode
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/ScientaCenterEnergy'); % in eV
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaCenterEnergy'); % in eV
catch
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/CenterEnergy'); % in eV
end
end
TmpFileList{2}(ii).scientaEkinRange_eV = [double(tmp(1))-TmpFileList{2}(ii).scientaPassEnergy_eV/20 double(tmp(1))+TmpFileList{2}(ii).scientaPassEnergy_eV/20];
TmpFileList{2}(ii).scientaEkinStep_eV = NaN;
TmpFileList{2}(ii).scientaEkin_eV = linspace(TmpFileList{2}(ii).scientaEkinRange_eV(:,1)...
,TmpFileList{2}(ii).scientaEkinRange_eV(:,2),size(TmpFileList{2}(ii).image,1))';
else
% NEXAFS kinetic energy window - scan mode
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/ScientaLowEnergy'); % in eV
tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/ScientaHighEnergy');
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaLowEnergy'); % in eV
tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaHighEnergy');
catch
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LowEnergy'); % in eV
tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/HighEnergy');
end
end
TmpFileList{2}(ii).scientaEkinRange_eV = [tmp(1) tmp2(1)];
clearvars tmp
try
TmpFileList{2}(ii).scientaEkinStep_eV = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/StepSize');
catch
try
TmpFileList{2}(ii).scientaEkinStep_eV = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/StepSize');
catch
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/StepSize');
TmpFileList{2}(ii).scientaEkinStep_eV = abs(tmp(1));
end
end
TmpFileList{2}(ii).scientaEkin_eV = [TmpFileList{2}(ii).scientaEkinRange_eV(:,1):TmpFileList{2}(ii).scientaEkinStep_eV(1):TmpFileList{2}(ii).scientaEkinRange_eV(:,2)]';
end
clearvars tmp
% read I_0
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/i0');
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/i0');
catch
end
end
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/RM2I0');
catch
end
try
TmpFileList{2}(ii).beamlineInt = abs(tmp(ind));
catch
TmpFileList{2}(ii).beamlineInt = NaN(size(ind,1),1);
end
% Lens Mode
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/LensMode');
TmpFileList{2}(ii).scientaLensMode = tmp;
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/LensMode');
TmpFileList{2}(ii).scientaLensMode = tmp{1};
catch
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LensMode');
TmpFileList{2}(ii).scientaLensMode = num2str(tmp(1));
end
end
clearvars tmp
try
TmpFileList{2}(ii).scientaRegionIterations = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/NumIterations');
catch
try
TmpFileList{2}(ii).scientaRegionIterations = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/NumIterations');
catch
try
tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/NumIterations');
TmpFileList{2}(ii).scientaRegionIterations = double(tmp(1));
catch
TmpFileList{2}(ii).scientaRegionIterations = 1;
end
end
end
TmpFileList{2}(ii).scientaSequenceIterations = 1;
catch
warning('Problem with a NEXAFS data file in NEXAFS list')
disp('Problem with a NEXAFS data file in NEXAFS list')
fprintf('Number %.0f: %s \n',ii,[TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name])
fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{2}(ii).datenum))
end
TmpFileList{2}(ii).imageRoh = TmpFileList{2}(ii).image;
TmpFileList{2}(ii).spectrum_countsPerSecondRoh = TmpFileList{2}(ii).spectrum_countsPerSecond;
% log
TmpFileList{2}(ii).importDate = datestr(now);
try
TmpFileList{2}(ii).analysisDir = file.mFile;
catch
TmpFileList{2}(ii).analysisDir = pwd;
end
end
% else
% for ii = 1:size(TmpFileList{2},1) % do this for each list element
% try
% % NEXAFS pass energy
% tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/PassEnergy'); % in eV
% % TODO: Confirm that this assignment is correct. (The number refer to the list number in epics software)
% if tmp(1) == 1
% TmpFileList{2}(ii).passEnergy_eV = {[20]};
% elseif tmp(1) == 2
% TmpFileList{2}(ii).passEnergy_eV = {[50]};
% elseif tmp(1) == 3
% TmpFileList{2}(ii).passEnergy_eV = {[100]};
% elseif tmp(1) == 4
% TmpFileList{2}(ii).passEnergy_eV = {[200]};
% end
%
% % NEXAFS dwell time
% if datenum(filedate_first) >= datenum('1-Feb-2020 23:59:00')
% tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/ScientaStepTime')*1000; % in ms
%
% else
% tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/StepTime')*1000; % in ms
% end
% TmpFileList{2}(ii).dwellTime_ms = tmp(1);
% % NEXAFS region Name
% TmpFileList{2}(ii).regionName = 'NEXAFS';
% % FIXME: Can we differentiate O K-edge, N K-edge, C K-edge based on the h5-file meta data?
%
% % NEXAFS kinetic energy window
% tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LowEnergy'); % in eV
% tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/HighEnergy');
% TmpFileList{2}(ii).kinRange_eV = [tmp(1) tmp2(1)];
%
% % Lens mode
% tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LensMode');
% TmpFileList{2}(ii).scientaLensMode = tmp{1};
% clearvars tmp
% catch
% warning('Problem with a NEXAFS data file in NEXAFS list')
% disp('Problem with a NEXAFS data file in NEXAFS list')
% fprintf('Number %.0f: %s \n',ii,[TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name])
% fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{2}(ii).datenum))
% end
%
% end
% end
disp('*************************************************')
disp('* Voila: Done reading and writing NEXAFS data! *')
disp('*************************************************')
%% Create Picture List
%
% The file structure needs same number of fields as SES and NEXAFS data.
% Set them to NaN (Not-a-Number).
for ii = 1:size(TmpFileList{3},1) % do this for each picture in the list
TmpFileList{3}(ii).creationDate = datevec(NaN);
TmpFileList{3}(ii).image = NaN;
TmpFileList{3}(ii).spectrum_counts = NaN;
TmpFileList{3}(ii).spectrum_countsPerSecond = NaN;
TmpFileList{3}(ii).xRayEkin_eV = NaN;
TmpFileList{3}(ii).xRayEkinRange_eV = [NaN NaN];
TmpFileList{3}(ii).scientaPassEnergy_eV = NaN;
TmpFileList{3}(ii).scientaDwellTime_ms = NaN;
TmpFileList{3}(ii).regionName = ' ';
TmpFileList{3}(ii).scientaAcquisitionMode = ' ';
TmpFileList{3}(ii).scientaEkinRange_eV = [NaN NaN];
TmpFileList{3}(ii).scientaEkinStep_eV = NaN;
TmpFileList{3}(ii).scientaEkin_eV = NaN;
TmpFileList{3}(ii).beamlineInt = cell(1);
TmpFileList{3}(ii).scientaLensMode = ' ';
TmpFileList{3}(ii).scientaRegionIterations = NaN;
TmpFileList{3}(ii).scientaSequenceIterations = NaN;
TmpFileList{3}(ii).imageRoh = {NaN};
TmpFileList{3}(ii).spectrum_countsPerSecondRoh = NaN;
TmpFileList{3}(ii).importDate = ' ';
TmpFileList{3}(ii).analysisDir = ' ';
end
%% Merge lists into one
%
% get index of non-empty structures
tf =~cellfun(@isempty,TmpFileList);
% merge the non-empty list structures into one structure
ListFiles = cat(1,TmpFileList{tf});
% remove lines with empty images
for ii = 1:size(ListFiles,1)
tmpLogic(ii) = isempty(ListFiles(ii).image);
noncells = tmpLogic;
end
ListFiles(noncells) = [];
clearvars -except ListFiles server dataPath* p
%%
%% Sort by date
% First, we need to convert into a table, because structures can not be sorted:
ListFiles = struct2table(ListFiles,'AsArray',true);
% Then, we sort the table based on the time (6th column, datenum)
ListFiles = sortrows(ListFiles,6);
% Then, remove un-needed columns
ListFiles = removevars(ListFiles,{'bytes','isdir'});
disp('*************************************************')
disp('* Voila: Done creating data list! *')
disp('*************************************************')
%% Import LabView data
%
if matches(p.Results.importLabView,'on')
if isfolder(dataPath_LabView) % verify that this folder exists.
try
% Get list of all files in that folder
TmpFileList = dir([dataPath_LabView filesep '**' filesep '20*.dat']);
% import LabView data
LabView = LabView_importData(fullfile(TmpFileList(1).folder,TmpFileList(1).name)); % Overwrite LabView data in workspace, if this is the first import
if size(TmpFileList,1) > 1 % for additional data files, add these to the LabView data structure.
for ii = 2:size(TmpFileList,1)
LabView = LabView_importData(fullfile(TmpFileList(ii).folder,TmpFileList(ii).name),LabView);
end
end
catch
warning('Problem with LabView file')
disp('Problem reading LabView file')
end
try
% Get list of all files in that folder
TmpFileList = dir([dataPath_LabView filesep '**' filesep 'logfile*.txt']);
% import LabView data
LabView = LabView_importData(fullfile(TmpFileList(1).folder,TmpFileList(1).name)); % Overwrite LabView data in workspace, if this is the first import
if size(TmpFileList,1) > 1 % for additional data files, add these to the LabView data structure.
for ii = 2:size(TmpFileList,1)
LabView = LabView_importData(fullfile(TmpFileList(ii).folder,TmpFileList(ii).name),LabView);
end
end
catch
warning('Problem with LabView file')
disp('Problem reading LabView file')
end
end
disp('*************************************************')
disp('* Voila: Done reading LabView data! *')
disp('*************************************************')
clearvars TmpFileList
try
% combine data and/or rename
LabView = LabView_combine(LabView);
catch
warning('Problem with LabView file')
disp('Problem merging LabView files')
disp('Check number of columns for each file')
end
%% Extent list with LabView data
%Query time of each data file and get LabView log for that time.
for ii = 1:size(ListFiles,1)
try
% The time of the data and the logging of LabView are not
% necessarily occurring at precisely the same time. So, here define
% a time range (50 seconds) and get all
% data between these times.
tse1 = tsdata.event('Start',datenum(ListFiles.datenum(ii)),'Datenum');
tse2 = tsdata.event('End',datenum(ListFiles.datenum(ii)+seconds(50)),'Datenum');
tmp = gettsbetweenevents(LabView.merge,tse1,tse2);
if ~isempty(tmp.Data) % only do this if LabView data are not missing (because data logging was off).
% Cryo-holder set T (Sample holder temperature is in field ts.Data:1)
ListFiles.sampleTemp_dC(ii) = tmp.Data(1,1); % in �C (only get data in first row of tmp)
% Cell pressure
% (TODO: Pick correct Baratron read out: either field: ts.Data 4,8,and 9)
ListFiles.cellPressure_mbar(ii) = tmp.Data(1,4); % in mbar
% vapour pressure over ice:
if ListFiles.cellPressure_mbar(ii) >= 0
% Marti, J., and K. Mauersberger, Geophys. Res.Lett., 20, 363?366, 1993.
% gives parameterisation from 169 - 273K in Pa. (log10(p)= A1/T+B)
A_1 = -2663.5;
B_1 = 12.537;
ListFiles.iceTemp_dC(ii) = A_1/(log10(ListFiles.cellPressure_mbar(ii)*100)-B_1)-273.15;
else
ListFiles.iceTemp_dC(ii) = NaN;
end
if datenum(ListFiles.datenum(ii)) >= datenum('04-Mar-2019 13:30:00') % import additional data, logged after 03/2019
ListFiles.smplX_mm(ii) = tmp.Data(1,10);
ListFiles.smplY_mm(ii) = tmp.Data(1,11);
ListFiles.smplZ_mm(ii) = tmp.Data(1,12);
ListFiles.sealingTemp(ii) = tmp.Data(1,13);
else % if these sub-set of data is not logged, set to NaN
ListFiles.smplX_mm(ii) = NaN;
ListFiles.smplY_mm(ii) = NaN;
ListFiles.smplZ_mm(ii) = NaN;
ListFiles.sealingTemp(ii) = NaN;
end
else % if there are no LabView data stored, set all entries to NaN
ListFiles.sampleTemp_dC(ii) = NaN;
ListFiles.iceTemp_dC(ii) = NaN;
ListFiles.cellPressure_mbar(ii) = NaN;
ListFiles.smplX_mm(ii) = NaN;
ListFiles.smplY_mm(ii) = NaN;
ListFiles.smplZ_mm(ii) = NaN;
ListFiles.sealingTemp(ii) = NaN;
end
clearvars tmp
catch
warning('Problem merging LabView data into data table')
disp('Problem merging LabView data into data table')
%fprintf('Number %.0f: %s \n',ii,[ListFiles.folder(ii) filesep ListFiles.name(ii)])
%fprintf('Number %.0f: %s \n',ii,datestr(ListFiles.datenum(ii)))
end
end
disp('*************************************************')
disp('* Voila: Done writing LabView data! *')
disp('*************************************************')
end
%% Make table nicer
%
if ~iscell(ListFiles.spectrum_countsPerSecond)
ListFiles.spectrum_countsPerSecond = cell(size(ListFiles,1),1);
end
if ~iscell(ListFiles.spectrum_counts)
ListFiles.spectrum_counts = cell(size(ListFiles,1),1);
end
if ~iscell(ListFiles.xRayEkin_eV)
ListFiles.xRayEkin_eV = cell(size(ListFiles,1),1);
end
if ~iscell(ListFiles.scientaEkin_eV)
for ii = 1:size(ListFiles,1)
tmp{ii,:} = ListFiles.scientaEkin_eV(ii,:);
end
ListFiles.scientaEkin_eV = tmp;
end
% rename date fields and delete those fields not needed
[ListFiles.lastModifiedDatestr] = ListFiles.date;
[ListFiles.lastModifiedDatenum] = ListFiles.datenum;
ListFiles = removevars(ListFiles, {'date','datenum'});
% Copy Ekin Data
existColumn = strcmp('scientaEkinRoh_eV',ListFiles.Properties.VariableNames);
if any(existColumn)
ListFiles = removevars(ListFiles,'scientaEkinRoh_eV');
end
existColumn = strcmp('scientaEkinRoh_eV',ListFiles.Properties.VariableNames);
if ~any(existColumn)
% save current ekin data before they are modified
ListFiles.scientaEkinRoh_eV= ListFiles.scientaEkin_eV;
end
if ~iscell(ListFiles.scientaEkinRoh_eV)
for ii = 1:size(ListFiles,1)
tmp{ii,:} = ListFiles.scientaEkinRoh_eV(ii,:);
end
ListFiles.scientaEkinRoh_eV = tmp;
end
% work around for table variables with fitting results having the wrong
% size:
% first delete the old entries
existColumn = strcmp('peakPosition',ListFiles.Properties.VariableNames);
if any(existColumn)
ListFiles = removevars(ListFiles,'peakPosition');
end
existColumn = strcmp('peakArea',ListFiles.Properties.VariableNames);
if any(existColumn)
ListFiles = removevars(ListFiles,'peakArea');
end
existColumn = strcmp('peakFWHM',ListFiles.Properties.VariableNames);
if any(existColumn)
ListFiles = removevars(ListFiles,'peakFWHM');
end
% then re-create them with right size
% 11/22 change to 5 entries from 4.
existColumn = strcmp('peakPosition',ListFiles.Properties.VariableNames);
if ~any(existColumn)
% save current ekin data before they are modified
ListFiles.peakPosition= NaN(size(ListFiles,1),5);
end
existColumn = strcmp('peakArea',ListFiles.Properties.VariableNames);
if ~any(existColumn)
% save current ekin data before they are modified
ListFiles.peakArea= NaN(size(ListFiles,1),5);
end
existColumn = strcmp('peakFWHM',ListFiles.Properties.VariableNames);
if ~any(existColumn)
% save current ekin data before they are modified
ListFiles.peakFWHM= NaN(size(ListFiles,1),5);
end
existColumn = strcmp('sample',ListFiles.Properties.VariableNames);
if ~any(existColumn)
ListFiles.sample = cellstr(repmat(' ',size(ListFiles,1),1));
end
% add the imported data to previous list (if passed to function upon call)
if istable(p.Results.DataTable) % do this only if previous list was input to function call
% find table variables that are missing in ListFiles
existColumn = strcmp('sample',p.Results.DataTable.Properties.VariableNames);
if ~any(existColumn)
p.Results.DataTable.sample = cellstr(repmat(' ',size(p.Results.DataTable,1),1));
end
tmpMissing = ~ismember(p.Results.DataTable.Properties.VariableNames,ListFiles.Properties.VariableNames);
% find all table variables that are cells
tmpCell = cellfun(@(x) iscell(p.Results.DataTable.(x)), p.Results.DataTable.Properties.VariableNames);
% find all table variables that are numbers
tmpNumber = cellfun(@(x) isnumeric(p.Results.DataTable.(x)), p.Results.DataTable.Properties.VariableNames);
% table variabes that hold cells and are missing
tmpCellVariables = p.Results.DataTable.Properties.VariableNames(tmpMissing & tmpCell);
% table variables that hold numbers and are missing
tmpNumberVariables = p.Results.DataTable.Properties.VariableNames(tmpMissing & tmpNumber);
% add empty table variables for missing cell and for number variables
if ~isempty(tmpCellVariables)
for i = 1: size(p.Results.DataTable(:,tmpMissing & tmpCell),2)
ListFiles = addvars(ListFiles,cell(size(ListFiles,1),1),'NewVariableNames',tmpCellVariables{i});
end
end
if ~isempty(tmpNumberVariables)
for i = 1: size(p.Results.DataTable(:,tmpMissing & tmpNumber),2)
ListFiles = addvars(ListFiles,nan(size(ListFiles,1),1),'NewVariableNames',tmpNumberVariables{i});
end
end
% % % sort the tables, if they have not the same order
% tmpCompareTables = cellfun(@isequal, ListFiles.Properties.VariableNames', p.Results.DataTable.Properties.VariableNames');
%
% if ~all(tmpCompareTables)
% indexDifferenzes = find(ismember(p.Results.DataTable.Properties.VariableNames',ListFiles.Properties.VariableNames(~tmpCompareTables)','rows'));
% nameListFiles =ListFiles.Properties.VariableNames(~tmpCompareTables);
%
% for ii = 1:size(indexDifferenzes,1)
% tmpIndx = find(ismember(p.Results.DataTable.Properties.VariableNames',nameListFiles{ii},'rows'));
% tmpPreviousVarName = p.Results.DataTable.Properties.VariableNames{tmpIndx-1};
% ListFiles = movevars(ListFiles,nameListFiles{ii},'After',tmpPreviousVarName);
% fprintf("%s comes after %s \n", nameListFiles{ii},tmpPreviousVarName)
% end
%
% end
% merge tables (append imported data to previous table)
ListFiles = [p.Results.DataTable; ListFiles];
% Then, we sort the table based on the time (datenum)
ListFiles = sortrows(ListFiles,'lastModifiedDatenum');
end
%% Export
ExportList = removevars(ListFiles, {'image', 'spectrum_counts', 'spectrum_countsPerSecond', 'scientaEkin_eV', 'beamlineInt', 'imageRoh', 'scientaEkinRoh_eV', 'xRayEkin_eV'});
% TODO check windows formatting
try
% write a ;-separated list for export to excel.
writetable(ExportList,fullfile(dataPath,[datestr(ExportList.lastModifiedDatenum(1),'mmmYYYY') '_DataList.csv']),'Delimiter',',');
if ispc
% and open in Excel; works only on Windows Machines
winopen(fullfile(dataPath,[datestr(ExportList.lastModifiedDatenum(1),'mmmYYYY') '_DataList.csv']));
end
catch
warning('Problem writing Excel File (*.csv). Make sure it is not open in Excel')
end
disp('*************************************************')
disp('* Voila: nappFileList.m finished ! *')
disp('*************************************************')
end
function LabView = LabView_importData(fullFileName,varargin)
%
% LabView_importData reads data log ASCII files stored by LabView into
% a Matlab time series.
%
% LabView = LabView_importData(A) imports file A
% and stores its content as time series in a structure called "LabView".
%
% LabView = LabView_importData(A,LabView) imports file A
% and appends its content as time series to the LabView structure.
%
%
% Example
% LabView = LabView_importData('exmpl.dat',LabView)
%
% Input
% * Napp_chamber_file: Complete and absolute path to the Napp chamber log file.
% * LabView: name of output structure. Always call this'LabView'
%
% Output
% * LabView: time series, the name starting with "ts"
% followed by date and time the recording started.
%
% See also LABVIEW_COMBINE, LABVIEW_PLOT, LABVIEW_CUTDATA
%
%% Data Structure
%
% Data structure before 4.3.2019
% col 1: Date
% col 2: Time
% col 3: Reservoir "Pressure 1" - Unit "1 water" - mbar -> ts.Data:2
% col 4: Reservoir "Pressure 2" - Unit "2 other" - mbar -> ts.Data:3
% col 5: Baratron 1 - cell - 1mbar max - mbar -> ts.Data:4
% col 6: Baratron 2 - cell - 10mbar max - mbar -> ts.Data:8
% col 7: Baratron 3 - cell - 0.05mbar max - mbar -> ts.Data:9
% col 8: Baratron 4 - - mbar
% col 9: Temp.Ice Sample - �C -> ts.Data:1
% col 10: Temp.Heated Sample
% col 11: none
% col 12: Temp. Reservoir "Cooler 1" - �C -> ts.Data:5
% col 13: Temp. Reservoir "Cooler 2" - �C -> ts.Data:6
% col 14: Flow He(cryoholder) - l/min -> ts.Data:7
% col 15: none
% col 16: none
%
% Data structure after 4.3.2019
% col 1: Date
% col 2: Time
% col 3: Reservoir "Pressure 1" - Unit "1 water" - mbar -> ts.Data:2
% col 4: Reservoir "Pressure 2" - Unit "2 other" - mbar -> ts.Data:3
% col 5: Baratron 1 - cell - 1mbar max - mbar -> ts.Data:4
% col 6: Baratron 2 - cell - 10mbar max - mbar -> ts.Data:8
% col 7: Baratron 3 - cell - 0.05mbar max - mbar -> ts.Data:9
% col 8: Baratron 4 - - mbar
% col 9: Temp.Ice Sample - �C -> ts.Data:1
% col 10: Temp.Heated Sample
% col 11: Temp. Reservoir "Cooler 1" - �C -> ts.Data:5
% col 12: Temp. Reservoir "Cooler 2" - �C -> ts.Data:6
% col 13: Flow He(cryoholder) - l/min -> ts.Data:7
% col 14: Flow gas 1
% col 15: Pressure chamber - mbar
% col 16: x-position of cryoholder - mm -> ts.Data:10
% col 17: y-position of cryoholder - mm -> ts.Data:11
% col 18: z-position of cryoholder - mm -> ts.Data:12
% col 19: nothing
% col 20: Temp.Sealing - �C -> ts.Data:13
%%
% Create an InputParser object
p = inputParser;
% Add inputs to the scheme.
defaultAppendToStructure = 0;
addRequired(p,'fullFileName',@isfile);
addOptional(p,'appendToStructure',defaultAppendToStructure,@isstruct);
% Parse the inputs.
parse(p,fullFileName,varargin{:})
%%
% get filedate
[~,name,~] = fileparts(p.Results.fullFileName);
tmp = strsplit(name,'_PC');
filedate = datenum(tmp{1},'yyyy-mm-dd_HH.MM.SS');
clearvars name tmp
%%
% parse structure to append to
if isstruct(p.Results.appendToStructure)
LabView = p.Results.appendToStructure;
end
%%
if datenum(filedate) <= datenum('04-Mar-2019 13:30:00')
fid = fopen(p.Results.fullFileName);
data = textscan(fid, ...
'%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f',...
'HeaderLines',1); % data file has 2 text columns, followed by 14 numeric columns
fclose(fid);
% define time series
LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ...
= timeseries([data{9} data{3} data{4} data{5} data{12} ...
data{13} data{14} data{6} data{7}] , ...
datestr(datenum([char(data{1}), char(data{2})],...
'dd.mm.yyyyHH:MM:SS'),0));
LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ...
= 'Readout Napp Chamber raw signal';
clear fid ans formatOut nans t x
elseif datenum(filedate) >= datenum('15-Sep-2021 01:00:00') && datenum(filedate) <= datenum('01-Apr-2023 01:00:00')
% during this time, date and time were in american format.
fid = fopen(p.Results.fullFileName);
data = textscan(fid, ...
'%s %s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',...
'HeaderLines',1); %
fclose(fid);
% the date formating in the dat file is really messed up. Try to make
% it consitent with MATLAB. The problem is that MATLAB does not
% understand 1 digit month and hours.
%
%
% Split the date in column 1 into year,month,day
newDateSplit = cellfun(@(x) split(char(x),'/'),data{1},'UniformOutput', false);
% Split the time in column 2 into hours, minutes, seconds
newTimeSplit = cellfun(@(x) split(char(x),':'),data{2},'UniformOutput', false);
% combine both with the am/pm string in column 3 to a new datestring
customDateStr = cellfun(@(x,y,z) [datestr(datenum([str2double(x{3}),str2double(x{1}),str2double(x{2}),str2double(y{1}),str2double(y{2}),str2double(y{3})])), ' ', char(z)], newDateSplit,newTimeSplit,data{3}, 'uniformOutput',false) ;
% define time series
LabView.(['ts_' datestr(customDateStr{1},'YYYYmmdd_HHMM')]) ...
= timeseries([data{10} data{4} data{5} data{6} data{12} ...
data{13} data{14} data{7} data{8} data{17} data{18} data{19} data{21}], ...
customDateStr);
LabView.(['ts_' datestr(customDateStr{1},'YYYYmmdd_HHMM')]).Name ...
= 'Readout Napp Chamber raw signal';
clearvars fid ans formatOut nans t x new* custom*
elseif datenum(filedate) >= datenum('01-Apr-2023 02:00:00')
fid = fopen(p.Results.fullFileName);
data = textscan(fid, ...
'%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',...
'HeaderLines',1);
% data file has 2 text columns, followed by 18 numeric columns (THE AM
% PM IS MISSING and some changes to the LabView file. Confirm with Luca....)
fclose(fid);
% define time series
LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ...
= timeseries([data{9} data{3} data{4} data{5} data{11} ...
data{12} data{20} data{6} data{7} data{15} data{16} data{17} data{19}] , ...
datestr(datenum([char(data{1}), char(data{2})],...
'dd.mm.yyyyHH:MM:SS'),0));
LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ...
= 'Readout Napp Chamber raw signal';
clear fid ans formatOut nans t x
else
fid = fopen(p.Results.fullFileName);
data = textscan(fid, ...
'%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',...
'HeaderLines',1); % data file has 2 text columns, followed by 14 numeric columns
fclose(fid);
% define time series
LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ...
= timeseries([data{9} data{3} data{4} data{5} data{11} ...
data{12} data{13} data{6} data{7} data{16} data{17} data{18} data{20}] , ...
datestr(datenum([char(data{1}), char(data{2})],...
'dd.mm.yyyyHH:MM:SS'),0));
LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ...
= 'Readout Napp Chamber raw signal';
clear fid ans formatOut nans t x
end
end
function data_structure = LabView_combine(data_structure)
% LabView_combine can be used to merge time series into one.
%
% LabView = LabView_combine(LabView) merges all timeseries in the structure
% LabView into the new timeseries "merge"
% and adds this timeseries to the LabView structure
%
%
% Input
% * data_structure: name of the structure holding the time series
%
% Output
% * data_structure.merge: New time series combining all ts in this structure
%
% See also LABVIEW_IMPORTDATA, LABVIEW_PLOT
% delete previous merged data
if isfield(data_structure,'merge')
data_structure = rmfield(data_structure, 'merge');
end
% sort them chronologically
data_structure = orderfields(data_structure);
% query list of time series
list_ts = fieldnames(data_structure);
% and merge all of them into one time series
for ii = 1:length(list_ts)
if ii == 1
data_structure.merge = data_structure.(list_ts{1});
else
data_structure.merge = append(data_structure.merge,data_structure.(list_ts{ii}));
end
end
end
function splitH5Scans(infile)
% splitH5Scans - Split an HDF5 file containing multiple scans into separate files
%
% This function takes an HDF5 file with multiple scan groups (scan_1, scan_2, etc.)
% and creates separate files for each scan. Each output file contains only one
% scan (renamed to scan_1) plus the logs group. The original file is deleted after
% successful splitting.
%
% SYNTAX:
% splitH5Scans(infile)
%
% INPUT:
% infile - String containing the full path to the input HDF5 file
% Example: '0002_20251119_145823_O_Kedge_Nov25.h5'
%
% OUTPUT:
% Creates new HDF5 files in the same directory as the input file.
% Each file is named: [original_name]_scan_N.h5
% Example outputs:
% 0002_20251119_145823_O_Kedge_Nov25_scan_1.h5
% 0002_20251119_145823_O_Kedge_Nov25_scan_2.h5
%
% REQUIREMENTS:
% - MATLAB with HDF5 support (R2011a or later)
% - Input file must have more than 3 root-level groups to be processed
%
% EXAMPLE:
% splitH5Scans('0002_20251119_145823_O_Kedge_Nov25.h5');
%
% Author: Perplexity and Thorsten
% Date: November 2025
% Read the structure of the HDF5 file to see what groups it contains
% h5info returns information about groups, datasets, and attributes
fileinfo = h5info(infile);
% Get the original file's modification date before we start
% This will be copied to all new files to preserve the original timestamp
orig_file_info = dir(infile);
orig_mod_datetime = orig_file_info.date; % Get as datetime object
% Extract the names of all groups at the root level of the file
% Example: {'/scan_1', '/scan_2', '/logs', '/__DATA_TYPES__'}
groupnames = {fileinfo.Groups.Name};
% Only process files that have more than 3 groups at the root level
% This ensures we only split files with multiple scans
if numel(groupnames) > 3
% Find which groups are scan groups (contain 'scan_' in their name)
% This returns a logical array: true for scan groups, false otherwise
is_scan = contains(groupnames, 'scan_');
% Extract only the scan group names
% Example: {'/scan_1', '/scan_2'}
scan_names = groupnames(is_scan);
% Split the input filename into its components
% filepath: directory where file is located (e.g., 'C:\Data\')
% base: filename without extension (e.g., '0002_20251119_145823_O_Kedge_Nov25')
% ext: file extension (e.g., '.h5')
[filepath, base, ext] = fileparts(infile);
% If no directory path, use current directory
if isempty(filepath)
filepath = '.';
end
% Extract the file number prefix and rest of the name using pattern matching
% regexp splits '0002_20251119_145823_O_Kedge_Nov25' into:
% prefix = '0002_'
% rest = '20251119_145823_O_Kedge_Nov25'
pieces = regexp(base, '(\d+_)(.*)', 'tokens', 'once');
prefix = pieces{1}; % The number at the start (e.g., '0002_')
rest = pieces{2}; % Everything after the number
% Loop through each scan group and create a separate file for it
for k = 1:numel(scan_names)
% Get the full path of the current scan group
% Example: '/scan_1' or '/scan_2'
scan_group_full = scan_names{k};
% Remove the leading '/' from the scan name for use in filename
% Example: '/scan_2' becomes 'scan_2'
scan_name_clean = regexprep(scan_group_full, '^/', '');
% Build the output filename in the same directory as input file
% Example: 'C:\Data\0002_20251119_145823_O_Kedge_Nov25_scan_2.h5'
outname = fullfile(filepath, sprintf('%s%s_%s%s', prefix, rest, scan_name_clean, ext));
% Copy the entire original file to create the new output file
% We'll modify this copy to keep only one scan
copyfile(infile, outname);
% Open the new file for reading and writing using low-level HDF5 functions
% fid is a file identifier used to reference this file
% 'H5F_ACC_RDWR' means read and write access
fid = H5F.open(outname, 'H5F_ACC_RDWR', 'H5P_DEFAULT');
% Delete all scan groups except the one we want to keep
% Loop through all scan groups in the original file
for j = 1:numel(scan_names)
other_scan = scan_names{j}; % Get the name of another scan group
% If this is not the scan we want to keep, delete it
if j ~= k
try
% H5L.delete removes the link (reference) to the group
% This makes the group inaccessible but doesn't free space
H5L.delete(fid, other_scan, 'H5P_DEFAULT');
catch
% If deletion fails (e.g., group doesn't exist), continue
% The catch block prevents the script from stopping on errors
end
end
end
% Rename the kept scan group to '/scan_1' for consistency
% This way every output file has a group called '/scan_1'
if ~strcmp(scan_group_full, '/scan_1')
% H5L.move renames/moves a group within the file
% Parameters: source_file, old_path, dest_file, new_path, options
H5L.move(fid, scan_group_full, fid, '/scan_1', 'H5P_DEFAULT', 'H5P_DEFAULT');
end
% Close the HDF5 file to save all changes
H5F.close(fid);
% Copy the modification date from the original file to the new file
% Convert MATLAB datenum to timestamp format: YYYYMMDDhhmm.ss
timestamp_str = string(datetime(orig_mod_datetime), 'yyyyMMddHHmm.ss');
if ismac
touch_cmd = sprintf('touch -t %s "%s"', timestamp_str, outname);
system(touch_cmd);
end
% Print confirmation message showing which file was created
% fprintf('Created: %s\n', outname);
end
% After all split files are successfully created, delete the original file
delete(infile);
% fprintf('Deleted original file: %s\n', infile);
else
% If the file has 3 or fewer groups, it doesn't need splitting
% disp('File has three or fewer groups. No processing required.');
end
end
Reference in new issue View Git Blame Copy Permalink