function ListFiles = nappFileList(dataPath,varargin) % nappFileList lists all XAS, PE, and picture data of a specific beam % time. Analyser and sample settings such as pass energy, lens mode, % temperature, position, and pressure are added. % % % * listFiles = nappFileList('2019\April_NanoXAS') imports data from beam % time folder on local computer, generates a new matlab table 'listFiles' % in the current workspace, and a csv file from data in the % 'SC_Data/2019/April_NanoXAS' . % % * listFiles = nappFileList('2019\April_NanoXAS',listFiles) imports data from beam % time folder on local computer, adds data to the existing matlab table % 'listFiles' in the current workspace, and a csv file from data in the % 'SC_Data/2019/April_NanoXAS' . % % Input Arguments % % * dataPath, mandatory: year and name of folder that holds the beam time data. % Text string with format year/name of folder: '2019/April_NanoXAS' % * DataTable, optional: DataTable to which imported data is added. % * 'mode','local'; default: imports from local server address % * 'mode','nappDrive'; imports from napp drive. Needs to be mounted. % * 'mode','localNew'; imports only new files that have not yet been % imported. Needs existing data table as input. % * 'mode','nappDrivePick'; lets user select files in file browser window. Default address is Napp beam time folder. % * 'mode','localPick'; lets user select files in file browser window. Default address is local beam time folder. % % Dependencies % % * beam time folder must be located at server/NAPP/SC Data/dataPath. % dataPath is user input. server needs to be defined at line ~100 of this % function. % * XPS data in igor *.ibw format need to be in SES subfolder % * Photos in *.bmp format need to be in Photos folder % * NEXAFS data in *.h5 format need to be in NEXAFS subfolder % * Labview pressure and temperature log need to be in Pressure subfolder. % * Uses the IgorPro importer for Matlab: % https://ch.mathworks.com/matlabcentral/fileexchange/42679-igor-pro-file-format-ibw-to-matlab-variable % * needs at least 1 .*ibw file in the SES subfolder % % % Output % % * ListFiles: Matlab table listing XPS, NEXAFS, and photo files with settings % * YYYYmmm_DataList.csv (2019Apr_DataList.csv): Semicolon separated ASCII % file for import to Excel in beamtime folder % % % Further info % % * Do not run while SES or NEXAFS data are being recorded. Data will be % lost by SES software. % * Region iterations are averaged during import % * counts are normalised to 1s dwell time during import. That is, the % counts are in cps after import. % % % Matlab Version % % * Matlab 2018a or higher % % % See also TABLE IBWread h5read DIR LabView_importData % %% % % TODO % % * FIXME: Add videos and jpg % %% Updated % % * January 2020, added new h5 file structure and naming convention. % * March 2021, added possibility to work on local file structure % * March 2021, added Lens Mode as column (have not tested this for older % NEXAFS files). % * April 2021, X-ray kinetic energy and images are now made unique % * March 2022, input now using varargin % * March 2022, strict data structure required for local and remote sources % * November 2025, PShell h5 files for NEXAFS have a new structure; added % creation date to mac for NEXAFS. %% Input % Define conditions and defaults of input defaultMode = 'local'; expectedMode = {'local','nappDrive','dataDrive','localNew','nappDrivePick','localPick','dataPick'}; defaultImportLabView = 'on'; expectedImportLabView = {'on','off'}; % Parse input p = inputParser; addRequired(p,'dataPath'); addOptional(p,'DataTable',@istable); addParameter(p,'mode',defaultMode,... @(x) any(validatestring(x,expectedMode))); addParameter(p,'importLabView',defaultImportLabView,... @(x) any(validatestring(x,expectedImportLabView))); parse(p,dataPath,varargin{:}); %% Data Paths % % Mac and Windows use different notations to address network drives. For % windows, we use the UNC naming convention. This is a more recent way to % address mounted drives that does not rely on the use of drive letters. % The letters might be different for the same drive on different computer. if contains(p.Results.mode,'local') % Data server address if ismac server = '/Users/bartels-rausch/Forschung_Einsam/PSI_SLS/'; elseif ispc server = userpath; end elseif contains(p.Results.mode,'nappDrive') % Data server address if ismac server = '/Volumes/'; elseif ispc server = '\\fs03\'; end elseif contains(p.Results.mode,'data') % Data server address if ismac server = '/Volumes/'; elseif ispc server = '\\fs101\'; end end if contains(p.Results.mode,'data') dataPath = fullfile(server,'5505','Data',dataPath); else dataPath = fullfile(server,'NAPP','SC Data',dataPath); end % Data Paths to subdirectories for specific files: dataPath_XPS = fullfile(dataPath, 'SES'); dataPath_Photo = fullfile(dataPath, 'Photos'); dataPath_NEXAFS = fullfile(dataPath, 'NEXAFS'); dataPath_LabView = fullfile(dataPath, 'Pressure'); % if ~isfolder(dataPath_XPS) warning('There is something wrong with the path. Check for typo and that the drives are mounted') disp('There is something wrong with the path. Check for typo and that the drives are mounted') disp('I quit now') return end %% Naming Convention % % See: MATLAB Programming Style Guidelines, by Richard Johnson % % * Variable: mixed case words starting with lower case: credibleThreat, qualityOfLife % * Variables with a large scope should have meaningful names. Variables % with a small scope can have short names. % * A convention on pluralisation should be followed consistently. % * Named constants (including globals) should be all uppercase using % underscore to separate words. MAX_ITERATIONS, COLOR_RED % * Structure names should begin with a capital letter: SegmentData.length % * The prefix n should be used for variables representing the number of % objects. A MATLAB-specific addition is the use of m for number of rows % (based on matrix notation). nFiles, mRows % * Iterator variables should be named or prefixed with i. for iPosition = % 1:nPositions, ii, ij, ik.... % * Functions should have meaningful names written in lower case. % computetotalwidth(), Avoid compwid(); % %% Query beam time folder content % % We use MATLAB's 'dir' command which writes the content of a folder into a % structure with the following fields: % % * name % * folder % * date % * bytes % * isdir % * datenum % % XPS data: Create a structure 'TmpFileList{1}' with all Igor files in the % directory and its subfolders (1st level). The '\N*' at the beginning is % important. Otherwise hidden files starting with a . will also be listed. % Obviously, Igor file names from SES software and h5 file names from Epics % (NEXAFS) need to start with an N, otherwise change the search term below: % % THE ** notation needs a recent version of Matlab, 2014 does not work. if endsWith(p.Results.mode,'Pick') % select files (Xps or Nexafs, ignoring photos) to be imported from data directory % (data can be taken from any directory; files need to be in same directory) [filePick,pathPick] = uigetfile({'*.ibw;*.h5';'*.xy'},... 'Select One or More Files', dataPath,... 'MultiSelect', 'on'); % make lists for Nexafs and for XPS if ischar(filePick) filePick = {filePick}; end filePickXps = filePick(endsWith(filePick,'.ibw')); filePickNexafs = filePick(endsWith(filePick,'.h5')); filePickXY = filePick(endsWith(filePick,'.xy')); if isempty(filePickXps) TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.trallalla']); else for ii = 1:size(filePickXps,2) TmpFileList{1}(ii,1) = dir([pathPick filePickXps{ii}]); end end if isempty(filePickNexafs) TmpFileList{2} = dir([dataPath_NEXAFS filesep '**' filesep '*.trallalla']); else for ii = 1:size(filePickNexafs,2) TmpFileList{2}(ii,1) = dir([pathPick filePickNexafs{ii}]); end end if isempty(filePickXY) TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.trallalla']); else for ii = 1:size(filePickXps,2) TmpFileList{1}(ii,1) = dir([pathPick filePickXps{ii}]); end end % make a empty file list for photos, to be consitent TmpFileList{3} = dir([dataPath_Photo filesep '**' filesep '*.trallalla']); else % SES files in IGOR format or SpecsLab in .xy format: list all files, then get rid of system files if contains(p.Results.mode,'data') TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.xy']); TmpFileList{1} = TmpFileList{1}(find(~cellfun(@isempty,regexp({TmpFileList{1}.name},'^[a-zA-Z0-9]\w*')))); %#ok else TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.ibw']); TmpFileList{1} = TmpFileList{1}(find(~cellfun(@isempty,regexp({TmpFileList{1}.name},'^[a-zA-Z0-9]\w*')))); %#ok end % EPICS files in h5 format list all files, then get rid of system files TmpFileList{2} = dir([dataPath_NEXAFS filesep '**' filesep '*.h5']); TmpFileList{2} = TmpFileList{2}(find(~cellfun(@isempty,regexp({TmpFileList{2}.name},'^[a-zA-Z0-9]\w*')))); %#ok % Photos list all files, then get rid of system files TmpFileList{3} = dir([dataPath_Photo filesep '**' filesep '*.bmp']); TmpFileList{3} = TmpFileList{3}(find(~cellfun(@isempty,regexp({TmpFileList{3}.name},'^[a-zA-Z0-9]\w*')))); %#ok % The second line of code filters all files that start with Numbers or % letters. All system files starting with a . or ~ are discarded. % ========================================================================= % SPLIT HDF5 FILES WITH MULTIPLE DATASETS INTO INDIVIDUAL FILES % ========================================================================= % % BACKGROUND: % HDF5 files from the beamline can contain multiple experimental scans % (e.g., scan_1, scan_2, scan_3) stored as separate groups within a % single file. However, the NEXAFS and XPS data processing pipelines % are designed to work with one scan per file. % % PURPOSE: % This script identifies HDF5 files that contain multiple scans and % splits each scan into a separate file with a descriptive name. % Each output file contains only one scan (renamed to scan_1) plus % the logs and metadata groups. % % PROCESS: % 1. Loop through all NEXAFS HDF5 files in the file list % 2. Check if each file contains more than 3 groups (indicating multiple scans) % 3. If it does, split it using the splitH5Scans function % 4. The original file is deleted after splitting % % OUTPUT: % For each multi-scan file, creates separate output files: % Example: 0002_..._O_Kedge.h5  0002_..._O_Kedge_scan_1.h5 % 0002_..._O_Kedge_scan_2.h5 % ========================================================================= % Loop through each file in the NEXAFS file list % TmpFileList{2} contains the full paths to all NEXAFS HDF5 files % ii is the loop counter (1, 2, 3, ... up to the total number of files) for ii = 1:size(TmpFileList{2}, 1) % Get the filename of the current file being processed current_file = [TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name]; % Read information about the structure of the HDF5 file % This tells us what groups (datasets) are stored in the file fileinfo = h5info(current_file); % Extract the names of all groups at the root level % Example: {'/scan_1', '/scan_2', '/logs', '/__DATA_TYPES__'} groupnames = {fileinfo.Groups.Name}; % Check if this file has more than 3 groups % Files with more than 3 groups likely contain multiple scans % (the extra groups beyond scans are typically '/logs' and '/__DATA_TYPES__') if numel(groupnames) > 3 % This file contains multiple scans, so split it % The splitH5Scans function will: % - Create separate files for each scan % - Rename each scan to '/scan_1' within its new file % - Delete the original multi-scan file when done splitH5Scans(current_file); end end % ========================================================================= % NOTE: After this loop completes, all NEXAFS files in data folder % will contain exactly one scan per file, ready for processing % ========================================================================= % EPICS files in h5 format list all files, then get rid of system files TmpFileList{2} = dir([dataPath_NEXAFS filesep '**' filesep '*.h5']); TmpFileList{2} = TmpFileList{2}(find(~cellfun(@isempty,regexp({TmpFileList{2}.name},'^[a-zA-Z0-9]\w*')))); %#ok disp('*************************************************') disp('* Voila: Splitting NEXAFS files done *') disp('*************************************************') % only process new files if (istable(p.Results.DataTable) & matches(p.Results.mode,'localNew')) % do this only if previous list was input to function call tmpXpsFiles = endsWith(p.Results.DataTable.name,'.ibw'); % get Scienta XPS files in current data table tmpMissingFiles = ~ismember({TmpFileList{1}(:).name},p.Results.DataTable.name(tmpXpsFiles,:)); % find those files that are yet imported TmpFileList{1} = TmpFileList{1}(tmpMissingFiles); tmpXpsFiles = endsWith(p.Results.DataTable.name,'.xy'); % get Specs XPS files in current data table tmpMissingFiles = ~ismember({TmpFileList{1}(:).name},p.Results.DataTable.name(tmpXpsFiles,:)); % find those files that are yet imported TmpFileList{1} = TmpFileList{1}(tmpMissingFiles); tmpNexafsFiles = endsWith(p.Results.DataTable.name,'.h5'); % get XPS files in current data table tmpMissingFiles = ~ismember({TmpFileList{2}(:).name},p.Results.DataTable.name(tmpNexafsFiles,:)); % find those files that are yet imported TmpFileList{2} = TmpFileList{2}(tmpMissingFiles); tmpPhotoFiles = endsWith(p.Results.DataTable.name,'.bmp'); % get XPS files in current data table tmpMissingFiles = ~ismember({TmpFileList{3}(:).name},p.Results.DataTable.name(tmpPhotoFiles,:)); % find those files that are yet imported TmpFileList{3} = TmpFileList{3}(tmpMissingFiles); end end % get date of first .ibw or .xy file in folder . if contains(p.Results.mode,'data') tmp.fileinfo = dir([dataPath_XPS '/*.xy']); else tmp.fileinfo = dir([dataPath_XPS '/*.ibw']); end if isempty(tmp.fileinfo) warning('There appears to be no file in the SES folder. Add at least one file first.') disp('There appears to be no file in the SES folder. Add at least one file first.') disp('I quit now') return end try if contains(p.Results.mode,'data') tmp.fileinfo = dir([dataPath_XPS '/*.xy']); filedate_first = tmp.fileinfo(1).datenum; clearvars name tmp else tmp.fileinfo = dir([dataPath_XPS '/*.ibw']); filedate_first = tmp.fileinfo(1).datenum; clearvars name tmp end catch warning('There appears to be something wrong with the SES or Specs file.') disp('There appears to be something wrong with the SES or Specs file.') end %% Import XPS data to get SES settings % % Uses the IgorPro importer for Matlab: % https://ch.mathworks.com/matlabcentral/fileexchange/42679-igor-pro-file-format-ibw-to-matlab-variable % % Normally, Matlab stops when an error occurs. I use 'try' to force Matlab % to continue even if an error occurs. Matlab will instead print a warning, % as defined after the 'catch'. Error might occur when single data files % can't be read. If a warning comes up, check the corresponding data file % and try to resolve it. I suggest to rename these original data files that % are damaged, or not complete (because measurement was aborted) to % "ignore_filename". if contains(p.Results.mode,'data') % for Specs xy data %TODO this is based on file location, needs work!!!! for ii = 1:size(TmpFileList{1},1) % cycle through each XPS file try %data [fid,~] = fopen([TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name],'rt'); tmpSpecsData = textscan(fid,'%f%f','Delimiter', '\b', 'CommentStyle','#'); % this reads the data; comment lines are ignored fclose(fid); % comments opts = delimitedTextImportOptions("NumVariables", 1); % Specify range and delimiter opts.DataLines = [1, 45]; opts.Delimiter = "\t"; % Specify column names and types opts.VariableNames = "metaData"; opts.VariableTypes = "char"; % Specify file level properties opts.ExtraColumnsRule = "ignore"; opts.EmptyLineRule = "skip"; opts.ConsecutiveDelimitersRule = "join"; % Specify variable properties opts = setvaropts(opts, "metaData", "WhitespaceRule", "preserve"); opts = setvaropts(opts, "metaData", "EmptyFieldRule", "auto"); tmpSpecsComments = readtable([TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name],opts); catch warning('Problem reading the Specs file') disp('Problem reading the Specs file') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) end try %% Convert Specs meta data into variables % data tmpEndFirstSpectrum = find(diff(tmpSpecsData{1})<0,1); % FIXME: imports only first spectrum % counts PE.y = tmpSpecsData{2}(1:tmpEndFirstSpectrum); % kinetic energy PE.x = tmpSpecsData{1}(1:tmpEndFirstSpectrum); % RayEnergy PE.xRayEkin_eV{:} = str2double(extractAfter(tmpSpecsComments(29,:).metaData{:},"# Excitation Energy:")); PE.xRayEkinRange_eV =[ PE.xRayEkin_eV{1} PE.xRayEkin_eV{end}]; % Start date/time of measurement PE.creationDate = extractBetween(char(tmpSpecsComments(20,:).metaData{:}),"# Acquisition Date:","UTC"); PE.creationDate = datetime(PE.creationDate,'InputFormat','MM/dd/yy HH:mm:ss','PivotYear',2000,'TimeZone','UTC'); % Pass energy PE.passEnergy = str2double(extractAfter(tmpSpecsComments(31,:).metaData{:},"# Pass Energy:")); % Dwell time (or step time) PE.dwellTime = str2double(extractAfter(tmpSpecsComments(28,:).metaData{:},"# Dwell Time: ")); % Energy step tmpEnergyStep = diff(PE.x); PE.scientaEkinStep = tmpEnergyStep(1); if range(tmpEnergyStep) ~= 0 disp("Ups something wrong with the energy steps") end % Region name PE.regionName = strtrim(extractAfter(tmpSpecsComments(18,:).metaData{:},"# Region:")); % Acquisition Mode PE.acquisitionMode = strtrim(extractAfter(tmpSpecsComments(25,:).metaData{:},"# Scan Mode:")); PE.regionIterations = str2double(extractAfter(tmpSpecsComments(39,:).metaData{:},"# Number of Scans:")); % counts/s PE.y = PE.y./PE.regionIterations; % Pass energy PE.passEnergy = str2double(extractAfter(tmpSpecsComments(31,:).metaData{:},"# Pass Energy:")); catch warning('Problem reading the PE variable') disp('Problem reading the PE variable') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) end %% Write SES meta data variables into XPS FileList structure try TmpFileList{1}(ii).creationDate = datevec(PE.creationDate); TmpFileList{1}(ii).image = {NaN}; TmpFileList{1}(ii).spectrum_counts = NaN; TmpFileList{1}(ii).spectrum_countsPerSecond = PE.y; TmpFileList{1}(ii).xRayEkin_eV = PE.xRayEkin_eV; TmpFileList{1}(ii).xRayEkinRange_eV = PE.xRayEkinRange_eV; % SES pass energy TmpFileList{1}(ii).scientaPassEnergy_eV = PE.passEnergy; % in eV % SES dwell time TmpFileList{1}(ii).scientaDwellTime_ms = PE.dwellTime; % in ms % SES region Name TmpFileList{1}(ii).regionName = PE.regionName; TmpFileList{1}(ii).scientaAcquisitionMode = PE.acquisitionMode; % SES kinetic energy window TmpFileList{1}(ii).scientaEkinRange_eV = [PE.x(1) PE.x(end)]; TmpFileList{1}(ii).scientaEkinStep_eV = PE.scientaEkinStep; TmpFileList{1}(ii).scientaEkin_eV = PE.x; % Beamline Intensity (I0 during NEXAFS) TmpFileList{1}(ii).beamlineInt = cell(1); % SES lens mode TmpFileList{1}(ii).scientaLensMode = {' '}; TmpFileList{1}(ii).scientaRegionIterations = PE.regionIterations; TmpFileList{1}(ii).scientaSequenceIterations = NaN; TmpFileList{1}(ii).imageRoh = {NaN}; TmpFileList{1}(ii).spectrum_countsPerSecondRoh = PE.y; TmpFileList{1}(ii).importDate = datestr(now); try TmpFileList{1}(ii).analysisDir = file.mFile; catch TmpFileList{1}(ii).analysisDir = pwd; end catch warning('Problem with a XPS data file for XPS list') disp('Problem with a XPS data file for XPS list') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{1}(ii).datenum)) end end else % for Scienta igor data for ii = 1:size(TmpFileList{1},1) % cycle through each XPS file try PE = IBWread([TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]); % read XPS file catch warning('Problem reading the Igor file') disp('Problem reading the Igor file') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) end fclose('all'); try %% Convert SES meta data into variables % % Upon import, all meta data are written into _WaveNotes_ variable. % Here, I select the meta data for each setting and write those % into individual variables that are then added to the list. % Split the WaveNotes text string into individual strings at each line break. % (This is a work-around. Did not manage to select values with regexp % function only.) tmp = regexp(PE.WaveNotes, '\r', 'split'); % Then, read out the values into variables using a combination of string operations: % Start date/time of measurement PE.creationDate = datevec([char(regexp(tmp{29},'((?<=Date=).*)','match')) ... ' ' char(regexp(tmp{30},'((?<=Time=).*)','match'))]); % Pass energy PE.passEnergy = str2double(char(regexp(tmp{5},'((?<=Pass Energy=).*)','match'))); % Dwell time (or step time) PE.dwellTime = str2double(char(regexp(tmp{15},'((?<=Step Time=).*)','match'))); % Energy step PE.scientaEkinStep = str2double(char(regexp(tmp{14},'((?<=Energy Step=).*)','match'))); % Region name PE.regionName = char(regexp(tmp{3},'((?<=Region Name=).*)','match')); % Acquisition Mode PE.acquisitionMode = char(regexp(tmp{9},'((?<=Acquisition Mode=).*)','match')); % kinetic energy for n = 1:size(PE.y,1) PE.x(n)= PE.x0(1) + (n-1)*PE.dx(1); end % lens mode PE.lensMode = char(regexp(tmp{4},'((?<=Lens Mode=).*)','match')); % define iterations % % There is a BUG in this code. The SES file apparently has the same % structure for 1 seq. iteration and several reg. iterations and % several seq. iterations and 1 region iteration. if PE.Ndim == 3 PE.regionIterations = size(PE.y,2); PE.sequenceIterations = size(PE.y,3); elseif PE.Ndim == 2 PE.regionIterations = 1; PE.sequenceIterations = size(PE.y,2); else PE.regionIterations = 1; PE.sequenceIterations = size(PE.y,2); end % average region iterations if PE.Ndim == 3 PE.y = squeeze(mean(PE.y,2)); elseif PE.Ndim == 2 % PE.y = squeeze(mean(PE.y,2)); end % normalise data to 1ms PE.y = PE.y .* 1000 ./ PE.dwellTime; % Pass energy PE.passEnergy = str2double(char(regexp(tmp{5},'((?<=Pass Energy=).*)','match'))); catch warning('Problem reading the PE variable') disp('Problem reading the PE variable') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) end %% Write SES meta data variables into XPS FileList structure try TmpFileList{1}(ii).creationDate = PE.creationDate; TmpFileList{1}(ii).image = PE.y; TmpFileList{1}(ii).spectrum_counts = NaN; TmpFileList{1}(ii).spectrum_countsPerSecond = NaN; TmpFileList{1}(ii).xRayEkin_eV = NaN; TmpFileList{1}(ii).xRayEkinRange_eV = [NaN NaN]; % SES pass energy TmpFileList{1}(ii).scientaPassEnergy_eV = PE.passEnergy; % in eV % SES dwell time TmpFileList{1}(ii).scientaDwellTime_ms = PE.dwellTime; % in ms % SES region Name TmpFileList{1}(ii).regionName = PE.regionName; TmpFileList{1}(ii).scientaAcquisitionMode = PE.acquisitionMode; % SES kinetic energy window TmpFileList{1}(ii).scientaEkinRange_eV = [PE.x(1) PE.x(end)]; TmpFileList{1}(ii).scientaEkinStep_eV = PE.scientaEkinStep; TmpFileList{1}(ii).scientaEkin_eV = PE.x; % Beamline Intensity (I0 during NEXAFS) TmpFileList{1}(ii).beamlineInt = cell(1); % SES lens mode TmpFileList{1}(ii).scientaLensMode = PE.lensMode; TmpFileList{1}(ii).scientaRegionIterations = PE.regionIterations; TmpFileList{1}(ii).scientaSequenceIterations = PE.sequenceIterations; TmpFileList{1}(ii).imageRoh = PE.y; TmpFileList{1}(ii).spectrum_countsPerSecondRoh = NaN; TmpFileList{1}(ii).importDate = datestr(now); try TmpFileList{1}(ii).analysisDir = file.mFile; catch TmpFileList{1}(ii).analysisDir = pwd; end catch warning('Problem with a XPS data file for XPS list') disp('Problem with a XPS data file for XPS list') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{1}(ii).datenum)) end end end disp('*************************************************') disp('* Voila: Done reading & writing XPS data! *') disp('*************************************************') %% Read and write NEXAFS meta data % % The h5 file is much nicer organised, so we do not need string operations % to select individual meta data. The data can be directly addressed and % written into the list structure: % if datenum(filedate_first) >= datenum('31-Dec-2017 23:59:00') %('31-Dec-2019 23:59:00') for ii = 1:size(TmpFileList{2},1) % do this for each list element %if ~isempty(h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/energy')) % creation date of file if ismac tmpCommand = sprintf('stat -f "%%SB" "%s"', [TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name]); [~,creation] = system(tmpCommand); TmpFileList{2}(ii).creationDate = datevec(creation); else TmpFileList{2}(ii).creationDate = datevec(NaN); end try % read data %if datenum(filedate_first) <= datenum('21-Nov-2020 00:01:00') % Date not exaclty certain. try TmpImage= h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/spectrum'); catch try TmpImage= h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/spectrum'); %else catch try TmpImage = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/scienta spectrum'); catch TmpImage = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/detector/data'); % pre-2020 (?) h5 structure end end end % 21.8.20 changed from spectrum to scienta spectrum, later % switched back to spectrum. % Spectrum TmpFileList{2}(ii).spectrum_counts = NaN; TmpFileList{2}(ii).spectrum_countsPerSecond = NaN; % import Photon Energy (cols) try TmpXrayEkin = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/energy'); % X-ray excitation energy; Photon energy in eV TmpXrayEkin(TmpXrayEkin==0)=[]; catch try TmpXrayEkin = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/energy'); % X-ray excitation energy; Photon energy in eV TmpXrayEkin(TmpXrayEkin==0)=[]; catch TmpXrayEkin = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/PhotonEnergy'); % X-ray excitation energy; Photon energy in eV TmpXrayEkin(TmpXrayEkin==0)=[]; end end if isempty(TmpXrayEkin) TmpFileList{2}(ii).xRayEkinRange_eV = [NaN NaN]; else TmpFileList{2}(ii).xRayEkinRange_eV = [TmpXrayEkin(1) TmpXrayEkin(end)]; end % Make unique [TmpFileList{2}(ii).xRayEkin_eV, ind] = unique(TmpXrayEkin); TmpFileList{2}(ii).image = TmpImage(:,ind); TmpFileList{2}(ii).image = TmpImage(:,ind); % NEXAFS pass energy try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/PassEnergy'); % in eV TmpFileList{2}(ii).scientaPassEnergy_eV = str2double(tmp(1,:)); catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/PassEnergy'); % in eV TmpFileList{2}(ii).scientaPassEnergy_eV = str2double(tmp(1)); catch tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/PassEnergy'); % in eV if tmp(1) == 1 TmpFileList{2}(ii).scientaPassEnergy_eV = 20; elseif tmp(1) == 2 TmpFileList{2}(ii).scientaPassEnergy_eV = 50; elseif tmp(1) == 3 TmpFileList{2}(ii).scientaPassEnergy_eV = 100; elseif tmp(1) == 4 TmpFileList{2}(ii).scientaPassEnergy_eV = 200; end end end % NEXAFS dwell time % if datenum(filedate_first) >= datenum('1-Feb-2020 23:59:00') try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/ScientaStepTime')*1000; % in ms catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaStepTime')*1000; % in ms catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/StepTime')*1000; % in ms catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/StepTime')*1000; % in ms catch tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/ScientaStepTime')*1000; end end end end TmpFileList{2}(ii).scientaDwellTime_ms = tmp(1); % NEXAFS region Name if TmpFileList{2}(ii).xRayEkinRange_eV(:,1) <= 530 && TmpFileList{2}(ii).xRayEkinRange_eV(:,2) >= 530 TmpFileList{2}(ii).regionName = 'O K-edge NEXAFS'; else TmpFileList{2}(ii).regionName = 'NEXAFS'; end % Acquisition mode try TmpFileList{2}(ii).scientaAcquisitionMode = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/AcquisitionMode'); catch try TmpFileList{2}(ii).scientaAcquisitionMode = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/AcquisitionMode'); catch tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/AcquireMode'); if tmp(1) == 0 TmpFileList{2}(ii).scientaAcquisitionMode = "Swept"; elseif tmp(1) == 1 TmpFileList{2}(ii).scientaAcquisitionMode = "Fixed"; end clear tmp end end if strcmp(TmpFileList{2}(ii).scientaAcquisitionMode,'Fixed') % NEXAFS kinetic energy window - fixed mode try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/ScientaCenterEnergy'); % in eV catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaCenterEnergy'); % in eV catch tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/CenterEnergy'); % in eV end end TmpFileList{2}(ii).scientaEkinRange_eV = [double(tmp(1))-TmpFileList{2}(ii).scientaPassEnergy_eV/20 double(tmp(1))+TmpFileList{2}(ii).scientaPassEnergy_eV/20]; TmpFileList{2}(ii).scientaEkinStep_eV = NaN; TmpFileList{2}(ii).scientaEkin_eV = linspace(TmpFileList{2}(ii).scientaEkinRange_eV(:,1)... ,TmpFileList{2}(ii).scientaEkinRange_eV(:,2),size(TmpFileList{2}(ii).image,1))'; else % NEXAFS kinetic energy window - scan mode try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/ScientaLowEnergy'); % in eV tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/ScientaHighEnergy'); catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaLowEnergy'); % in eV tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaHighEnergy'); catch tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LowEnergy'); % in eV tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/HighEnergy'); end end TmpFileList{2}(ii).scientaEkinRange_eV = [tmp(1) tmp2(1)]; clearvars tmp try TmpFileList{2}(ii).scientaEkinStep_eV = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/StepSize'); catch try TmpFileList{2}(ii).scientaEkinStep_eV = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/StepSize'); catch tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/StepSize'); TmpFileList{2}(ii).scientaEkinStep_eV = abs(tmp(1)); end end TmpFileList{2}(ii).scientaEkin_eV = [TmpFileList{2}(ii).scientaEkinRange_eV(:,1):TmpFileList{2}(ii).scientaEkinStep_eV(1):TmpFileList{2}(ii).scientaEkinRange_eV(:,2)]'; end clearvars tmp % read I_0 try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/i0'); catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/i0'); catch end end try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/RM2I0'); catch end try TmpFileList{2}(ii).beamlineInt = abs(tmp(ind)); catch TmpFileList{2}(ii).beamlineInt = NaN(size(ind,1),1); end % Lens Mode try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/LensMode'); TmpFileList{2}(ii).scientaLensMode = tmp; catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/LensMode'); TmpFileList{2}(ii).scientaLensMode = tmp{1}; catch tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LensMode'); TmpFileList{2}(ii).scientaLensMode = num2str(tmp(1)); end end clearvars tmp try TmpFileList{2}(ii).scientaRegionIterations = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan_1/snaps/NumIterations'); catch try TmpFileList{2}(ii).scientaRegionIterations = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/NumIterations'); catch try tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/NumIterations'); TmpFileList{2}(ii).scientaRegionIterations = double(tmp(1)); catch TmpFileList{2}(ii).scientaRegionIterations = 1; end end end TmpFileList{2}(ii).scientaSequenceIterations = 1; catch warning('Problem with a NEXAFS data file in NEXAFS list') disp('Problem with a NEXAFS data file in NEXAFS list') fprintf('Number %.0f: %s \n',ii,[TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name]) fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{2}(ii).datenum)) end TmpFileList{2}(ii).imageRoh = TmpFileList{2}(ii).image; TmpFileList{2}(ii).spectrum_countsPerSecondRoh = TmpFileList{2}(ii).spectrum_countsPerSecond; % log TmpFileList{2}(ii).importDate = datestr(now); try TmpFileList{2}(ii).analysisDir = file.mFile; catch TmpFileList{2}(ii).analysisDir = pwd; end end % else % for ii = 1:size(TmpFileList{2},1) % do this for each list element % try % % NEXAFS pass energy % tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/PassEnergy'); % in eV % % TODO: Confirm that this assignment is correct. (The number refer to the list number in epics software) % if tmp(1) == 1 % TmpFileList{2}(ii).passEnergy_eV = {[20]}; % elseif tmp(1) == 2 % TmpFileList{2}(ii).passEnergy_eV = {[50]}; % elseif tmp(1) == 3 % TmpFileList{2}(ii).passEnergy_eV = {[100]}; % elseif tmp(1) == 4 % TmpFileList{2}(ii).passEnergy_eV = {[200]}; % end % % % NEXAFS dwell time % if datenum(filedate_first) >= datenum('1-Feb-2020 23:59:00') % tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/ScientaStepTime')*1000; % in ms % % else % tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/StepTime')*1000; % in ms % end % TmpFileList{2}(ii).dwellTime_ms = tmp(1); % % NEXAFS region Name % TmpFileList{2}(ii).regionName = 'NEXAFS'; % % FIXME: Can we differentiate O K-edge, N K-edge, C K-edge based on the h5-file meta data? % % % NEXAFS kinetic energy window % tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LowEnergy'); % in eV % tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/HighEnergy'); % TmpFileList{2}(ii).kinRange_eV = [tmp(1) tmp2(1)]; % % % Lens mode % tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LensMode'); % TmpFileList{2}(ii).scientaLensMode = tmp{1}; % clearvars tmp % catch % warning('Problem with a NEXAFS data file in NEXAFS list') % disp('Problem with a NEXAFS data file in NEXAFS list') % fprintf('Number %.0f: %s \n',ii,[TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name]) % fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{2}(ii).datenum)) % end % % end % end disp('*************************************************') disp('* Voila: Done reading and writing NEXAFS data! *') disp('*************************************************') %% Create Picture List % % The file structure needs same number of fields as SES and NEXAFS data. % Set them to NaN (Not-a-Number). for ii = 1:size(TmpFileList{3},1) % do this for each picture in the list TmpFileList{3}(ii).creationDate = datevec(NaN); TmpFileList{3}(ii).image = NaN; TmpFileList{3}(ii).spectrum_counts = NaN; TmpFileList{3}(ii).spectrum_countsPerSecond = NaN; TmpFileList{3}(ii).xRayEkin_eV = NaN; TmpFileList{3}(ii).xRayEkinRange_eV = [NaN NaN]; TmpFileList{3}(ii).scientaPassEnergy_eV = NaN; TmpFileList{3}(ii).scientaDwellTime_ms = NaN; TmpFileList{3}(ii).regionName = ' '; TmpFileList{3}(ii).scientaAcquisitionMode = ' '; TmpFileList{3}(ii).scientaEkinRange_eV = [NaN NaN]; TmpFileList{3}(ii).scientaEkinStep_eV = NaN; TmpFileList{3}(ii).scientaEkin_eV = NaN; TmpFileList{3}(ii).beamlineInt = cell(1); TmpFileList{3}(ii).scientaLensMode = ' '; TmpFileList{3}(ii).scientaRegionIterations = NaN; TmpFileList{3}(ii).scientaSequenceIterations = NaN; TmpFileList{3}(ii).imageRoh = {NaN}; TmpFileList{3}(ii).spectrum_countsPerSecondRoh = NaN; TmpFileList{3}(ii).importDate = ' '; TmpFileList{3}(ii).analysisDir = ' '; end %% Merge lists into one % % get index of non-empty structures tf =~cellfun(@isempty,TmpFileList); % merge the non-empty list structures into one structure ListFiles = cat(1,TmpFileList{tf}); % remove lines with empty images for ii = 1:size(ListFiles,1) tmpLogic(ii) = isempty(ListFiles(ii).image); noncells = tmpLogic; end ListFiles(noncells) = []; clearvars -except ListFiles server dataPath* p %% %% Sort by date % First, we need to convert into a table, because structures can not be sorted: ListFiles = struct2table(ListFiles,'AsArray',true); % Then, we sort the table based on the time (6th column, datenum) ListFiles = sortrows(ListFiles,6); % Then, remove un-needed columns ListFiles = removevars(ListFiles,{'bytes','isdir'}); disp('*************************************************') disp('* Voila: Done creating data list! *') disp('*************************************************') %% Import LabView data % if matches(p.Results.importLabView,'on') if isfolder(dataPath_LabView) % verify that this folder exists. try % Get list of all files in that folder TmpFileList = dir([dataPath_LabView filesep '**' filesep '20*.dat']); % import LabView data LabView = LabView_importData(fullfile(TmpFileList(1).folder,TmpFileList(1).name)); % Overwrite LabView data in workspace, if this is the first import if size(TmpFileList,1) > 1 % for additional data files, add these to the LabView data structure. for ii = 2:size(TmpFileList,1) LabView = LabView_importData(fullfile(TmpFileList(ii).folder,TmpFileList(ii).name),LabView); end end catch warning('Problem with LabView file') disp('Problem reading LabView file') end try % Get list of all files in that folder TmpFileList = dir([dataPath_LabView filesep '**' filesep 'logfile*.txt']); % import LabView data LabView = LabView_importData(fullfile(TmpFileList(1).folder,TmpFileList(1).name)); % Overwrite LabView data in workspace, if this is the first import if size(TmpFileList,1) > 1 % for additional data files, add these to the LabView data structure. for ii = 2:size(TmpFileList,1) LabView = LabView_importData(fullfile(TmpFileList(ii).folder,TmpFileList(ii).name),LabView); end end catch warning('Problem with LabView file') disp('Problem reading LabView file') end end disp('*************************************************') disp('* Voila: Done reading LabView data! *') disp('*************************************************') clearvars TmpFileList try % combine data and/or rename LabView = LabView_combine(LabView); catch warning('Problem with LabView file') disp('Problem merging LabView files') disp('Check number of columns for each file') end %% Extent list with LabView data %Query time of each data file and get LabView log for that time. for ii = 1:size(ListFiles,1) try % The time of the data and the logging of LabView are not % necessarily occurring at precisely the same time. So, here define % a time range (50 seconds) and get all % data between these times. tse1 = tsdata.event('Start',datenum(ListFiles.datenum(ii)),'Datenum'); tse2 = tsdata.event('End',datenum(ListFiles.datenum(ii)+seconds(50)),'Datenum'); tmp = gettsbetweenevents(LabView.merge,tse1,tse2); if ~isempty(tmp.Data) % only do this if LabView data are not missing (because data logging was off). % Cryo-holder set T (Sample holder temperature is in field ts.Data:1) ListFiles.sampleTemp_dC(ii) = tmp.Data(1,1); % in °C (only get data in first row of tmp) % Cell pressure % (TODO: Pick correct Baratron read out: either field: ts.Data 4,8,and 9) ListFiles.cellPressure_mbar(ii) = tmp.Data(1,4); % in mbar % vapour pressure over ice: if ListFiles.cellPressure_mbar(ii) >= 0 % Marti, J., and K. Mauersberger, Geophys. Res.Lett., 20, 363?366, 1993. % gives parameterisation from 169 - 273K in Pa. (log10(p)= A1/T+B) A_1 = -2663.5; B_1 = 12.537; ListFiles.iceTemp_dC(ii) = A_1/(log10(ListFiles.cellPressure_mbar(ii)*100)-B_1)-273.15; else ListFiles.iceTemp_dC(ii) = NaN; end if datenum(ListFiles.datenum(ii)) >= datenum('04-Mar-2019 13:30:00') % import additional data, logged after 03/2019 ListFiles.smplX_mm(ii) = tmp.Data(1,10); ListFiles.smplY_mm(ii) = tmp.Data(1,11); ListFiles.smplZ_mm(ii) = tmp.Data(1,12); ListFiles.sealingTemp(ii) = tmp.Data(1,13); else % if these sub-set of data is not logged, set to NaN ListFiles.smplX_mm(ii) = NaN; ListFiles.smplY_mm(ii) = NaN; ListFiles.smplZ_mm(ii) = NaN; ListFiles.sealingTemp(ii) = NaN; end else % if there are no LabView data stored, set all entries to NaN ListFiles.sampleTemp_dC(ii) = NaN; ListFiles.iceTemp_dC(ii) = NaN; ListFiles.cellPressure_mbar(ii) = NaN; ListFiles.smplX_mm(ii) = NaN; ListFiles.smplY_mm(ii) = NaN; ListFiles.smplZ_mm(ii) = NaN; ListFiles.sealingTemp(ii) = NaN; end clearvars tmp catch warning('Problem merging LabView data into data table') disp('Problem merging LabView data into data table') %fprintf('Number %.0f: %s \n',ii,[ListFiles.folder(ii) filesep ListFiles.name(ii)]) %fprintf('Number %.0f: %s \n',ii,datestr(ListFiles.datenum(ii))) end end disp('*************************************************') disp('* Voila: Done writing LabView data! *') disp('*************************************************') end %% Make table nicer % if ~iscell(ListFiles.spectrum_countsPerSecond) ListFiles.spectrum_countsPerSecond = cell(size(ListFiles,1),1); end if ~iscell(ListFiles.spectrum_counts) ListFiles.spectrum_counts = cell(size(ListFiles,1),1); end if ~iscell(ListFiles.xRayEkin_eV) ListFiles.xRayEkin_eV = cell(size(ListFiles,1),1); end if ~iscell(ListFiles.scientaEkin_eV) for ii = 1:size(ListFiles,1) tmp{ii,:} = ListFiles.scientaEkin_eV(ii,:); end ListFiles.scientaEkin_eV = tmp; end % rename date fields and delete those fields not needed [ListFiles.lastModifiedDatestr] = ListFiles.date; [ListFiles.lastModifiedDatenum] = ListFiles.datenum; ListFiles = removevars(ListFiles, {'date','datenum'}); % Copy Ekin Data existColumn = strcmp('scientaEkinRoh_eV',ListFiles.Properties.VariableNames); if any(existColumn) ListFiles = removevars(ListFiles,'scientaEkinRoh_eV'); end existColumn = strcmp('scientaEkinRoh_eV',ListFiles.Properties.VariableNames); if ~any(existColumn) % save current ekin data before they are modified ListFiles.scientaEkinRoh_eV= ListFiles.scientaEkin_eV; end if ~iscell(ListFiles.scientaEkinRoh_eV) for ii = 1:size(ListFiles,1) tmp{ii,:} = ListFiles.scientaEkinRoh_eV(ii,:); end ListFiles.scientaEkinRoh_eV = tmp; end % work around for table variables with fitting results having the wrong % size: % first delete the old entries existColumn = strcmp('peakPosition',ListFiles.Properties.VariableNames); if any(existColumn) ListFiles = removevars(ListFiles,'peakPosition'); end existColumn = strcmp('peakArea',ListFiles.Properties.VariableNames); if any(existColumn) ListFiles = removevars(ListFiles,'peakArea'); end existColumn = strcmp('peakFWHM',ListFiles.Properties.VariableNames); if any(existColumn) ListFiles = removevars(ListFiles,'peakFWHM'); end % then re-create them with right size % 11/22 change to 5 entries from 4. existColumn = strcmp('peakPosition',ListFiles.Properties.VariableNames); if ~any(existColumn) % save current ekin data before they are modified ListFiles.peakPosition= NaN(size(ListFiles,1),5); end existColumn = strcmp('peakArea',ListFiles.Properties.VariableNames); if ~any(existColumn) % save current ekin data before they are modified ListFiles.peakArea= NaN(size(ListFiles,1),5); end existColumn = strcmp('peakFWHM',ListFiles.Properties.VariableNames); if ~any(existColumn) % save current ekin data before they are modified ListFiles.peakFWHM= NaN(size(ListFiles,1),5); end existColumn = strcmp('sample',ListFiles.Properties.VariableNames); if ~any(existColumn) ListFiles.sample = cellstr(repmat(' ',size(ListFiles,1),1)); end % add the imported data to previous list (if passed to function upon call) if istable(p.Results.DataTable) % do this only if previous list was input to function call % find table variables that are missing in ListFiles existColumn = strcmp('sample',p.Results.DataTable.Properties.VariableNames); if ~any(existColumn) p.Results.DataTable.sample = cellstr(repmat(' ',size(p.Results.DataTable,1),1)); end tmpMissing = ~ismember(p.Results.DataTable.Properties.VariableNames,ListFiles.Properties.VariableNames); % find all table variables that are cells tmpCell = cellfun(@(x) iscell(p.Results.DataTable.(x)), p.Results.DataTable.Properties.VariableNames); % find all table variables that are numbers tmpNumber = cellfun(@(x) isnumeric(p.Results.DataTable.(x)), p.Results.DataTable.Properties.VariableNames); % table variabes that hold cells and are missing tmpCellVariables = p.Results.DataTable.Properties.VariableNames(tmpMissing & tmpCell); % table variables that hold numbers and are missing tmpNumberVariables = p.Results.DataTable.Properties.VariableNames(tmpMissing & tmpNumber); % add empty table variables for missing cell and for number variables if ~isempty(tmpCellVariables) for i = 1: size(p.Results.DataTable(:,tmpMissing & tmpCell),2) ListFiles = addvars(ListFiles,cell(size(ListFiles,1),1),'NewVariableNames',tmpCellVariables{i}); end end if ~isempty(tmpNumberVariables) for i = 1: size(p.Results.DataTable(:,tmpMissing & tmpNumber),2) ListFiles = addvars(ListFiles,nan(size(ListFiles,1),1),'NewVariableNames',tmpNumberVariables{i}); end end % % % sort the tables, if they have not the same order % tmpCompareTables = cellfun(@isequal, ListFiles.Properties.VariableNames', p.Results.DataTable.Properties.VariableNames'); % % if ~all(tmpCompareTables) % indexDifferenzes = find(ismember(p.Results.DataTable.Properties.VariableNames',ListFiles.Properties.VariableNames(~tmpCompareTables)','rows')); % nameListFiles =ListFiles.Properties.VariableNames(~tmpCompareTables); % % for ii = 1:size(indexDifferenzes,1) % tmpIndx = find(ismember(p.Results.DataTable.Properties.VariableNames',nameListFiles{ii},'rows')); % tmpPreviousVarName = p.Results.DataTable.Properties.VariableNames{tmpIndx-1}; % ListFiles = movevars(ListFiles,nameListFiles{ii},'After',tmpPreviousVarName); % fprintf("%s comes after %s \n", nameListFiles{ii},tmpPreviousVarName) % end % % end % merge tables (append imported data to previous table) ListFiles = [p.Results.DataTable; ListFiles]; % Then, we sort the table based on the time (datenum) ListFiles = sortrows(ListFiles,'lastModifiedDatenum'); end %% Export ExportList = removevars(ListFiles, {'image', 'spectrum_counts', 'spectrum_countsPerSecond', 'scientaEkin_eV', 'beamlineInt', 'imageRoh', 'scientaEkinRoh_eV', 'xRayEkin_eV'}); % TODO check windows formatting try % write a ;-separated list for export to excel. writetable(ExportList,fullfile(dataPath,[datestr(ExportList.lastModifiedDatenum(1),'mmmYYYY') '_DataList.csv']),'Delimiter',','); if ispc % and open in Excel; works only on Windows Machines winopen(fullfile(dataPath,[datestr(ExportList.lastModifiedDatenum(1),'mmmYYYY') '_DataList.csv'])); end catch warning('Problem writing Excel File (*.csv). Make sure it is not open in Excel') end disp('*************************************************') disp('* Voila: nappFileList.m finished ! *') disp('*************************************************') end function LabView = LabView_importData(fullFileName,varargin) % % LabView_importData reads data log ASCII files stored by LabView into % a Matlab time series. % % LabView = LabView_importData(A) imports file A % and stores its content as time series in a structure called "LabView". % % LabView = LabView_importData(A,LabView) imports file A % and appends its content as time series to the LabView structure. % % % Example % LabView = LabView_importData('exmpl.dat',LabView) % % Input % * Napp_chamber_file: Complete and absolute path to the Napp chamber log file. % * LabView: name of output structure. Always call this'LabView' % % Output % * LabView: time series, the name starting with "ts" % followed by date and time the recording started. % % See also LABVIEW_COMBINE, LABVIEW_PLOT, LABVIEW_CUTDATA % %% Data Structure % % Data structure before 4.3.2019 % col 1: Date % col 2: Time % col 3: Reservoir "Pressure 1" - Unit "1 water" - mbar -> ts.Data:2 % col 4: Reservoir "Pressure 2" - Unit "2 other" - mbar -> ts.Data:3 % col 5: Baratron 1 - cell - 1mbar max - mbar -> ts.Data:4 % col 6: Baratron 2 - cell - 10mbar max - mbar -> ts.Data:8 % col 7: Baratron 3 - cell - 0.05mbar max - mbar -> ts.Data:9 % col 8: Baratron 4 - - mbar % col 9: Temp.Ice Sample - °C -> ts.Data:1 % col 10: Temp.Heated Sample % col 11: none % col 12: Temp. Reservoir "Cooler 1" - °C -> ts.Data:5 % col 13: Temp. Reservoir "Cooler 2" - °C -> ts.Data:6 % col 14: Flow He(cryoholder) - l/min -> ts.Data:7 % col 15: none % col 16: none % % Data structure after 4.3.2019 % col 1: Date % col 2: Time % col 3: Reservoir "Pressure 1" - Unit "1 water" - mbar -> ts.Data:2 % col 4: Reservoir "Pressure 2" - Unit "2 other" - mbar -> ts.Data:3 % col 5: Baratron 1 - cell - 1mbar max - mbar -> ts.Data:4 % col 6: Baratron 2 - cell - 10mbar max - mbar -> ts.Data:8 % col 7: Baratron 3 - cell - 0.05mbar max - mbar -> ts.Data:9 % col 8: Baratron 4 - - mbar % col 9: Temp.Ice Sample - °C -> ts.Data:1 % col 10: Temp.Heated Sample % col 11: Temp. Reservoir "Cooler 1" - °C -> ts.Data:5 % col 12: Temp. Reservoir "Cooler 2" - °C -> ts.Data:6 % col 13: Flow He(cryoholder) - l/min -> ts.Data:7 % col 14: Flow gas 1 % col 15: Pressure chamber - mbar % col 16: x-position of cryoholder - mm -> ts.Data:10 % col 17: y-position of cryoholder - mm -> ts.Data:11 % col 18: z-position of cryoholder - mm -> ts.Data:12 % col 19: nothing % col 20: Temp.Sealing - °C -> ts.Data:13 %% % Create an InputParser object p = inputParser; % Add inputs to the scheme. defaultAppendToStructure = 0; addRequired(p,'fullFileName',@isfile); addOptional(p,'appendToStructure',defaultAppendToStructure,@isstruct); % Parse the inputs. parse(p,fullFileName,varargin{:}) %% % get filedate [~,name,~] = fileparts(p.Results.fullFileName); tmp = strsplit(name,'_PC'); filedate = datenum(tmp{1},'yyyy-mm-dd_HH.MM.SS'); clearvars name tmp %% % parse structure to append to if isstruct(p.Results.appendToStructure) LabView = p.Results.appendToStructure; end %% if datenum(filedate) <= datenum('04-Mar-2019 13:30:00') fid = fopen(p.Results.fullFileName); data = textscan(fid, ... '%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f',... 'HeaderLines',1); % data file has 2 text columns, followed by 14 numeric columns fclose(fid); % define time series LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ... = timeseries([data{9} data{3} data{4} data{5} data{12} ... data{13} data{14} data{6} data{7}] , ... datestr(datenum([char(data{1}), char(data{2})],... 'dd.mm.yyyyHH:MM:SS'),0)); LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ... = 'Readout Napp Chamber raw signal'; clear fid ans formatOut nans t x elseif datenum(filedate) >= datenum('15-Sep-2021 01:00:00') && datenum(filedate) <= datenum('01-Apr-2023 01:00:00') % during this time, date and time were in american format. fid = fopen(p.Results.fullFileName); data = textscan(fid, ... '%s %s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',... 'HeaderLines',1); % fclose(fid); % the date formating in the dat file is really messed up. Try to make % it consitent with MATLAB. The problem is that MATLAB does not % understand 1 digit month and hours. % % % Split the date in column 1 into year,month,day newDateSplit = cellfun(@(x) split(char(x),'/'),data{1},'UniformOutput', false); % Split the time in column 2 into hours, minutes, seconds newTimeSplit = cellfun(@(x) split(char(x),':'),data{2},'UniformOutput', false); % combine both with the am/pm string in column 3 to a new datestring customDateStr = cellfun(@(x,y,z) [datestr(datenum([str2double(x{3}),str2double(x{1}),str2double(x{2}),str2double(y{1}),str2double(y{2}),str2double(y{3})])), ' ', char(z)], newDateSplit,newTimeSplit,data{3}, 'uniformOutput',false) ; % define time series LabView.(['ts_' datestr(customDateStr{1},'YYYYmmdd_HHMM')]) ... = timeseries([data{10} data{4} data{5} data{6} data{12} ... data{13} data{14} data{7} data{8} data{17} data{18} data{19} data{21}], ... customDateStr); LabView.(['ts_' datestr(customDateStr{1},'YYYYmmdd_HHMM')]).Name ... = 'Readout Napp Chamber raw signal'; clearvars fid ans formatOut nans t x new* custom* elseif datenum(filedate) >= datenum('01-Apr-2023 02:00:00') fid = fopen(p.Results.fullFileName); data = textscan(fid, ... '%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',... 'HeaderLines',1); % data file has 2 text columns, followed by 18 numeric columns (THE AM % PM IS MISSING and some changes to the LabView file. Confirm with Luca....) fclose(fid); % define time series LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ... = timeseries([data{9} data{3} data{4} data{5} data{11} ... data{12} data{20} data{6} data{7} data{15} data{16} data{17} data{19}] , ... datestr(datenum([char(data{1}), char(data{2})],... 'dd.mm.yyyyHH:MM:SS'),0)); LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ... = 'Readout Napp Chamber raw signal'; clear fid ans formatOut nans t x else fid = fopen(p.Results.fullFileName); data = textscan(fid, ... '%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',... 'HeaderLines',1); % data file has 2 text columns, followed by 14 numeric columns fclose(fid); % define time series LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ... = timeseries([data{9} data{3} data{4} data{5} data{11} ... data{12} data{13} data{6} data{7} data{16} data{17} data{18} data{20}] , ... datestr(datenum([char(data{1}), char(data{2})],... 'dd.mm.yyyyHH:MM:SS'),0)); LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ... = 'Readout Napp Chamber raw signal'; clear fid ans formatOut nans t x end end function data_structure = LabView_combine(data_structure) % LabView_combine can be used to merge time series into one. % % LabView = LabView_combine(LabView) merges all timeseries in the structure % LabView into the new timeseries "merge" % and adds this timeseries to the LabView structure % % % Input % * data_structure: name of the structure holding the time series % % Output % * data_structure.merge: New time series combining all ts in this structure % % See also LABVIEW_IMPORTDATA, LABVIEW_PLOT % delete previous merged data if isfield(data_structure,'merge') data_structure = rmfield(data_structure, 'merge'); end % sort them chronologically data_structure = orderfields(data_structure); % query list of time series list_ts = fieldnames(data_structure); % and merge all of them into one time series for ii = 1:length(list_ts) if ii == 1 data_structure.merge = data_structure.(list_ts{1}); else data_structure.merge = append(data_structure.merge,data_structure.(list_ts{ii})); end end end function splitH5Scans(infile) % splitH5Scans - Split an HDF5 file containing multiple scans into separate files % % This function takes an HDF5 file with multiple scan groups (scan_1, scan_2, etc.) % and creates separate files for each scan. Each output file contains only one % scan (renamed to scan_1) plus the logs group. The original file is deleted after % successful splitting. % % SYNTAX: % splitH5Scans(infile) % % INPUT: % infile - String containing the full path to the input HDF5 file % Example: '0002_20251119_145823_O_Kedge_Nov25.h5' % % OUTPUT: % Creates new HDF5 files in the same directory as the input file. % Each file is named: [original_name]_scan_N.h5 % Example outputs: % 0002_20251119_145823_O_Kedge_Nov25_scan_1.h5 % 0002_20251119_145823_O_Kedge_Nov25_scan_2.h5 % % REQUIREMENTS: % - MATLAB with HDF5 support (R2011a or later) % - Input file must have more than 3 root-level groups to be processed % % EXAMPLE: % splitH5Scans('0002_20251119_145823_O_Kedge_Nov25.h5'); % % Author: Perplexity and Thorsten % Date: November 2025 % Read the structure of the HDF5 file to see what groups it contains % h5info returns information about groups, datasets, and attributes fileinfo = h5info(infile); % Get the original file's modification date before we start % This will be copied to all new files to preserve the original timestamp orig_file_info = dir(infile); orig_mod_datetime = orig_file_info.date; % Get as datetime object % Extract the names of all groups at the root level of the file % Example: {'/scan_1', '/scan_2', '/logs', '/__DATA_TYPES__'} groupnames = {fileinfo.Groups.Name}; % Only process files that have more than 3 groups at the root level % This ensures we only split files with multiple scans if numel(groupnames) > 3 % Find which groups are scan groups (contain 'scan_' in their name) % This returns a logical array: true for scan groups, false otherwise is_scan = contains(groupnames, 'scan_'); % Extract only the scan group names % Example: {'/scan_1', '/scan_2'} scan_names = groupnames(is_scan); % Split the input filename into its components % filepath: directory where file is located (e.g., 'C:\Data\') % base: filename without extension (e.g., '0002_20251119_145823_O_Kedge_Nov25') % ext: file extension (e.g., '.h5') [filepath, base, ext] = fileparts(infile); % If no directory path, use current directory if isempty(filepath) filepath = '.'; end % Extract the file number prefix and rest of the name using pattern matching % regexp splits '0002_20251119_145823_O_Kedge_Nov25' into: % prefix = '0002_' % rest = '20251119_145823_O_Kedge_Nov25' pieces = regexp(base, '(\d+_)(.*)', 'tokens', 'once'); prefix = pieces{1}; % The number at the start (e.g., '0002_') rest = pieces{2}; % Everything after the number % Loop through each scan group and create a separate file for it for k = 1:numel(scan_names) % Get the full path of the current scan group % Example: '/scan_1' or '/scan_2' scan_group_full = scan_names{k}; % Remove the leading '/' from the scan name for use in filename % Example: '/scan_2' becomes 'scan_2' scan_name_clean = regexprep(scan_group_full, '^/', ''); % Build the output filename in the same directory as input file % Example: 'C:\Data\0002_20251119_145823_O_Kedge_Nov25_scan_2.h5' outname = fullfile(filepath, sprintf('%s%s_%s%s', prefix, rest, scan_name_clean, ext)); % Copy the entire original file to create the new output file % We'll modify this copy to keep only one scan copyfile(infile, outname); % Open the new file for reading and writing using low-level HDF5 functions % fid is a file identifier used to reference this file % 'H5F_ACC_RDWR' means read and write access fid = H5F.open(outname, 'H5F_ACC_RDWR', 'H5P_DEFAULT'); % Delete all scan groups except the one we want to keep % Loop through all scan groups in the original file for j = 1:numel(scan_names) other_scan = scan_names{j}; % Get the name of another scan group % If this is not the scan we want to keep, delete it if j ~= k try % H5L.delete removes the link (reference) to the group % This makes the group inaccessible but doesn't free space H5L.delete(fid, other_scan, 'H5P_DEFAULT'); catch % If deletion fails (e.g., group doesn't exist), continue % The catch block prevents the script from stopping on errors end end end % Rename the kept scan group to '/scan_1' for consistency % This way every output file has a group called '/scan_1' if ~strcmp(scan_group_full, '/scan_1') % H5L.move renames/moves a group within the file % Parameters: source_file, old_path, dest_file, new_path, options H5L.move(fid, scan_group_full, fid, '/scan_1', 'H5P_DEFAULT', 'H5P_DEFAULT'); end % Close the HDF5 file to save all changes H5F.close(fid); % Copy the modification date from the original file to the new file % Convert MATLAB datenum to timestamp format: YYYYMMDDhhmm.ss timestamp_str = string(datetime(orig_mod_datetime), 'yyyyMMddHHmm.ss'); if ismac touch_cmd = sprintf('touch -t %s "%s"', timestamp_str, outname); system(touch_cmd); end % Print confirmation message showing which file was created % fprintf('Created: %s\n', outname); end % After all split files are successfully created, delete the original file delete(infile); % fprintf('Deleted original file: %s\n', infile); else % If the file has 3 or fewer groups, it doesn't need splitting % disp('File has three or fewer groups. No processing required.'); end end