function ListFiles = nappFileList(dataPath,varargin) % nappFileList lists all XAS, PE, and picture data of a specific beam % time. Analyser and sample settings such as pass energy, lens mode, % temperature, position, and pressure are added. % % % * listFiles = nappFileList('2019\April_NanoXAS') imports data from beam % time folder on local computer, generates a new matlab table 'listFiles' % in the current workspace, and a csv file from data in the % 'SC_Data/2019/April_NanoXAS' . % % * listFiles = nappFileList('2019\April_NanoXAS',listFiles) imports data from beam % time folder on local computer, adds data to the existing matlab table % 'listFiles' in the current workspace, and a csv file from data in the % 'SC_Data/2019/April_NanoXAS' . % % Input Arguments % % * dataPath, mandatory: year and name of folder that holds the beam time data. % Text string with format year/name of folder: '2019/April_NanoXAS' % * DataTable, optional: DataTable to which imported data is added. % * 'mode','local'; default: imports from local server address % * 'mode','nappDrive'; imports from napp drive. Needs to be mounted. % * 'mode','localNew'; imports only new files that have not yet been % imported. Needs existing data table as input. % * 'mode','nappDrivePick'; lets user select files in file browser window. Default address is Napp beam time folder. % * 'mode','localPick'; lets user select files in file browser window. Default address is local beam time folder. % % Dependencies % % * beam time folder must be located at server/NAPP/SC Data/dataPath. % dataPath is user input. server needs to be defined at line ~100 of this % function. % * XPS data in igor *.ibw format need to be in SES subfolder % * Photos in *.bmp format need to be in Photos folder % * NEXAFS data in *.h5 format need to be in NEXAFS subfolder % * Labview pressure and temperature log need to be in Pressure subfolder. % * Uses the IgorPro importer for Matlab: % https://ch.mathworks.com/matlabcentral/fileexchange/42679-igor-pro-file-format-ibw-to-matlab-variable % * needs at least 1 .*ibw file in the SES subfolder % % % Output % % * ListFiles: Matlab table listing XPS, NEXAFS, and photo files with settings % * YYYYmmm_DataList.csv (2019Apr_DataList.csv): Semicolon separated ASCII % file for import to Excel in beamtime folder % % % Further info % % * Do not run while SES or NEXAFS data are being recorded. Data will be % lost by SES software. % * Region iterations are averaged during import % * counts are normalised to 1s dwell time during import. That is, the % counts are in cps after import. % % % Matlab Version % % * Matlab 2018a or higher % % % See also TABLE IBWread h5read DIR LabView_importData % %% % % TODO % % * FIXME: Add videos and jpg % %% Updated % % * January 2020, added new h5 file structure and naming convention. % * March 2021, added possibility to work on local file structure % * March 2021, added Lens Mode as column (have not tested this for older % NEXAFS files). % * April 2021, X-ray kinetic energy and images are now made unique % * March 2022, input now using varargin % * March 2022, strict data structure required for local and remote sources %% Input % Define conditions and defaults of input defaultMode = 'local'; expectedMode = {'local','nappDrive','localNew','nappDrivePick','localPick'}; defaultImportLabView = 'on'; expectedImportLabView = {'on','off'}; % Parse input p = inputParser; addRequired(p,'dataPath'); addOptional(p,'DataTable',@istable); addParameter(p,'mode',defaultMode,... @(x) any(validatestring(x,expectedMode))); addParameter(p,'importLabView',defaultImportLabView,... @(x) any(validatestring(x,expectedImportLabView))); parse(p,dataPath,varargin{:}); %% Data Paths % % Mac and Windows use different notations to address network drives. For % windows, we use the UNC naming convention. This is a more recent way to % address mounted drives that does not rely on the use of drive letters. % The letters might be different for the same drive on different computer. if contains(p.Results.mode,'local') % Data server address if ismac server = '/Users/bartels-rausch/Forschung_Einsam/PSI_SLS/'; elseif ispc server = userpath; end elseif contains(p.Results.mode,'nappDrive') % Data server address if ismac server = '/Volumes/'; elseif ispc server = '\\fs03\'; end end dataPath = fullfile(server,'NAPP','SC Data',dataPath); % Data Paths to subdirectories for specific files: dataPath_XPS = fullfile(dataPath, 'SES'); dataPath_Photo = fullfile(dataPath, 'Photos'); dataPath_NEXAFS = fullfile(dataPath, 'NEXAFS'); dataPath_LabView = fullfile(dataPath, 'Pressure'); % if ~isfolder(dataPath_XPS) warning('There is something wrong with the path. Check for typo and that the drives are mounted') disp('There is something wrong with the path. Check for typo and that the drives are mounted') disp('I quit now') return end %% Naming Convention % % See: MATLAB Programming Style Guidelines, by Richard Johnson % % * Variable: mixed case words starting with lower case: credibleThreat, qualityOfLife % * Variables with a large scope should have meaningful names. Variables % with a small scope can have short names. % * A convention on pluralisation should be followed consistently. % * Named constants (including globals) should be all uppercase using % underscore to separate words. MAX_ITERATIONS, COLOR_RED % * Structure names should begin with a capital letter: SegmentData.length % * The prefix n should be used for variables representing the number of % objects. A MATLAB-specific addition is the use of m for number of rows % (based on matrix notation). nFiles, mRows % * Iterator variables should be named or prefixed with i. for iPosition = % 1:nPositions, ii, ij, ik.... % * Functions should have meaningful names written in lower case. % computetotalwidth(), Avoid compwid(); % %% Query beam time folder content % % We use MATLAB's 'dir' command which writes the content of a folder into a % structure with the following fields: % % * name % * folder % * date % * bytes % * isdir % * datenum % % XPS data: Create a structure 'TmpFileList{1}' with all Igor files in the % directory and its subfolders (1st level). The '\N*' at the beginning is % important. Otherwise hidden files starting with a . will also be listed. % Obviously, Igor file names from SES software and h5 file names from Epics % (NEXAFS) need to start with an N, otherwise change the search term below: % % THE ** notation needs a recent version of Matlab, 2014 does not work. if endsWith(p.Results.mode,'Pick') % select files (Xps or Nexafs, ignoring photos) to be imported from data directory % (data can be taken from any directory; files need to be in same directory) [filePick,pathPick] = uigetfile({'*.ibw;*.h5'},... 'Select One or More Files', dataPath,... 'MultiSelect', 'on'); % make lists for Nexafs and for XPS if ischar(filePick) filePick = {filePick}; end filePickXps = filePick(endsWith(filePick,'.ibw')); filePickNexafs = filePick(endsWith(filePick,'.h5')); if isempty(filePickXps) TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.trallalla']); else for ii = 1:size(filePickXps,2) TmpFileList{1}(ii,1) = dir([pathPick filePickXps{ii}]); end end if isempty(filePickNexafs) TmpFileList{2} = dir([dataPath_NEXAFS filesep '**' filesep '*.trallalla']); else for ii = 1:size(filePickNexafs,2) TmpFileList{2}(ii,1) = dir([pathPick filePickNexafs{ii}]); end end % make a empty file list for photos, to be consitent with code TmpFileList{3} = dir([dataPath_Photo filesep '**' filesep '*.trallalla']); else % SES files in IGOR format list all files, then get rid of system files TmpFileList{1} = dir([dataPath_XPS filesep '**' filesep '*.ibw']); TmpFileList{1} = TmpFileList{1}(find(~cellfun(@isempty,regexp({TmpFileList{1}.name},'^[a-zA-Z0-9]\w*')))); %#ok % EPICS files in h5 format list all files, then get rid of system files TmpFileList{2} = dir([dataPath_NEXAFS filesep '**' filesep '*.h5']); TmpFileList{2} = TmpFileList{2}(find(~cellfun(@isempty,regexp({TmpFileList{2}.name},'^[a-zA-Z0-9]\w*')))); %#ok % Photos list all files, then get rid of system files TmpFileList{3} = dir([dataPath_Photo filesep '**' filesep '*.bmp']); TmpFileList{3} = TmpFileList{3}(find(~cellfun(@isempty,regexp({TmpFileList{3}.name},'^[a-zA-Z0-9]\w*')))); %#ok % The second line of code filters all files that start with Numbers or % letters. All system files starting with a . or ~ are discarded. % only process new files if (istable(p.Results.DataTable) & matches(p.Results.mode,'localNew')) % do this only if previous list was input to function call tmpXpsFiles = endsWith(p.Results.DataTable.name,'.ibw'); % get XPS files in current data table tmpMissingFiles = ~ismember({TmpFileList{1}(:).name},p.Results.DataTable.name(tmpXpsFiles,:)); % find those files that are yet imported TmpFileList{1} = TmpFileList{1}(tmpMissingFiles); tmpNexafsFiles = endsWith(p.Results.DataTable.name,'.h5'); % get XPS files in current data table tmpMissingFiles = ~ismember({TmpFileList{2}(:).name},p.Results.DataTable.name(tmpNexafsFiles,:)); % find those files that are yet imported TmpFileList{2} = TmpFileList{2}(tmpMissingFiles); tmpPhotoFiles = endsWith(p.Results.DataTable.name,'.bmp'); % get XPS files in current data table tmpMissingFiles = ~ismember({TmpFileList{3}(:).name},p.Results.DataTable.name(tmpPhotoFiles,:)); % find those files that are yet imported TmpFileList{3} = TmpFileList{3}(tmpMissingFiles); end end % get date of last and first .ibw file in folder . tmp.fileinfo = dir([dataPath_XPS '/*.ibw']); if isempty(tmp.fileinfo) warning('There appears to be no file in the SES folder. Add at least one file first.') disp('There appears to be no file in the SES folder. Add at least one file first.') disp('I quit now') return end try tmp.fileinfo = dir([dataPath_XPS '/*.ibw']); filedate_first = tmp.fileinfo(1).datenum; clearvars name tmp catch disp('Error getting ibw file date.') end %% Import XPS data to get SES settings % % Uses the IgorPro importer for Matlab: % https://ch.mathworks.com/matlabcentral/fileexchange/42679-igor-pro-file-format-ibw-to-matlab-variable % % Normally, Matlab stops when an error occurs. I use 'try' to force Matlab % to continue even if an error occurs. Matlab will instead print a warning, % as defined after the 'catch'. Error might occur when single data files % can't be read. If a warning comes up, check the corresponding data file % and try to resolve it. I suggest to rename these original data files that % are damaged, or not complete (because measurement was aborted) to % "ignore_filename". for ii = 1:size(TmpFileList{1},1) % cycle through each XPS file try PE = IBWread([TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]); % read XPS file catch warning('Problem reading the Igor file') disp('Problem reading the Igor file') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) end fclose('all'); try %% Convert SES meta data into variables % % Upon import, all meta data are written into _WaveNotes_ variable. % Here, I select the meta data for each setting and write those % into individual variables that are then added to the list. % Split the WaveNotes text string into individual strings at each line break. % (This is a work-around. Did not manage to select values with regexp % function only.) tmp = regexp(PE.WaveNotes, '\r', 'split'); % Then, read out the values into variables using a combination of string operations: % Start date/time of measurement PE.creationDate = datevec([char(regexp(tmp{29},'((?<=Date=).*)','match')) ... ' ' char(regexp(tmp{30},'((?<=Time=).*)','match'))]); % Pass energy PE.passEnergy = str2double(char(regexp(tmp{5},'((?<=Pass Energy=).*)','match'))); % Dwell time (or step time) PE.dwellTime = str2double(char(regexp(tmp{15},'((?<=Step Time=).*)','match'))); % Energy step PE.scientaEkinStep = str2double(char(regexp(tmp{14},'((?<=Energy Step=).*)','match'))); % Region name PE.regionName = char(regexp(tmp{3},'((?<=Region Name=).*)','match')); % Acquisition Mode PE.acquisitionMode = char(regexp(tmp{9},'((?<=Acquisition Mode=).*)','match')); % kinetic energy for n = 1:size(PE.y,1) PE.x(n)= PE.x0(1) + (n-1)*PE.dx(1); end % lens mode PE.lensMode = char(regexp(tmp{4},'((?<=Lens Mode=).*)','match')); % define iterations if PE.Ndim == 3 PE.regionIterations = size(PE.y,2); PE.sequenceIterations = size(PE.y,3); else PE.regionIterations = 1; PE.sequenceIterations = size(PE.y,2); end % average region iterations if PE.Ndim == 3 PE.y = squeeze(mean(PE.y,2)); end % normalise data to 1ms PE.y = PE.y .* 1000 ./ PE.dwellTime; % Pass energy PE.passEnergy = str2double(char(regexp(tmp{5},'((?<=Pass Energy=).*)','match'))); catch warning('Problem reading the PE variable') disp('Problem reading the PE variable') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) end %% Write SES meta data variables into XPS FileList structure try TmpFileList{1}(ii).creationDate = PE.creationDate; TmpFileList{1}(ii).image = PE.y; TmpFileList{1}(ii).spectrum_counts = NaN; TmpFileList{1}(ii).spectrum_countsPerSecond = NaN; TmpFileList{1}(ii).xRayEkin_eV = NaN; TmpFileList{1}(ii).xRayEkinRange_eV = [NaN NaN]; % SES pass energy TmpFileList{1}(ii).scientaPassEnergy_eV = PE.passEnergy; % in eV % SES dwell time TmpFileList{1}(ii).scientaDwellTime_ms = PE.dwellTime; % in ms % SES region Name TmpFileList{1}(ii).regionName = PE.regionName; TmpFileList{1}(ii).scientaAcquisitionMode = PE.acquisitionMode; % SES kinetic energy window TmpFileList{1}(ii).scientaEkinRange_eV = [PE.x(1) PE.x(end)]; TmpFileList{1}(ii).scientaEkinStep_eV = PE.scientaEkinStep; TmpFileList{1}(ii).scientaEkin_eV = PE.x; % Beamline Intensity (I0 during NEXAFS) TmpFileList{1}(ii).beamlineInt = cell(1); % SES lens mode TmpFileList{1}(ii).scientaLensMode = PE.lensMode; TmpFileList{1}(ii).scientaRegionIterations = PE.regionIterations; TmpFileList{1}(ii).scientaSequenceIterations = PE.sequenceIterations; TmpFileList{1}(ii).imageRoh = PE.y; TmpFileList{1}(ii).spectrum_countsPerSecondRoh = NaN; TmpFileList{1}(ii).importDate = datestr(now); try TmpFileList{1}(ii).analysisDir = file.mFile; catch TmpFileList{1}(ii).analysisDir = pwd; end catch warning('Problem with a XPS data file for XPS list') disp('Problem with a XPS data file for XPS list') fprintf('Number %.0f: %s \n',ii,[TmpFileList{1}(ii).folder filesep TmpFileList{1}(ii).name]) fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{1}(ii).datenum)) end end disp('*************************************************') disp('* Voila: Done reading & writing XPS data! *') disp('*************************************************') %% Read and write NEXAFS meta data % % The h5 file is much nicer organised, so we do not need string operations % to select individual meta data. The data can be directly addressed and % written into the list structure: if datenum(filedate_first) >= datenum('31-Dec-2019 23:59:00') for ii = 1:size(TmpFileList{2},1) % do this for each list element %if ~isempty(h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/energy')) % creation date of file TmpFileList{2}(ii).creationDate = datevec(NaN); try % read data %if datenum(filedate_first) <= datenum('21-Nov-2020 00:01:00') % Date not exaclty certain. try TmpImage= h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/spectrum'); %else catch TmpImage = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/scienta spectrum'); end % 21.8.20 changed from spectrum to scienta spectrum, later % switched back to scienta. % Spectrum TmpFileList{2}(ii).spectrum_counts = NaN; TmpFileList{2}(ii).spectrum_countsPerSecond = NaN; % import Photon Energy (cols) TmpXrayEkin = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/energy'); % X-ray excitation energy; Photon energy in eV TmpXrayEkin(TmpXrayEkin==0)=[]; if isempty(TmpXrayEkin) TmpFileList{2}(ii).xRayEkinRange_eV = [NaN NaN]; else TmpFileList{2}(ii).xRayEkinRange_eV = [TmpXrayEkin(1) TmpXrayEkin(end)]; end % Make unique [TmpFileList{2}(ii).xRayEkin_eV, ind] = unique(TmpXrayEkin); TmpFileList{2}(ii).image = TmpImage(:,ind); % NEXAFS pass energy tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/PassEnergy'); % in eV TmpFileList{2}(ii).scientaPassEnergy_eV = str2double(tmp(1)); % NEXAFS dwell time if datenum(filedate_first) >= datenum('1-Feb-2020 23:59:00') tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaStepTime')*1000; % in ms else tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/StepTime')*1000; % in ms end TmpFileList{2}(ii).scientaDwellTime_ms = tmp(1); % NEXAFS region Name if TmpFileList{2}(ii).xRayEkinRange_eV(:,1) <= 530 && TmpFileList{2}(ii).xRayEkinRange_eV(:,2) >= 530 TmpFileList{2}(ii).regionName = 'O K-edge NEXAFS'; else TmpFileList{2}(ii).regionName = 'NEXAFS'; end % Acquisition mode TmpFileList{2}(ii).scientaAcquisitionMode = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/AcquisitionMode'); if strcmp(TmpFileList{2}(ii).scientaAcquisitionMode,'Fixed') % NEXAFS kinetic energy window - fixed mode tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaCenterEnergy'); % in eV TmpFileList{2}(ii).scientaEkinRange_eV = [tmp(1)-TmpFileList{2}(ii).scientaPassEnergy_eV/20 tmp(1)+TmpFileList{2}(ii).scientaPassEnergy_eV/20]; TmpFileList{2}(ii).scientaEkinStep_eV = NaN; TmpFileList{2}(ii).scientaEkin_eV = linspace(TmpFileList{2}(ii).scientaEkinRange_eV(:,1)... ,TmpFileList{2}(ii).scientaEkinRange_eV(:,2),size(TmpFileList{2}(ii).image,1))'; else % NEXAFS kinetic energy window - scan mode tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaLowEnergy'); % in eV tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/ScientaHighEnergy'); TmpFileList{2}(ii).scientaEkinRange_eV = [tmp(1) tmp2(1)]; TmpFileList{2}(ii).scientaEkinStep_eV = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/StepSize'); TmpFileList{2}(ii).scientaEkin_eV = [TmpFileList{2}(ii).scientaEkinRange_eV(:,1):TmpFileList{2}(ii).scientaEkinStep_eV(1):TmpFileList{2}(ii).scientaEkinRange_eV(:,2)]'; end % read I_0 tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/i0'); TmpFileList{2}(ii).beamlineInt = abs(tmp(ind)); % Lens Mode tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/LensMode'); TmpFileList{2}(ii).scientaLensMode = tmp{1}; clearvars tmp TmpFileList{2}(ii).scientaRegionIterations = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/scan 1/attrs/NumIterations'); TmpFileList{2}(ii).scientaSequenceIterations = 1; catch warning('Problem with a NEXAFS data file in NEXAFS list') disp('Problem with a NEXAFS data file in NEXAFS list') fprintf('Number %.0f: %s \n',ii,[TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name]) fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{2}(ii).datenum)) end TmpFileList{2}(ii).imageRoh = TmpFileList{2}(ii).image; TmpFileList{2}(ii).spectrum_countsPerSecondRoh = TmpFileList{2}(ii).spectrum_countsPerSecond; % log TmpFileList{2}(ii).importDate = datestr(now); try TmpFileList{2}(ii).analysisDir = file.mFile; catch TmpFileList{2}(ii).analysisDir = pwd; end end %end else for ii = 1:size(TmpFileList{2},1) % do this for each list element try % NEXAFS pass energy tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/PassEnergy'); % in eV % TODO: Confirm that this assignment is correct. (The number refer to the list number in epics software) if tmp(1) == 1 TmpFileList{2}(ii).passEnergy_eV = {[20]}; elseif tmp(1) == 2 TmpFileList{2}(ii).passEnergy_eV = {[50]}; elseif tmp(1) == 3 TmpFileList{2}(ii).passEnergy_eV = {[100]}; elseif tmp(1) == 4 TmpFileList{2}(ii).passEnergy_eV = {[200]}; end % NEXAFS dwell time if datenum(filedate_first) >= datenum('1-Feb-2020 23:59:00') tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/ScientaStepTime')*1000; % in ms else tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/StepTime')*1000; % in ms end TmpFileList{2}(ii).dwellTime_ms = tmp(1); % NEXAFS region Name TmpFileList{2}(ii).regionName = 'NEXAFS'; % FIXME: Can we differentiate O K-edge, N K-edge, C K-edge based on the h5-file meta data? % NEXAFS kinetic energy window tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LowEnergy'); % in eV tmp2 = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/HighEnergy'); TmpFileList{2}(ii).kinRange_eV = [tmp(1) tmp2(1)]; % Lens mode tmp = h5read([TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name],'/entry/instrument/NDAttributes/LensMode'); TmpFileList{2}(ii).scientaLensMode = tmp{1}; clearvars tmp catch warning('Problem with a NEXAFS data file in NEXAFS list') disp('Problem with a NEXAFS data file in NEXAFS list') fprintf('Number %.0f: %s \n',ii,[TmpFileList{2}(ii).folder filesep TmpFileList{2}(ii).name]) fprintf('Number %.0f: %s \n',ii,datestr(TmpFileList{2}(ii).datenum)) end end end % check for empty entries disp('*************************************************') disp('* Voila: Done reading and writing NEXAFS data! *') disp('*************************************************') %% Create Picture List % % The file structure needs same number of fields as SES and NEXAFS data. % Set them to NaN (Not-a-Number). for ii = 1:size(TmpFileList{3},1) % do this for each picture in the list TmpFileList{3}(ii).creationDate = datevec(NaN); TmpFileList{3}(ii).image = NaN; TmpFileList{3}(ii).spectrum_counts = NaN; TmpFileList{3}(ii).spectrum_countsPerSecond = NaN; TmpFileList{3}(ii).xRayEkin_eV = NaN; TmpFileList{3}(ii).xRayEkinRange_eV = [NaN NaN]; TmpFileList{3}(ii).scientaPassEnergy_eV = NaN; TmpFileList{3}(ii).scientaDwellTime_ms = NaN; TmpFileList{3}(ii).regionName = ' '; TmpFileList{3}(ii).scientaAcquisitionMode = ' '; TmpFileList{3}(ii).scientaEkinRange_eV = [NaN NaN]; TmpFileList{3}(ii).scientaEkinStep_eV = NaN; TmpFileList{3}(ii).scientaEkin_eV = NaN; TmpFileList{3}(ii).beamlineInt = cell(1); TmpFileList{3}(ii).scientaLensMode = ' '; TmpFileList{3}(ii).scientaRegionIterations = NaN; TmpFileList{3}(ii).scientaSequenceIterations = NaN; TmpFileList{3}(ii).imageRoh = {NaN}; TmpFileList{3}(ii).spectrum_countsPerSecondRoh = NaN; TmpFileList{3}(ii).importDate = ' '; TmpFileList{3}(ii).analysisDir = ' '; end %% Merge lists into one % % get index of non-empty structures tf =~cellfun(@isempty,TmpFileList); % merge the non-empty list structures into one structure ListFiles = cat(1,TmpFileList{tf}); % remove lines with empty images for ii = 1:size(ListFiles,1) tmpLogic(ii) = isempty(ListFiles(ii).image); noncells = tmpLogic; end ListFiles(noncells) = []; clearvars -except ListFiles server dataPath* p %% %% Sort by date % First, we need to convert into a table, because structures can not be sorted: ListFiles = struct2table(ListFiles,'AsArray',true); % Then, we sort the table based on the time (6th column, datenum) ListFiles = sortrows(ListFiles,6); % Then, remove un-needed columns ListFiles = removevars(ListFiles,{'bytes','isdir'}); disp('*************************************************') disp('* Voila: Done creating data list! *') disp('*************************************************') %% Import LabView data % if matches(p.Results.importLabView,'on') if isfolder(dataPath_LabView) % verify that this folder exists. try % Get list of all files in that folder TmpFileList = dir([dataPath_LabView filesep '**' filesep '20*.dat']); % import LabView data LabView = LabView_importData(fullfile(TmpFileList(1).folder,TmpFileList(1).name)); % Overwrite LabView data in workspace, if this is the first import if size(TmpFileList,1) > 1 % for additional data files, add these to the LabView data structure. for ii = 2:size(TmpFileList,1) LabView = LabView_importData(fullfile(TmpFileList(ii).folder,TmpFileList(ii).name),LabView); end end catch warning('Problem with LabView file') disp('Problem reading LabView file') end end disp('*************************************************') disp('* Voila: Done reading LabView data! *') disp('*************************************************') clearvars TmpFileList try % combine data and/or rename LabView = LabView_combine(LabView); catch warning('Problem with LabView file') disp('Problem merging LabView files') disp('Check number of columns for each file') end %% Extent list with LabView data %Query time of each data file and get LabView log for that time. for ii = 1:size(ListFiles,1) try % The time of the data and the logging of LabView are not % necessarily occurring at precisely the same time. So, here define % a time range (50 seconds) and get all % data between these times. tse1 = tsdata.event('Start',datenum(ListFiles.datenum(ii)),'Datenum'); tse2 = tsdata.event('End',datenum(ListFiles.datenum(ii)+seconds(50)),'Datenum'); tmp = gettsbetweenevents(LabView.merge,tse1,tse2); if ~isempty(tmp.Data) % only do this if LabView data are not missing (because data logging was off). % Cryo-holder set T (Sample holder temperature is in field ts.Data:1) ListFiles.sampleTemp_dC(ii) = tmp.Data(1,1); % in °C (only get data in first row of tmp) % Cell pressure % (TODO: Pick correct Baratron read out: either field: ts.Data 4,8,and 9) ListFiles.cellPressure_mbar(ii) = tmp.Data(1,4); % in mbar % vapour pressure over ice: if ListFiles.cellPressure_mbar(ii) >= 0 % Marti, J., and K. Mauersberger, Geophys. Res.Lett., 20, 363?366, 1993. % gives parameterisation from 169 - 273K in Pa. (log10(p)= A1/T+B) A_1 = -2663.5; B_1 = 12.537; ListFiles.iceTemp_dC(ii) = A_1/(log10(ListFiles.cellPressure_mbar(ii)*100)-B_1)-273.15; else ListFiles.iceTemp_dC(ii) = NaN; end if datenum(ListFiles.datenum(ii)) >= datenum('04-Mar-2019 13:30:00') % import additional data, logged after 03/2019 ListFiles.smplX_mm(ii) = tmp.Data(1,10); ListFiles.smplY_mm(ii) = tmp.Data(1,11); ListFiles.smplZ_mm(ii) = tmp.Data(1,12); ListFiles.sealingTemp(ii) = tmp.Data(1,13); else % if these sub-set of data is not logged, set to NaN ListFiles.smplX_mm(ii) = NaN; ListFiles.smplY_mm(ii) = NaN; ListFiles.smplZ_mm(ii) = NaN; ListFiles.sealingTemp(ii) = NaN; end else % if there are no LabView data stored, set all entries to NaN ListFiles.sampleTemp_dC(ii) = NaN; ListFiles.iceTemp_dC(ii) = NaN; ListFiles.cellPressure_mbar(ii) = NaN; ListFiles.smplX_mm(ii) = NaN; ListFiles.smplY_mm(ii) = NaN; ListFiles.smplZ_mm(ii) = NaN; ListFiles.sealingTemp(ii) = NaN; end clearvars tmp catch warning('Problem merging LabView data into data table') disp('Problem merging LabView data into data table') %fprintf('Number %.0f: %s \n',ii,[ListFiles.folder(ii) filesep ListFiles.name(ii)]) %fprintf('Number %.0f: %s \n',ii,datestr(ListFiles.datenum(ii))) end end disp('*************************************************') disp('* Voila: Done writing LabView data! *') disp('*************************************************') end %% Make table nicer % if ~iscell(ListFiles.spectrum_countsPerSecond) ListFiles.spectrum_countsPerSecond = cell(size(ListFiles,1),1); end if ~iscell(ListFiles.spectrum_counts) ListFiles.spectrum_counts = cell(size(ListFiles,1),1); end if ~iscell(ListFiles.xRayEkin_eV) ListFiles.xRayEkin_eV = cell(size(ListFiles,1),1); end if ~iscell(ListFiles.scientaEkin_eV) for ii = 1:size(ListFiles,1) tmp{ii,:} = ListFiles.scientaEkin_eV(ii,:); end ListFiles.scientaEkin_eV = tmp; end % rename date fields and delete those fields not needed [ListFiles.lastModifiedDatestr] = ListFiles.date; [ListFiles.lastModifiedDatenum] = ListFiles.datenum; ListFiles = removevars(ListFiles, {'date','datenum'}); % Copy Ekin Data existColumn = strcmp('scientaEkinRoh_eV',ListFiles.Properties.VariableNames); if any(existColumn) ListFiles = removevars(ListFiles,'scientaEkinRoh_eV'); end existColumn = strcmp('scientaEkinRoh_eV',ListFiles.Properties.VariableNames); if ~any(existColumn) % save current ekin data before they are modified ListFiles.scientaEkinRoh_eV= ListFiles.scientaEkin_eV; end if ~iscell(ListFiles.scientaEkinRoh_eV) for ii = 1:size(ListFiles,1) tmp{ii,:} = ListFiles.scientaEkinRoh_eV(ii,:); end ListFiles.scientaEkinRoh_eV = tmp; end % work around for table variables with fitting results having the wrong % size: % first delete the old entries existColumn = strcmp('peakPosition',ListFiles.Properties.VariableNames); if any(existColumn) ListFiles = removevars(ListFiles,'peakPosition'); end existColumn = strcmp('peakArea',ListFiles.Properties.VariableNames); if any(existColumn) ListFiles = removevars(ListFiles,'peakArea'); end existColumn = strcmp('peakFWHM',ListFiles.Properties.VariableNames); if any(existColumn) ListFiles = removevars(ListFiles,'peakFWHM'); end % then re-create them with right size % 11/22 change to 5 entries from 4. existColumn = strcmp('peakPosition',ListFiles.Properties.VariableNames); if ~any(existColumn) % save current ekin data before they are modified ListFiles.peakPosition= NaN(size(ListFiles,1),5); end existColumn = strcmp('peakArea',ListFiles.Properties.VariableNames); if ~any(existColumn) % save current ekin data before they are modified ListFiles.peakArea= NaN(size(ListFiles,1),5); end existColumn = strcmp('peakFWHM',ListFiles.Properties.VariableNames); if ~any(existColumn) % save current ekin data before they are modified ListFiles.peakFWHM= NaN(size(ListFiles,1),5); end existColumn = strcmp('sample',ListFiles.Properties.VariableNames); if ~any(existColumn) ListFiles.sample = cellstr(repmat(' ',size(ListFiles,1),1)); end % add the imported data to previous list (if passed to function upon call) if istable(p.Results.DataTable) % do this only if previous list was input to function call % find table variables that are missing in ListFiles existColumn = strcmp('sample',p.Results.DataTable.Properties.VariableNames); if ~any(existColumn) p.Results.DataTable.sample = cellstr(repmat(' ',size(p.Results.DataTable,1),1)); end tmpMissing = ~ismember(p.Results.DataTable.Properties.VariableNames,ListFiles.Properties.VariableNames); % find all table variables that are cells tmpCell = cellfun(@(x) iscell(p.Results.DataTable.(x)), p.Results.DataTable.Properties.VariableNames); % find all table variables that are numbers tmpNumber = cellfun(@(x) isnumeric(p.Results.DataTable.(x)), p.Results.DataTable.Properties.VariableNames); % table variabes that hold cells and are missing tmpCellVariables = p.Results.DataTable.Properties.VariableNames(tmpMissing & tmpCell); % table variables that hold numbers and are missing tmpNumberVariables = p.Results.DataTable.Properties.VariableNames(tmpMissing & tmpNumber); % add empty table variables for missing cell and for number variables if ~isempty(tmpCellVariables) for i = 1: size(p.Results.DataTable(:,tmpMissing & tmpCell),2) ListFiles = addvars(ListFiles,cell(size(ListFiles,1),1),'NewVariableNames',tmpCellVariables{i}); end end if ~isempty(tmpNumberVariables) for i = 1: size(p.Results.DataTable(:,tmpMissing & tmpNumber),2) ListFiles = addvars(ListFiles,nan(size(ListFiles,1),1),'NewVariableNames',tmpNumberVariables{i}); end end % % % sort the tables, if they have not the same order % tmpCompareTables = cellfun(@isequal, ListFiles.Properties.VariableNames', p.Results.DataTable.Properties.VariableNames'); % % if ~all(tmpCompareTables) % indexDifferenzes = find(ismember(p.Results.DataTable.Properties.VariableNames',ListFiles.Properties.VariableNames(~tmpCompareTables)','rows')); % nameListFiles =ListFiles.Properties.VariableNames(~tmpCompareTables); % % for ii = 1:size(indexDifferenzes,1) % tmpIndx = find(ismember(p.Results.DataTable.Properties.VariableNames',nameListFiles{ii},'rows')); % tmpPreviousVarName = p.Results.DataTable.Properties.VariableNames{tmpIndx-1}; % ListFiles = movevars(ListFiles,nameListFiles{ii},'After',tmpPreviousVarName); % fprintf("%s comes after %s \n", nameListFiles{ii},tmpPreviousVarName) % end % % end % merge tables (append imported data to previous table) ListFiles = [p.Results.DataTable; ListFiles]; % Then, we sort the table based on the time (datenum) ListFiles = sortrows(ListFiles,'lastModifiedDatenum'); end %% Export ExportList = removevars(ListFiles, {'image', 'spectrum_counts', 'spectrum_countsPerSecond', 'scientaEkin_eV', 'beamlineInt', 'imageRoh', 'scientaEkinRoh_eV', 'xRayEkin_eV'}); % TODO check windows formatting try % write a ;-separated list for export to excel. writetable(ExportList,fullfile(dataPath,[datestr(ExportList.lastModifiedDatenum(1),'mmmYYYY') '_DataList.csv']),'Delimiter',','); if ispc % and open in Excel; works only on Windows Machines winopen(fullfile(dataPath,[datestr(ExportList.lastModifiedDatenum(1),'mmmYYYY') '_DataList.csv'])); end catch warning('Problem writing Excel File (*.csv). Make sure it is not open in Excel') end disp('*************************************************') disp('* Voila: nappFileList.m finished ! *') disp('*************************************************') end function LabView = LabView_importData(fullFileName,varargin) % % LabView_importData reads data log ASCII files stored by LabView into % a Matlab time series. % % LabView = LabView_importData(A) imports file A % and stores its content as time series in a structure called "LabView". % % LabView = LabView_importData(A,LabView) imports file A % and appends its content as time series to the LabView structure. % % % Example % LabView = LabView_importData('exmpl.dat',LabView) % % Input % * Napp_chamber_file: Complete and absolute path to the Napp chamber log file. % * LabView: name of output structure. Always call this'LabView' % % Output % * LabView: time series, the name starting with "ts" % followed by date and time the recording started. % % See also LABVIEW_COMBINE, LABVIEW_PLOT, LABVIEW_CUTDATA % %% Data Structure % % Data structure before 4.3.2019 % col 1: Date % col 2: Time % col 3: Reservoir "Pressure 1" - Unit "1 water" - mbar -> ts.Data:2 % col 4: Reservoir "Pressure 2" - Unit "2 other" - mbar -> ts.Data:3 % col 5: Baratron 1 - cell - 1mbar max - mbar -> ts.Data:4 % col 6: Baratron 2 - cell - 10mbar max - mbar -> ts.Data:8 % col 7: Baratron 3 - cell - 0.05mbar max - mbar -> ts.Data:9 % col 8: Baratron 4 - - mbar % col 9: Temp.Ice Sample - °C -> ts.Data:1 % col 10: Temp.Heated Sample % col 11: none % col 12: Temp. Reservoir "Cooler 1" - °C -> ts.Data:5 % col 13: Temp. Reservoir "Cooler 2" - °C -> ts.Data:6 % col 14: Flow He(cryoholder) - l/min -> ts.Data:7 % col 15: none % col 16: none % % Data structure after 4.3.2019 % col 1: Date % col 2: Time % col 3: Reservoir "Pressure 1" - Unit "1 water" - mbar -> ts.Data:2 % col 4: Reservoir "Pressure 2" - Unit "2 other" - mbar -> ts.Data:3 % col 5: Baratron 1 - cell - 1mbar max - mbar -> ts.Data:4 % col 6: Baratron 2 - cell - 10mbar max - mbar -> ts.Data:8 % col 7: Baratron 3 - cell - 0.05mbar max - mbar -> ts.Data:9 % col 8: Baratron 4 - - mbar % col 9: Temp.Ice Sample - °C -> ts.Data:1 % col 10: Temp.Heated Sample % col 11: Temp. Reservoir "Cooler 1" - °C -> ts.Data:5 % col 12: Temp. Reservoir "Cooler 2" - °C -> ts.Data:6 % col 13: Flow He(cryoholder) - l/min -> ts.Data:7 % col 14: Flow gas 1 % col 15: Pressure chamber - mbar % col 16: x-position of cryoholder - mm -> ts.Data:10 % col 17: y-position of cryoholder - mm -> ts.Data:11 % col 18: z-position of cryoholder - mm -> ts.Data:12 % col 19: nothing % col 20: Temp.Sealing - °C -> ts.Data:13 %% % Create an InputParser object p = inputParser; % Add inputs to the scheme. defaultAppendToStructure = 0; addRequired(p,'fullFileName',@isfile); addOptional(p,'appendToStructure',defaultAppendToStructure,@isstruct); % Parse the inputs. parse(p,fullFileName,varargin{:}) %% % get filedate [~,name,~] = fileparts(p.Results.fullFileName); tmp = strsplit(name,'_PC'); filedate = datenum(tmp{1},'yyyy-mm-dd_HH.MM.SS'); clearvars name tmp %% % parse structure to append to if isstruct(p.Results.appendToStructure) LabView = p.Results.appendToStructure; end %% if datenum(filedate) <= datenum('04-Mar-2019 13:30:00') fid = fopen(p.Results.fullFileName); data = textscan(fid, ... '%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f',... 'HeaderLines',1); % data file has 2 text columns, followed by 14 numeric columns fclose(fid); % define time series LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ... = timeseries([data{9} data{3} data{4} data{5} data{12} ... data{13} data{14} data{6} data{7}] , ... datestr(datenum([char(data{1}), char(data{2})],... 'dd.mm.yyyyHH:MM:SS'),0)); LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ... = 'Readout Napp Chamber raw signal'; clear fid ans formatOut nans t x elseif datenum(filedate) >= datenum('15-Sep-2021 01:00:00') && datenum(filedate) <= datenum('01-Apr-2023 01:00:00') % during this time, date and time were in american format. fid = fopen(p.Results.fullFileName); data = textscan(fid, ... '%s %s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',... 'HeaderLines',1); % fclose(fid); % the date formating in the dat file is really messed up. Try to make % it consitent with MATLAB. The problem is that MATLAB does not % understand 1 digit month and hours. % % % Split the date in column 1 into year,month,day newDateSplit = cellfun(@(x) split(char(x),'/'),data{1},'UniformOutput', false); % Split the time in column 2 into hours, minutes, seconds newTimeSplit = cellfun(@(x) split(char(x),':'),data{2},'UniformOutput', false); % combine both with the am/pm string in column 3 to a new datestring customDateStr = cellfun(@(x,y,z) [datestr(datenum([str2double(x{3}),str2double(x{1}),str2double(x{2}),str2double(y{1}),str2double(y{2}),str2double(y{3})])), ' ', char(z)], newDateSplit,newTimeSplit,data{3}, 'uniformOutput',false) ; % define time series LabView.(['ts_' datestr(customDateStr{1},'YYYYmmdd_HHMM')]) ... = timeseries([data{10} data{4} data{5} data{6} data{12} ... data{13} data{14} data{7} data{8} data{17} data{18} data{19} data{21}], ... customDateStr); LabView.(['ts_' datestr(customDateStr{1},'YYYYmmdd_HHMM')]).Name ... = 'Readout Napp Chamber raw signal'; clearvars fid ans formatOut nans t x new* custom* elseif datenum(filedate) >= datenum('01-Apr-2023 02:00:00') fid = fopen(p.Results.fullFileName); data = textscan(fid, ... '%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',... 'HeaderLines',1); % data file has 2 text columns, followed by 18 numeric columns (THE AM % PM IS MISSING and some changes to the LabView file. Confirm with Luca....) fclose(fid); % define time series LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ... = timeseries([data{9} data{3} data{4} data{5} data{11} ... data{12} data{20} data{6} data{7} data{15} data{16} data{17} data{19}] , ... datestr(datenum([char(data{1}), char(data{2})],... 'dd.mm.yyyyHH:MM:SS'),0)); LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ... = 'Readout Napp Chamber raw signal'; clear fid ans formatOut nans t x else fid = fopen(p.Results.fullFileName); data = textscan(fid, ... '%s %s %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f',... 'HeaderLines',1); % data file has 2 text columns, followed by 14 numeric columns fclose(fid); % define time series LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]) ... = timeseries([data{9} data{3} data{4} data{5} data{11} ... data{12} data{13} data{6} data{7} data{16} data{17} data{18} data{20}] , ... datestr(datenum([char(data{1}), char(data{2})],... 'dd.mm.yyyyHH:MM:SS'),0)); LabView.(['ts_' datestr(datenum([char(data{1}{1}), char(data{2}{1})],'dd.mm.yyyyHH:MM:SS'),'YYYYmmdd_HHMM')]).Name ... = 'Readout Napp Chamber raw signal'; clear fid ans formatOut nans t x end end function data_structure = LabView_combine(data_structure) % LabView_combine can be used to merge time series into one. % % LabView = LabView_combine(LabView) merges all timeseries in the structure % LabView into the new timeseries "merge" % and adds this timeseries to the LabView structure % % % Input % * data_structure: name of the structure holding the time series % % Output % * data_structure.merge: New time series combining all ts in this structure % % See also LABVIEW_IMPORTDATA, LABVIEW_PLOT % delete previous merged data if isfield(data_structure,'merge') data_structure = rmfield(data_structure, 'merge'); end % sort them chronologically data_structure = orderfields(data_structure); % query list of time series list_ts = fieldnames(data_structure); % and merge all of them into one time series for ii = 1:length(list_ts) if ii == 1 data_structure.merge = data_structure.(list_ts{1}); else data_structure.merge = append(data_structure.merge,data_structure.(list_ts{ii})); end end end