# PEARL Procedures: pearl-pshell-import.ipf File Reference

pearl-pshell-import.ipf File Reference
ARPES package

import data from PShell More...

`#include <HDF5 Browser>`
`#include "pearl-compat"`
`#include "pearl-gui-tools"`
`#include "pearl-area-import"`

Go to the source code of this file.

## Namespaces

| PearlPShellImport
| import data from PShell

## Functions

dfr | psh5_load (string path_name, string file_name, string scans, string regions, string datasets, variable classes=defaultValue, variable max_rank=defaultValue, string reduction_func=defaultValue, string reduction_params=defaultValue, dfref dest_df=defaultValue)
| main data loading function More...
dfr | psh5_preview (string path_name, string file_name, dfref dest_df=defaultValue, string preview_datasets=defaultValue)
| load preview More...
dfr | psh5_open_file (string path_name, string file_name, dfref dest_df=defaultValue)
| open a HDF5 file created by the PShell data acquisition program and prepare the data folder. More...
variable | psh5_close_file (dfref file_df)
| close a HDF5 file opened by psh5_open_file. More...
static string | twave2list (wave wt, string sep)
| convert text wave to list. More...
static string | wave2list (wave w, string format, string sep)
| convert numeric wave to list. More...
string | psh5_list_scans (variable file_id)
| list scan groups of a PShell data file. More...
string | psh5_list_all_datasets (variable file_id)
| list all datasets in a file More...
variable[string datatypes, string ranks, string dimensions] | psh5_list_dataset_info (string variable, string file_id, variable string, sds datasets)
| list data types and dimensions of datasets More...
string | psh5_filter_datasets_rank (string datasets, string ranks, variable min_rank, variable max_rank)
| filter datasets by rank More...
static string | unique_strings (string list)
| remove duplicate items from list More...
string | psh5_extract_scan_paths (string datasets)
| trim dataset paths to the scan part More...
string | psh5_extract_region_paths (string datasets)
| trim dataset paths to the scan/region part More...
string | psh5_match_dataset_classes (string datasets, variable classes, string positioners=defaultValue, string detectors=defaultValue)
| filter a list of datasets by classification More...
dfr | psh5_create_folders (string datasetpath)
| create all data folders along a dataset path More...
dfr | psh5_dataset_to_folder (dfref parent_df, string datasetpath)
| map dataset path to datafolder path More...
string | ps_fix_folder_name (string group_name)
| convert HDF5 group name to data folder name and fix compatibility issues More...
string | psh5_load_datasets (dfref file_df, string datasets, variable create_folders=defaultValue, string reduction_func=defaultValue, string reduction_params=defaultValue)
| load multiple datasets from open file More...
string | psh5_load_dataset (dfref file_df, string datasetpath, variable create_folders=defaultValue, string reduction_func=defaultValue, string reduction_params=defaultValue)
| load a dataset from an open PShell HDF5 file. More...
string | psh5_load_dataset_slabs (dfref file_df, string datasetpath, variable create_folders=defaultValue, variable progress=defaultValue)
| load a dataset slab-wise from the open PShell HDF5 file. More...
string | psh5_load_dataset_reduced (dfref file_df, string datasetpath, funcref reduction_func, string reduction_params, variable create_folders=defaultValue, variable progress=defaultValue, variable nthreads=defaultValue)
| load a dataset with reduced dimensionality More...
static threadsafe variable | reduce_slab_worker (funcref reduction_func)
static threadsafe wave | reduce_slab_image (wave slabdata, wave image, funcref reduction_func, string reduction_params)
string | psh5_load_general_group (dfref file_df)
| load organizational metadata from the general group. More...
string | psh_load_general_string (dfref file_df, string name)
| load a string from the general group. More...
variable | psh5_load_dataset_meta (dfref file_df, string datasetpath, wave datawave)
| load metadata of a PShell dataset. More...
string | psh5_load_scan_meta (dfref file_df, string scanpath)
| load metadata of a PShell scan group. More...
variable | ps_set_dimlabels (wave data)
| set dimension labels according to the axis type More...
variable | ps_set_dimlabels2 (wave data, string name)
| set dimension labels according to the axis type More...
dfr | ps_find_scan_folder (dfref data_df)
| find the scan folder of current data More...
dfr | ps_find_attr_folder (dfref scan_df)
| find the attributes data folder More...
wave | ps_find_scale_wave (string name, dfref df1, dfref df2, dfref df3)
| find a wave in scan and attr data folders More...
variable | ps_detect_scale (dfref data_df, wave ax, wave lo, wave hi, wave un)
| detect the dimension scales from attributes. More...
variable | ps_scale_dataset_2 (wave data, wave ax, wave lo, wave hi, wave un)
| set the dimension scales of a dataset. More...
variable | ps_scale_datasets (dfref scan_df)
| set the dimension scales of loaded PShell Scienta datasets according to attributes. More...
variable | ps_scale_dataset (wave data)
| set the dimension scales of a loaded PShell Scienta dataset according to attributes. More...
string | kill_matching_waves (dfref dfr, string pattern, variable recurse, string killed=defaultValue)
| kill any waves matching a pattern in the experiment More...

## Variables

const string | kEnergyDimLabel = "energy"
| Dimension label for the energy dispersive dimension of multi-dimensional datasets. More...
const string | kAngleDimLabel = "angle"
| Dimension label for the angle dispersive dimension of multi-dimensional datasets. More...
const string | kScanDimLabel = "scan"
| Dimension label for the scan dimension of multi-dimensional datasets. More...
const string | kDataDimLabel = "data"
| Dimension label for the data dimension. More...
const string | kPreviewDatasets = "ImageEnergyDistribution;ScientaSpectrum;ScientaImage;Counts;SampleCurrent;"
| List of preferred datasets to load for preview. More...
const string | kScientaScalingDatasets = "LensMode;ScientaChannelBegin;ScientaChannelEnd;ScientaSliceBegin;ScientaSliceEnd;Eph;"
| List of datasets that must be loaded to determine the axis scaling of a Scienta image. More...
const string | kEssentialDiagnostics = "ManipulatorX;ManipulatorY;ManipulatorZ;ManipulatorTheta;ManipulatorTilt;ManipulatorPhi;MonoEnergy;"
| List of diagnostic datasets that are normally loaded with a scan. More...
const string | kTransposedDatasets = "ScientaImage;"
| List of datasets that must be transposed upon loading. More...
const variable | kDSCPositioners = 0x0001
const variable | kDSCDetectors = 0x0002
const variable | kDSCScientaScaling = 0x0004
const variable | kDSCPreview = 0x0008
const variable | kDSCEssentialDiags = 0x0010
const variable | kDSCAttrs = 0x0020
const variable | kDSCDiags = 0x0040
const variable | kDSCSnaps = 0x0080
const variable | kDSCMeta = 0x0100
const variable | kDSCMonitors = 0x0200
const variable | kDSCRegions = 0x0400
const variable | kDSCOther = 0x8000
const variable | kDSCAll = 0xffff

## Detailed Description

import data from PShell

HDF5 file import from the PShell data acquisition program.

the module provides two main entry functions:

- psh5_load() for almost all data loading tasks including data reduction.
- psh5_preview() to load a simple 1d or 2d preview of the first and most relevant dataset in the file.

Version
  up to igor 8, this module requires the HDF5 XOP which must be enabled manually. as of igor 9 and later, HDF5 is built in.
  in version 2.0, the interface has changed significantly.

Author
  matthias muntwiler, matth.nosp@m.ias..nosp@m.muntw.nosp@m.iler.nosp@m.@psi..nosp@m.ch

Copyright
  2013-22 Paul Scherrer Institut
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Definition in file pearl-pshell-import.ipf.

## Function Documentation

## kill_matching_waves()

string kill_matching_waves | ( | dfref | dfr,
|  | string | pattern,
|  | variable | recurse,
|  | string | killed = `defaultValue`
| ) |  |

kill any waves matching a pattern in the experiment

this may be used to kill big waves of original data before saving.

example: to kill all ScientaImage waves:

kill_matching_waves($"root:", "ScientaImage", 1)

Definition at line 2554 of file pearl-pshell-import.ipf.

## ps_detect_scale()

variable ps_detect_scale | ( | dfref | data_df,
|  | wave | ax,
|  | wave | lo,
|  | wave | hi,
|  | wave | un
| ) |  |

detect the dimension scales from attributes.

the function checks the data , scan and attributes folders for scan parameters. the results are written to the provided waves. the function is normally called by ps_scale_datasets() but can also be used independently.

the data folder contains the waves that are to be scaled. the function looks for the scan positions and diagnostics as necessary. if the scaling data is not found, the scales are not changed. the kEssentialDiags flag can be used with psh5_load() to select the necessary datasets.

the provided waves are redimensioned by the function, and dimension labels are set. the scale parameters can then be extracted by keyword, e.g.,

- `lo[%energy]` analyser energy dimension.
- `lo[%angle]` analyser angle dimension.
- `lo[%scan]` scan dimension.
- `lo[%data]` data dimension.

the function tries to read the following waves, in the data, scan, and attributes/diagnostics folders, where the first folder in the list takes precedence. it may fall back to more or less reasonable default values if no data is not found.

- `LensMode`
- `ScientaChannelBegin`
- `ScientaChannelEnd`
- `ScientaSliceBegin`
- `ScientaSliceEnd`
- `ScanWritables`
- wave referenced by `ScanWritables[0]`

Parameters

  data_df | data folder which contains the waves to be scaled. this is usually the "scan" or "region" folder.
  ax | text wave to receive the axis labels.
  lo | wave to receive the lower limits.
  hi | wave to receive the upper limits.
  un | text wave to receive the unit labels.

Returns
  the function results are written to the lo, hi, un, and ax waves.

Definition at line 2225 of file pearl-pshell-import.ipf.

## ps_find_attr_folder()

dfr ps_find_attr_folder | ( | dfref | scan_df | ) |

find the attributes data folder

the attributes folder contains diagnostic beamline data at each scan point. the folder can have one of several names due to different pshell versions: "attr", "attrs", or "diags" (from 2022 on). historically, the folder was named "attr" due to the area detector software.

assuming we are in the scan folder (where the ScanWritables, etc.) are, find the associated attributes folder.

Definition at line 2134 of file pearl-pshell-import.ipf.

## ps_find_scale_wave()

wave ps_find_scale_wave | ( | string | name,
|  | dfref | df1,
|  | dfref | df2,
|  | dfref | df3
| ) |  |

find a wave in scan and attr data folders

look up a wave by name in the given three data folders. return the first one found.

Parameters

  df1 | first data folder to check
  df2 | second data folder to check
  df3 | third data folder to check

Returns
  wave reference, empty reference if not found

Definition at line 2161 of file pearl-pshell-import.ipf.

## ps_find_scan_folder()

dfr ps_find_scan_folder | ( | dfref | data_df | ) |

find the scan folder of current data

assuming we are in the data folder (where the scan results, ScientaSpectrum, etc.) are, find the associated scan folder. this can either be the same (usually) or the parent folder (multi-region scans).

the scan folder is the one that contains the ScanWritables wave. the data and scan folders may refer to the same folder.

Definition at line 2111 of file pearl-pshell-import.ipf.

## ps_fix_folder_name()

string ps_fix_folder_name | ( | string | group_name | ) |

convert HDF5 group name to data folder name and fix compatibility issues

Definition at line 1065 of file pearl-pshell-import.ipf.

## ps_scale_dataset()

variable ps_scale_dataset | ( | wave | data | ) |

set the dimension scales of a loaded PShell Scienta dataset according to attributes.

the current datafolder must contain the :attr folder. the data wave can be in the current folder or a sub-folder.

the dimension labels of the dataset waves must have been set correctly, e.g. by ps_set_dimlabels(). this is implicitly done by the high-level load functions.

the function is useful if a single dataset is loaded and scaled. if multiple datasets are loaded, ps_scale_datasets() is slightly more efficient.

Parameters

  data | data wave to be scaled. dimension labels (index -1) must be set correctly, cf. ps_set_dimlabels().

Version
  this function supports regions from version 1.03.

Definition at line 2528 of file pearl-pshell-import.ipf.

## ps_scale_dataset_2()

variable ps_scale_dataset_2 | ( | wave | data,
|  | wave | ax,
|  | wave | lo,
|  | wave | hi,
|  | wave | un
| ) |  |

set the dimension scales of a dataset.

the function is normally called by ps_scale_datasets() but can also be used independently. the limits and units must be given as function arguments with proper dimension labels.

the provided limit and unit waves must have dimension labels matching the -1 index dimension labels of the data wave, such as set by the ps_detect_scale() function. the scale parameters are extracted by keyword, e.g.,

- `lo[%energy]` analyser energy dimension.
- `lo[%angle]` analyser angle dimension.
- `lo[%scan]` scan dimension.
- `lo[%data]` data dimension.

if the data dimension labels and units are at their defaults ("value" and "arb.", respectively), the function tries to read them from the existing wave note ("AxisLabelD" and "AxisUnitD"), or based on the wave name if the name is one of the known measurement variables: "ScientaImage", "ImageAngleDistribution", "ScientaAngleDistribution", "ScientaSpectrum", "ImageEnergyDistribution", "ScientaEnergyDistribution", "SampleCurrent", "RefCurrent", "AuxCurrent", "MachineCurrent".

Parameters

  data | data wave to be scaled. dimension labels (index -1) must be set to match the limit waves.
  ax | axis labels. the axis labels are written to the wave note in the format `AxisLabel%s=%s` where `X`, `Y`, `Z`, `D` is substituted for the first place holder and the label for the second one.
  lo | lower limits. the lower limits are applied using the SetScale operation.
  hi | upper limits. the upper limits are applied using the SetScale operation.
  un | unit labels. the unit labels are applied using the SetScale operation.

Version
  this function supports regions from version 1.03.

Definition at line 2389 of file pearl-pshell-import.ipf.

## ps_scale_datasets()

variable ps_scale_datasets | ( | dfref | scan_df | ) |

set the dimension scales of loaded PShell Scienta datasets according to attributes.

datasets listed in the ScanReadables waves are scaled according to the attribute waves in the data, scan, and attributes folders, whichever is found first.

the specified datafolder must contain the ScanReadables wave and the :attr folder. the ScanReadables text wave contains names of the waves to scale. wave names can include a relative path to a sub-folder. the path separator is "/".

the dimension labels of the dataset waves must have been set correctly, e.g. by ps_set_dimlabels(). this is implicitly done by the high-level load functions.

Parameters

  scan_df | scan data folder. must contain the ScanReadables wave.

Definition at line 2483 of file pearl-pshell-import.ipf.

## ps_set_dimlabels()

variable ps_set_dimlabels | ( | wave | data | ) |

set dimension labels according to the axis type

this function asserts a particular ordering of dimensions types based on the name of the wave for ScientaImage, ScientaSpectrum, ImageAngleDistribution, ImageEnergyDistribution. all other waves must be one-dimensional, and the dimension must be the scan dimension.

dimension labels are required by scaling functions.

Parameters

  data | data wave as loaded from PShell file

Returns
  - 0 all labels set successfully.
  - 1 unidentified data source.
  - 2 wave does not contain data.

Definition at line 2031 of file pearl-pshell-import.ipf.

## ps_set_dimlabels2()

variable ps_set_dimlabels2 | ( | wave | data,
|  | string | name
| ) |  |

set dimension labels according to the axis type

same as ps_set_dimlabels() except that the dimension labels are set according to a separate name argument instead of the wave name.

Parameters

  data | data wave as loaded from PShell file.
  name | original name of the dataset in the PShell file.

Returns
  - 0 all labels set successfully.
  - 1 unidentified data source.
  - 2 wave does not contain data.

Definition at line 2050 of file pearl-pshell-import.ipf.

## psh5_close_file()

variable psh5_close_file | ( | dfref | file_df | ) |

close a HDF5 file opened by psh5_open_file.

this function just closes the HDF5 file. no change is made to the loaded data.

the function requires the specified data folder to contain a variable named `file_id` that specifies the HDF5 file ID. the variable may also be in a parent folder. the variable is killed after the file has been closed. if the folder or variable can't be found, the function does nothing.

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). the reference may also point to a child folder, the function will look for a file_id variable in all parent folders.

Note
  on the command line, data folder references can be specified using the $-notation like `$"foldername"`. the current folder is written as `$":"`.

Definition at line 509 of file pearl-pshell-import.ipf.

## psh5_create_folders()

dfr psh5_create_folders | ( | string | datasetpath | ) |

create all data folders along a dataset path

if the path ends with a slash, the path is interpreted as a group path, and each part is mapped to a data folder. else, the last part of the path is the name of a dataset and will not produce a folder.

the path will always be interpreted as starting from the root, regardless whether it starts with a slash or not.

spaces are removed from folder names, and the names are cleaned up to produce simple names.

a string variable named "s_hdf5_group" is added to each created folder and contains the incremental path.

the first child folder is created in the current data folder. at the end, the lowest child folder is selected and returned as the function result.

Parameters

  datasetpath | hdf5 group path to dataset, e.g. "/scan 1/region 1/ScientaImage".

Returns
  data folder reference of the lowest child folder.

Definition at line 990 of file pearl-pshell-import.ipf.

## psh5_dataset_to_folder()

dfr psh5_dataset_to_folder | ( | dfref | parent_df,
|  | string | datasetpath
| ) |  |

map dataset path to datafolder path

if the path ends with a slash, the path is interpreted as a group path, and each part maps to a data folder. if the last part of the path is the name of a dataset, it is discarded.

spaces are removed from folder names, and the names are cleaned up to produce simple names.

the path is interpreted as relative to the specified parent data folder. regardless whether it starts with a slash or not.

Parameters

  datasetpath | hdf5 group path to dataset, e.g. "/scan 1/region 1/ScientaImage".
  parent_df | parent data folder

Returns
  data folder reference

Definition at line 1034 of file pearl-pshell-import.ipf.

## psh5_extract_region_paths()

string psh5_extract_region_paths | ( | string | datasets | ) |

trim dataset paths to the scan/region part

return dataset paths stripped to the form /scan*‍/region*‍/.

the function matches each path for scan and region tokens in the first two path elements and strips off the remainder. if there are no region-based datasets, the function returns an empty string.

the function operates on a single path or a semicolon-separated list of paths. the items of the returned list are unique.

Parameters

  datasets | semicolon separated list of dataset paths

Returns
  list of scan/region paths (no duplicates)

Definition at line 853 of file pearl-pshell-import.ipf.

## psh5_extract_scan_paths()

string psh5_extract_scan_paths | ( | string | datasets | ) |

trim dataset paths to the scan part

return dataset paths stripped to the form /scan*‍/.

the function matches each path for a scan token in the first path element and strips off the remaining path. if there are no scan-based datasets, the function returns an empty string.

the function operates on a single path or a semicolon-separated list of paths. the items of the returned list are unique.

Parameters

  datasets | semicolon separated list of dataset paths

Returns
  list of scan paths (no duplicates)

Definition at line 803 of file pearl-pshell-import.ipf.

## psh5_filter_datasets_rank()

string psh5_filter_datasets_rank | ( | string | datasets,
|  | string | ranks,
|  | variable | min_rank,
|  | variable | max_rank
| ) |  |

filter datasets by rank

Parameters

  datasets | semicolon-separated list of datasets to be checked.
  ranks | semicolon-separated list of ranks of each dataset.

Returns
  filtered dataset list.

Definition at line 728 of file pearl-pshell-import.ipf.

## psh5_list_all_datasets()

string psh5_list_all_datasets | ( | variable | file_id | ) |

list all datasets in a file

the function returns a list of all datasets in a file. each dataset is listed by its full path like, e.g., "/scan 1/region 1/dataset 1".

this function wraps a one-line HDF5 operation and is provided just to be more memorable.

Parameters

  file_id | ID of open HDF5 file from psh5_open_file().

Returns
  semicolon-separated list of absolute dataset paths.

Definition at line 605 of file pearl-pshell-import.ipf.

## psh5_list_dataset_info()

variable [string datatypes, string ranks, string dimensions] psh5_list_dataset_info | ( | string | variable,
|  | string | file_id,
|  | variable | string,
|  | sds | datasets
| ) |  |

list data types and dimensions of datasets

this function has multiple returns.

Parameters

  file_id | ID of open HDF5 file from psh5_open_file().

Returns
  semicolon-separated list of (simplified) datatypes. datatypes are marked as "i" (integer), "f" (float), "s" (string) or "?" (unknown).
  semicolon-separated list of ranks (number of dimensions).
  semicolon-separated list of dimensions. each item is a comma-separated list of dimension sizes. items do not contain trailing commas.

filter a list of datasets by string matching

this function can be used to extract certain dataset paths from a list of all datasets in a file. the matching is insensitive to spaces and case.

examples match strings:

- `"*&zwj;/scan1/region1/*"` match all datasets in scan 1, region 1
- `"!*&zwj;/diags/*"` remove diagnostics from list

Parameters

  datasets | semicolon separated list of dataset paths
  match | match string for igor's StringMatch function

Returns
  list of matching datasets

Definition at line 631 of file pearl-pshell-import.ipf.

## psh5_list_scans()

string psh5_list_scans | ( | variable | file_id | ) |

list scan groups of a PShell data file.

the function returns a list of all top-level groups whose name starts with "scan".

Parameters

  file_id | ID of open HDF5 file from psh5_open_file().

Returns
  semicolon-separated list of group paths.

Definition at line 574 of file pearl-pshell-import.ipf.

## psh5_load()

dfr psh5_load | ( | string | path_name,
|  | string | file_name,
|  | string | scans,
|  | string | regions,
|  | string | datasets,
|  | variable | classes = `defaultValue`,
|  | variable | max_rank = `defaultValue`,
|  | string | reduction_func = `defaultValue`,
|  | string | reduction_params = `defaultValue`,
|  | dfref | dest_df = `defaultValue`
| ) |  |

main data loading function

load the requested elements from the given file.

scans, regions and datasets are additive. wildcards can be used to select multiple or all datasets.

classes are subtractive: only datasets of specified classes are loaded. by default, only positioners, detectors, scaling and essential diagnostics are loaded.

essential diags, scaling, positioners related to requested detectors are always loaded

data reduction (if specified) applies to 3d data, see psh5_load_dataset_reduced() for details.

Parameters

  path_name | igor symbolic path name. can be empty if the path is specified in file_name or a dialog box should be displayed
  file_name | if empty a dialog box shows up
  dest_df | destination folder reference. if dest_df is specified, data is loaded into this folder. else, a new folder derived from the file name is created under root:

  scans

  semicolon-separated list of scan paths to load. scan groups are at the top level, their name consists of "scan", an optional space and a number. all datasets in the group and sub-groups are considered for loading unless excluded by other arguments. if empty, no datasets are loaded based on their relation to a scan. names are matched by Igor's StringMatch function. the matching is insensitive to case and spaces. to load all scans, pass "/scan*". the leading slash before "scan" can be omitted.

  regions

  semicolon-separated list of region paths to load. region groups are children of scan groups, their name consists of "region", an optional space and a number. all datasets in the group and sub-groups are considered for loading unless excluded by other arguments. if empty, no datasets are loaded based on their relation to a region. names are matched by Igor's StringMatch function. the matching is insensitive to case and spaces. to load all regions of scan 1, pass "/scan1/region*". to load regions 1 of all scans, pass "/scan*&zwj;/region1". the leading slash before "scan" can be omitted.

  datasets | semicolon-separated list of dataset paths to load. this allows to load individual datasets. names are matched by Igor's StringMatch function against full dataset paths. to load all datasets named "SampleCurrent", pass "*&zwj;/SampleCurrent". the matching is insensitive to case and spaces. additional datasets may be loaded for scaling.

  classes

  filter datasets (that were selected by the scans, regions and datasets arguments) by class. this allows, for example, to exclude the diagnostics. note that scaling datasets are always loaded. the value is a bit-wise switch, typically the arithmetic disjunction of kDSCXxxx constants. by default, only positioners, detectors, scaling and essential diagnostics are loaded. to completely load all datasets, specify kDSCAll.

  max_rank | load only datasets with lower or equal rank.

Returns
  data folder reference of the file-level data folder. same as dest_df if specified.
  global string s_filepath in new data folder contains the full file path on disk.
  global string s_scanpaths in new data folder contains a list of scan groups inside the file.
  global string s_loaded_datasets in new data folder contains a list of loaded datasets. the items are full group paths of the HDF5 file. dataset paths can be mapped to loaded data folders using the psh5_dataset_to_folder function.

Definition at line 158 of file pearl-pshell-import.ipf.

## psh5_load_dataset()

string psh5_load_dataset | ( | dfref | file_df,
|  | string | datasetpath,
|  | variable | create_folders = `defaultValue`,
|  | string | reduction_func = `defaultValue`,
|  | string | reduction_params = `defaultValue`
| ) |  |

load a dataset from an open PShell HDF5 file.

if the dataset has a maximum of two dimensions, the function loads it at once. if it has more than two dimension, the function calls psh5_load_dataset_slabs() to load the data slab by slab.

- the metadata (HDF5 attributes) are loaded into the wave note, cf. psh5_load_dataset_meta().
- dimension labels are set according the dataset name, cf. ps_set_dimlabels().

the dataset is loaded into the current data folder or a tree based on the group path given in the datasetpath argument. the function returns from the original data folder.

only numeric and string data types are supported, string datasets must have rank 1.

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). if undefined, the current datafolder is assumed.
  datasetpath | group path and name of the dataset, e.g. "/scan 1/ScientaImage". HDF5 groups map to igor data folders below the current data folder, the wave is placed into the leaf folder. the names of groups and waves are cleaned up to produce simple names, in particular, spaces and other illegal characters are removed.
  create_folders | if 1 (default), data folders according to the group path are created. if 0, the dataset is loaded into the current folder.
  reduction_func | data reduction function. three-dimensional datasets can be reduced in dimensionality by on-the-fly data reduction. by default (or if empty string), no reduction is applied. see psh5_load_dataset_reduced().
  reduction_params | parameter string for the reduction function.

Returns
  semicolon-separated list of loaded wave names. multiple waves are loaded if the dataset has a compound data type. in that case the wave name is a concatenation of the dataset and field names (see HDF5LoadData).

Definition at line 1176 of file pearl-pshell-import.ipf.

## psh5_load_dataset_meta()

variable psh5_load_dataset_meta | ( | dfref | file_df,
|  | string | datasetpath,
|  | wave | datawave
| ) |  |

load metadata of a PShell dataset.

metadata are the HDF5 attributes attached to a dataset. they are mapped to "key=value" pairs and added to the wave note in separate lines. the following attributes are loaded. names and mappings are hard-coded.

- "Writable Dimension" -> "ScanDimension"
- "Writable Index" -> "WriteableIndex"
- "Readable Index" -> "ReadableIndex"

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). if undefined, the current datafolder is assumed.
  datasetpath | group path and name of the dataset. path separator is the slash "/".
  datawave | metadata is added to the wave note of this wave.

Returns
  0 if successful, non-zero if an error occurred.

Definition at line 1887 of file pearl-pshell-import.ipf.

## psh5_load_dataset_reduced()

string psh5_load_dataset_reduced | ( | dfref | file_df,
|  | string | datasetpath,
|  | funcref | reduction_func,
|  | string | reduction_params,
|  | variable | create_folders = `defaultValue`,
|  | variable | progress = `defaultValue`,
|  | variable | nthreads = `defaultValue`
| ) |  |

load a dataset with reduced dimensionality

the function loads the dataset image by image using the hyperslab option and applies a custom reduction function like numeric integration, curve fitting, etc. to each image. the results from the reduction function are written to the `ReducedData1`, `ReducedData2`, etc. waves. the raw data are discarded.

example reduction functions can be found in the PearlScientaPreprocess module. they must implement the adh5_default_reduction() interface.

by default, the reduction function is called in separate threads to reduce the total loading time. (psh5_load() reports the total run time in the global variable psh5_perf_secs.) the effect varies depending on the balance between file loading (image size) and data processing (complexity of the reduction function).

the function loads images (as hyperslabs) one by one and passes them to the reduction function. only a limited number of images are held in the queue at a time to limit memory use. for debugging the reduction function, multi-threading can be disabled (also remove threadsafe attributes from reduce_slab_image() and the reduction function!)

if the reduction function requires the image waves to be scaled properly, the attributes must have been loaded by psh5_load_scan_attrs() before. in this case, the scales of the result waves are also set by the function. otherwise, the results can also be scaled by ps_scale_dataset() later.

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). if undefined, the current datafolder is assumed.
  scanpath | path to scan group in the HDF5 file.
  datasetname | name of the dataset. this must currently be "ScientaImage", other data is not supported. the name of the loaded wave is a cleaned up version of the dataset name. the name can include the region name as a relative path, e.g. "region1/ScientaImage". in this case, the dataset is loaded into a sub-folder named "region1".
  reduction_func | custom data reduction function. this can be any user-defined function which has the same parameters as adh5_default_reduction. some reduction functions are predefined in the PearlScientaPreprocess module.
  reduction_params | parameter string for the reduction function.
  create_folders | if 1 (default), data folders according to the group path are created. if 0, the dataset is loaded into the current folder.

  progress

  progress window.
  - 1 (default) show progress window
  - 0 do not show progress window

  nthreads

  - -1 (default) use as many threads as there are processor cores (in addition to main thread).
  - 0 use main thread only (for debugging and profiling).
  - >= 1 use a fixed number of (additional) threads.

Returns
  semicolon-separated list of the loaded dataset `ReducedData1`, `ReducedData2`, etc. if successful. auxiliary waves, scan positions, attributes are loaded but not listed in the string. empty string if an error occurred. error messages are printed to the history.

Definition at line 1472 of file pearl-pshell-import.ipf.

## psh5_load_dataset_slabs()

string psh5_load_dataset_slabs | ( | dfref | file_df,
|  | string | datasetpath,
|  | variable | create_folders = `defaultValue`,
|  | variable | progress = `defaultValue`
| ) |  |

load a dataset slab-wise from the open PShell HDF5 file.

the function loads the dataset image by image using the hyperslab option. the wave is loaded into the current data folder.

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). if undefined, the current datafolder is assumed.
  datasetpath | group path and name of the dataset. the dataset name defines the name of the loaded wave (after cleaning up).

  progress

  select whether a progress window is displayed during the process.
  - 1 (default) show progress window.
  - 0 do not show progress window.

  create_folders | if 1 (default), data folders according to the group path are created. if 0, the dataset is loaded into the current folder.

Returns
  semicolon-separated list of loaded wave names. in the current version, the function returns zero or one wave, as it does not support compound types.

Definition at line 1289 of file pearl-pshell-import.ipf.

## psh5_load_datasets()

string psh5_load_datasets | ( | dfref | file_df,
|  | string | datasets,
|  | variable | create_folders = `defaultValue`,
|  | string | reduction_func = `defaultValue`,
|  | string | reduction_params = `defaultValue`
| ) |  |

load multiple datasets from open file

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). if undefined, the current datafolder is assumed.
  create_folders | if 1 (default), data folders according to the group path are created. if 0, the dataset is loaded into the current folder. the latter option should be used with care because datasets with same names may be overwritten.
  reduction_func | data reduction function. three-dimensional datasets can be reduced in dimensionality by on-the-fly data reduction. by default (or if empty string), no reduction is applied. see psh5_load_dataset_reduced().
  reduction_params | parameter string for the reduction function.

Returns
  (string) semicolon-separated list of loaded datasets

Definition at line 1098 of file pearl-pshell-import.ipf.

## psh5_load_general_group()

string psh5_load_general_group | ( | dfref | file_df | ) |

load organizational metadata from the general group.

the general group contains the following datasets: authors, pgroup, proposal, proposer, sample.

data is loaded into the current data folder. all items are loaded into strings, authors is a comma-separated list. missing items default to empty strings.

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). if undefined, the current datafolder is assumed.

Returns
  semicolon-separated list of the objects.

Definition at line 1803 of file pearl-pshell-import.ipf.

## psh5_load_scan_meta()

string psh5_load_scan_meta | ( | dfref | file_df,
|  | string | scanpath
| ) |  |

load metadata of a PShell scan group.

metadata are the HDF5 attributes attached to the scan group. the following attributes are loaded. the respective wave names under Igor are given in parentheses.

- Dimensions (ScanDimensions)
- Writables (ScanWritables)
- Readables (ScanReadables)
- Steps (ScanSteps)
- Iterations (ScanIterations) - if present (XPSSpectrum script)
- Step Size (ScanStepSize) - if present (XPSSpectrum script)
- Step Time (ScanStepTime) - if present (XPSSpectrum script)

if they are missing in the file, `ScanDimensions` and `ScanReadables` are set to default values assuming the file contains a single spectrum.

data is loaded into the current data folder.

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). if undefined, the current datafolder is assumed.
  scanpath | path to the scan group in the HDF5 file, e.g. "/scan 1".

Returns
  semicolon-separated list of the loaded waves.

Definition at line 1954 of file pearl-pshell-import.ipf.

## psh5_match_dataset_classes()

string psh5_match_dataset_classes | ( | string | datasets,
|  | variable | classes,
|  | string | positioners = `defaultValue`,
|  | string | detectors = `defaultValue`
| ) |  |

filter a list of datasets by classification

Parameters

  datasets | semicolon separated list of dataset paths
  classes | dataset classes. arithmetic OR of the kDSCXxxx constants.

Returns
  list of scan/region paths (no duplicates)

Definition at line 899 of file pearl-pshell-import.ipf.

## psh5_open_file()

dfr psh5_open_file | ( | string | path_name,
|  | string | file_name,
|  | dfref | dest_df = `defaultValue`
| ) |  |

open a HDF5 file created by the PShell data acquisition program and prepare the data folder.

the function opens a specified or interactively selected HDF5 file, and loads general information about the file including a list of contained datasets.

data can be loaded into an existing or new data folder under root.

the file must be closed by psh5_close_file() after use. the HDF5 file ID is stored in the global variable file_id until the file is closed.

Parameters

  path_name | igor symbolic path name. can be empty if the path is specified in FileName or a dialog box should be displayed
  file_name | if empty a dialog box shows up
  dest_df | destination folder reference. if dest_df is specified, data is loaded into this folder. else, by default, a new folder derived from the file name is created in root:

Returns
  the return value of the function is a data folder reference of the created data folder.
  global variable file_id contains ID number of open HDF5 file from HDF5OpenFile. zero if an error occurred.
  global string s_filepath in new data folder contains the full file path on disk.
  global string s_scanpaths in new data folder contains a list of scan groups inside the file.

Definition at line 450 of file pearl-pshell-import.ipf.

## psh5_preview()

dfr psh5_preview | ( | string | path_name,
|  | string | file_name,
|  | dfref | dest_df = `defaultValue`,
|  | string | preview_datasets = `defaultValue`
| ) |  |

load preview

load information about the file structure and a preview dataset

Definition at line 345 of file pearl-pshell-import.ipf.

## psh_load_general_string()

string psh_load_general_string | ( | dfref | file_df,
|  | string | name
| ) |  |

load a string from the general group.

the general group contains the following datasets: authors, pgroup, proposal, proposer, sample.

data is loaded into a global string in the current data folder. arrays with multiple items are loaded into a comma-separated list. a missing item defaults to the empty string.

Parameters

  file_df | data folder reference of open HDF5 file from psh5_open_file(). if undefined, the current datafolder is assumed.

Returns
  comma-separated list of values.

Definition at line 1838 of file pearl-pshell-import.ipf.

## reduce_slab_image()

static threadsafe wave reduce_slab_image | ( | wave | slabdata,
|  | wave | image,
|  | funcref | reduction_func,
|  | string | reduction_params
| ) |  |

static

Definition at line 1776 of file pearl-pshell-import.ipf.

## reduce_slab_worker()

static threadsafe variable reduce_slab_worker | ( | funcref | reduction_func | ) |

static

Definition at line 1737 of file pearl-pshell-import.ipf.

## twave2list()

static string twave2list | ( | wave | wt,
|  | string | sep
| ) |  |

static

convert text wave to list.

Definition at line 532 of file pearl-pshell-import.ipf.

## unique_strings()

static string unique_strings | ( | string | list | ) |

static

remove duplicate items from list

Parameters

  list | semicolon-separated list of strings. strings can contain any printable character except the semicolon.

Returns
  list of strings with duplicates (second and further instances) removed. all remaining items retain the position of their first occurrence in the original list. the function uses Igor's FindDuplicates operation.

Definition at line 764 of file pearl-pshell-import.ipf.

## wave2list()

static string wave2list | ( | wave | w,
|  | string | format,
|  | string | sep
| ) |  |

static

convert numeric wave to list.

Definition at line 549 of file pearl-pshell-import.ipf.

## Variable Documentation

## kAngleDimLabel

const string kAngleDimLabel = "angle"

Dimension label for the angle dispersive dimension of multi-dimensional datasets.

Definition at line 53 of file pearl-pshell-import.ipf.

## kDataDimLabel

const string kDataDimLabel = "data"

Dimension label for the data dimension.

This label may be used to store the parameters for the `setscale d` operation.

Definition at line 60 of file pearl-pshell-import.ipf.

## kDSCAll

const variable kDSCAll = 0xffff

Definition at line 86 of file pearl-pshell-import.ipf.

## kDSCAttrs

const variable kDSCAttrs = 0x0020

Definition at line 79 of file pearl-pshell-import.ipf.

## kDSCDetectors

const variable kDSCDetectors = 0x0002

Definition at line 75 of file pearl-pshell-import.ipf.

## kDSCDiags

const variable kDSCDiags = 0x0040

Definition at line 80 of file pearl-pshell-import.ipf.

## kDSCEssentialDiags

const variable kDSCEssentialDiags = 0x0010

Definition at line 78 of file pearl-pshell-import.ipf.

## kDSCMeta

const variable kDSCMeta = 0x0100

Definition at line 82 of file pearl-pshell-import.ipf.

## kDSCMonitors

const variable kDSCMonitors = 0x0200

Definition at line 83 of file pearl-pshell-import.ipf.

## kDSCOther

const variable kDSCOther = 0x8000

Definition at line 85 of file pearl-pshell-import.ipf.

## kDSCPositioners

const variable kDSCPositioners = 0x0001

Definition at line 74 of file pearl-pshell-import.ipf.

## kDSCPreview

const variable kDSCPreview = 0x0008

Definition at line 77 of file pearl-pshell-import.ipf.

## kDSCRegions

const variable kDSCRegions = 0x0400

Definition at line 84 of file pearl-pshell-import.ipf.

## kDSCScientaScaling

const variable kDSCScientaScaling = 0x0004

Definition at line 76 of file pearl-pshell-import.ipf.

## kDSCSnaps

const variable kDSCSnaps = 0x0080

Definition at line 81 of file pearl-pshell-import.ipf.

## kEnergyDimLabel

const string kEnergyDimLabel = "energy"

Dimension label for the energy dispersive dimension of multi-dimensional datasets.

Definition at line 50 of file pearl-pshell-import.ipf.

## kEssentialDiagnostics

const string kEssentialDiagnostics = "ManipulatorX;ManipulatorY;ManipulatorZ;ManipulatorTheta;ManipulatorTilt;ManipulatorPhi;MonoEnergy;"

List of diagnostic datasets that are normally loaded with a scan.

Definition at line 69 of file pearl-pshell-import.ipf.

## kPreviewDatasets

const string kPreviewDatasets = "ImageEnergyDistribution;ScientaSpectrum;ScientaImage;Counts;SampleCurrent;"

List of preferred datasets to load for preview.

Definition at line 63 of file pearl-pshell-import.ipf.

## kScanDimLabel

const string kScanDimLabel = "scan"

Dimension label for the scan dimension of multi-dimensional datasets.

Definition at line 56 of file pearl-pshell-import.ipf.

## kScientaScalingDatasets

const string kScientaScalingDatasets = "LensMode;ScientaChannelBegin;ScientaChannelEnd;ScientaSliceBegin;ScientaSliceEnd;Eph;"

List of datasets that must be loaded to determine the axis scaling of a Scienta image.

Definition at line 66 of file pearl-pshell-import.ipf.

## kTransposedDatasets

const string kTransposedDatasets = "ScientaImage;"

List of datasets that must be transposed upon loading.

Definition at line 72 of file pearl-pshell-import.ipf.

kill_matching_waves
string kill_matching_waves(dfref dfr, string pattern, variable recurse, string killed=defaultValue)
kill any waves matching a pattern in the experiment
Definition: pearl-pshell-import.ipf:2554