Files
ADAndor/documentation/NDPluginFile.html

1029 lines
42 KiB
HTML
Executable File

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>areaDetector Plugin NDPluginFile</title>
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type" />
</head>
<body>
<div style="text-align: center">
<h1>
areaDetector Plugin NDPluginFile</h1>
<h2>
August 8, 2011</h2>
<h2>
Mark Rivers</h2>
<h2>
University of Chicago</h2>
</div>
<h2>
Contents</h2>
<ul>
<li><a href="#Overview">Overview</a></li>
<li><a href="#JPEG">JPEG file plugin</a></li>
<li><a href="#TIFF">TIFF file plugin</a></li>
<li><a href="#Magick">GraphicsMagick file plugin</a></li>
<li><a href="#netCDF">netCDF file plugin</a></li>
<li><a href="#NeXus">NeXus (HDF) file plugin</a></li>
<li><a href="#HDF5">HDF5 file plugin</a></li>
</ul>
<h2 id="Overview">
Overview
</h2>
<p>
NDPluginFile is a base class from which actual file plugins are derived. There are
currently file plugins for JPEG, TIFF, netCDF, Nexus, and HD5 file formats. The
GraphicsMagick plugin can write a large number of formats, including JPEG, TIFF,
PNG, PDF and many others.
</p>
<p>
NDPluginFile inherits from NDPluginDriver. The <a href="areaDetectorDoxygenHTML/class_n_d_plugin_file.html">
NDPluginFile class documentation</a> describes this class in detail. This class
is designed to simplify the task of supporting a new file format. A derived class
to support a new file format will typically need to implement only the pure virtual
functions openFile(), readFile(), writeFile(), and closeFile(). Note that none of
the current file plugins actually support the readFile() function yet, but this
is planned for future releases.
</p>
<p>
The NDArray callback data can be written to disk in 1 of 3 modes:
</p>
<ol>
<li>Single mode. In this mode each NDArray callback results in a separate disk file.</li>
<li>Capture mode. In this mode a memory buffer is allocated before saving begins.
Callback arrays are placed into this buffer, and when capture stops the file is
written to disk. This mode limits the number of frames that can be saved, because
they all must fit in a memory buffer. It is the fastest mode, with the least probability
of dropping arrays, because no disk I/O is required while capture is in progress.</li>
<li>Stream mode. In this mode the data are written to a single disk file for those
file formats that support multiple arrays per file (currently only netCDF; NeXus/HDF
support for multiple arrays per file is planned for a future release). Each frame
is appended to the file without closing it. It is intermediate in speed between
single mode and capture mode, but unlike capture mode it is not limited by available
memory in the number of arrays that can be saved. For file formats that do not support
multiple arrays per file (e.g. JPEG, TIFF and currently NeXus/HDF) this mode is
really the same as Single mode, except that one can specify a total number of files
to save before stopping.</li>
</ol>
<p>
At least one array with the same datatype, array size, and attributes must have
been collected by the driver (and/or plugins) from which the file saving plugin
is getting its data <b>before</b> capture or stream mode file saving is started.
This is required so that the openFile() function can know the dimensions and datatype
of the arrays.
</p>
<p>
NDPluginFile supports all of the file saving parameters defined in <a href="areaDetectorDoc.html#asynNDArrayDriver">
asynNDArrayDriver</a>, e.g. NDFilePath, NDFileName, etc. Thus, the same interface
that is used for saving files directly in a driver is used for this plugin.
</p>
<p>
The base class will delete the "original" file that the driver created for that
array if the following are all true:</p>
<ol>
<li>The DeleteDriverFile record is "Yes".</li>
<li>The file plugin has successfully written a new file.</li>
<li>The array contains an attribute called "DriverFileName" that contains the full
file name of the original file. The driver attributes XML file should contain a
line like the following:<br />
<tt>&lt;Attribute name="DriverFileName" type="PARAM" source="FULL_FILE_NAME" datatype="string"
description="Driver file name"/&gt;</tt><br />
</li>
</ol>
<p>
The file saving plugins normally determine the name of the file from the FileName
and FileNumber records. However, it is possible to have these values come instead
from attributes in the array passed to the callback. The following 3 special attributes
are used:</p>
<ol>
<li>FilePluginFileName: This attribute contains the file name.</li>
<li>FilePluginFileNumber - This attribute contains the file number.</li>
<li>FilePluginDestination - If this attribute contains the string "all" or the name
of the asyn port for this plugin (e.g. FileTIFF1) then the plugin will write the
array to a file. If this attribute has any other value then the plugin will ignore
this array, and not write a file.</li>
</ol>
<p>
Having the file information come from the array allows the driver to control which
plugin saves a particular array. For example, there may be two file writing plugins
active; the first saves the flat field files for a tomography experiment, and the
second saves the normal projections. These plugins each stream data to a separate
file. The driver knows which files are flat fields and which are normal projections,
and adds the appropriate attributes to control which plugin saves each array. This
would not be possible using a single plugin and EPICS PVs to switch the file, because
of the problem of frames being buffered in the plugin queue.</p>
<h2 id="JPEG">
JPEG file plugin
</h2>
<p>
NDFileJPEG inherits from NDPluginFile. This plugin saves data in the <a href="http://en.wikipedia.org/wiki/JPEG">
JPEG</a> file format, which is a compressed file format for storing images. There
is JPEG support for almost all languages and programs such as IDL and Matlab.
</p>
<p>
The JPEG plugin is limited to 8-bit arrays. It supports all color modes (Mono, RGB1,
RGB2, and RGB3). It is limited to a single array per file, but capture and stream
mode are supported by writing multiple JPEG files.
</p>
<p>
The JPEG plugin supports the Int32 parameter NDFileJPEGQuality to control the amount
of compression in the file. This parameter varies from 0 (maximum compression, lowest
quality) to 100 (least compression, best quality). NDFileJPEG.template defines 2
records to support this: $(P)$(R)JPEGQuality (longout) and $(P)$(R)JPEGQuality_RBV
(longin).
</p>
<p>
The <a href="areaDetectorDoxygenHTML/class_n_d_file_j_p_e_g.html">NDFileJPEG class
documentation </a>describes this class in detail.
</p>
<p>
The NDFileJPEG plugin is created with the NDFileJPEGConfigure command, either from
C/C++ or from the EPICS IOC shell.</p>
<pre>NDFileJPEGConfigure (const char *portName, int queueSize, int blockingCallbacks,
const char *NDArrayPort, int NDArrayAddr, size_t maxMemory,
int priority, int stackSize)
</pre>
<p>
For details on the meaning of the parameters to this function refer to the detailed
documentation on the NDFileJPEGConfigure function in the <a href="areaDetectorDoxygenHTML/_n_d_file_j_p_e_g_8cpp.html">
NDFileJPEG.cpp documentation</a> and in the documentation for the constructor
for the <a href="areaDetectorDoxygenHTML/class_n_d_file_j_p_e_g.html">NDFileJPEG class</a>.
</p>
<div style="text-align: center">
<h3>
NDFileJPEG.adl</h3>
<p>
<img alt="NDFileJPEG.png" src="NDFileJPEG.png" /></p>
</div>
<h2 id="TIFF">
TIFF file plugin
</h2>
<p>
NDFileTIFF inherits from NDPluginFile. This plugin saves data in the <a href="http://en.wikipedia.org/wiki/Tagged_Image_File_Format">
TIFF</a> file format, which is a popular file format for storing images. There
is TIFF support for almost all languages and programs such as IDL and Matlab.
</p>
<p>
The TIFF plugin is limited to 8, 16 and 32-bit integer arrays. It supports all color
modes (Mono, RGB1, RGB2, and RGB3). Note that some TIFF readers do not support 16
or 32 bit TIFF files, and many do not support 16 or 32 bit color files. NDFileTIFF
is limited to a single array per file, but capture and stream mode are supported
by writing multiple TIFF files.
</p>
<p>
The <a href="areaDetectorDoxygenHTML/class_n_d_file_t_i_f_f.html">NDFileNetTIFF class
documentation </a>describes this class in detail.
</p>
<p>
The NDFileTIFF plugin is created with the NDFileTIFFConfigure command, either from
C/C++ or from the EPICS IOC shell.</p>
<pre>NDFileTIFFConfigure (const char *portName, int queueSize, int blockingCallbacks,
const char *NDArrayPort, int NDArrayAddr, size_t maxMemory,
int priority, int stackSize)
</pre>
<p>
For details on the meaning of the parameters to this function refer to the detailed
documentation on the NDFileTIFFConfigure function in the <a href="areaDetectorDoxygenHTML/_n_d_file_t_i_f_f_8cpp.html">
NDFileTIFF.cpp documentation</a> and in the documentation for the constructor
for the <a href="areaDetectorDoxygenHTML/class_n_d_file_t_i_f_f.html">NDFileTIFF class</a>.
</p>
<div style="text-align: center">
<h3>
NDFileTIFF.adl</h3>
<p>
<img alt="NDFileTIFF.png" src="NDFileTIFF.png" /></p>
</div>
<h2 id="Magick">
GraphicsMagick file plugin
</h2>
<p>
NDFileMagick inherits from NDPluginFile. This plugin saves data in any of the formats
supported by the <a href="http://www.graphicsmagick.org/">GraphicsMagick</a> package.
GraphicsMagick supports dozens of file formats, including TIFF, JPEG, PNG, PDF,
and many others. GraphicsMagick automatically selects the output file format based
on the extension of the file (.jpg=JPEG, .tif=TIFF, etc. The GraphicsMagick plugin
should be able to write files in any format in the <a href="http://www.graphicsmagick.org/formats.html">
list of GraphicsMagick supported file formats&quot;</a> that has a &quot;W&quot;
or &quot;RW&quot; in the Mode column.
</p>
<p>
The GraphicsMagick plugin is limited to 8 and 16-bit integer arrays. It supports
color modes Mono and RGB1. NDFileMagick is limited to a single array per file, but
capture and stream mode are supported by writing multiple files.
</p>
<p>
The GraphicsMagick plugin supports the Int32 parameter NDFileMagickCompressType
to control the compression mode of the file. NDFileMagick.template defines 2 records
to support this: $(P)$(R)CompressType (longout) and $(P)$(R)CompressType_RBV (longin).
The following are the supported compression types:</p>
<ul>
<li>&quot;None&quot;</li>
<li>&quot;BZip&quot;</li>
<li>&quot;FAX&quot;</li>
<li>&quot;Group 4&quot;</li>
<li>&quot;JPEG&quot;</li>
<li>&quot;LZW&quot;</li>
<li>&quot;RLE&quot;&quot;</li>
<li>&quot;Zip&quot;</li>
</ul>
<p>
No formats support all of these compression types. Many support only one, or have
an implicit compression mode and so ignore the CompressType parameter. For example,
the JPEG and PNG formats support only their respective implicit compression modes.
I have determined empirically that the TIFF format supports "None", "JPEG", "LZW",
and "ZIP", while the PDF format supports "None", "BZip", "FAX", and "JPEG".</p>
<p>
The <a href="areaDetectorDoxygenHTML/class_n_d_file_magick.html">NDFileMagick class
documentation </a>describes this class in detail.
</p>
<p>
The NDFileMagick plugin is created with the NDFileMagickConfigure command, either
from C/C++ or from the EPICS IOC shell.</p>
<pre>NDFileMagickConfigure(const char *portName, int queueSize, int blockingCallbacks,
const char *NDArrayPort, int NDArrayAddr, size_t maxMemory,
int priority, int stackSize)
</pre>
<p>
For details on the meaning of the parameters to this function refer to the detailed
documentation on the NDFileMagickConfigure function in the <a href="areaDetectorDoxygenHTML/_n_d_file_magick_8cpp.html">
NDFileMagick.cpp documentation</a> and in the documentation for the constructor
for the <a href="areaDetectorDoxygenHTML/class_n_d_file_magick.html">NDFileMagick
class</a>.
</p>
<div style="text-align: center">
<h3>
NDFileMagick.adl</h3>
<p>
<img alt="NDFileMagick.png" src="NDFileMagick.png" /></p>
</div>
<h2 id="netCDF">
netCDF file plugin
</h2>
<p>
NDFileNetCDF inherits from NDPluginFile. This plugin saves data in the <a href="http://www.unidata.ucar.edu/software/netcdf/">
netCDF</a> file format, which is a portable self-describing binary file format
supported by <a href="http://www.unidata.ucar.edu/">UniData</a> at <a href="http://www.ucar.edu/">
UCAR (University Corporation for Atmospheric Research).</a> There are netCDF libraries
for C, C++, Fortran, and Java. Other languages, including Matlab and IDL have built-in
support for netCDF. There are also add-on interfaces available for Python, Ruby
and other languages.
</p>
<p>
The netCDF plugin supports all NDArray data types and any number of array dimensions.
It also has full support for NDArray attributes. It will write all attributes associated
with the NDArray to the file. If multiple arrays are written to a single netCDF
file (stream or capture mode) then each attribute will be an array, with the attribute
value for each NDArray in the file being stored. Note that the number and data types
of attributes must not be changed while file capture or file streaming are in progress
because that would change the structure of the attribute array. Also the colorMode
attribute must not be changed while capture or streaming is in progress, because
that would change the structure of the NDArray data.</p>
<p>
The <a href="areaDetectorDoxygenHTML/class_n_d_file_net_c_d_f.html">NDFileNetCDF class
documentation </a>describes this class in detail.
</p>
<p>
The NDFileNetCDF plugin is created with the NDFileNetCDFConfigure command, either
from C/C++ or from the EPICS IOC shell.</p>
<pre>NDFileNetCDFConfigure (const char *portName, int queueSize, int blockingCallbacks,
const char *NDArrayPort, int NDArrayAddr, size_t maxMemory,
int priority, int stackSize)
</pre>
<p>
For details on the meaning of the parameters to this function refer to the detailed
documentation on the NDFileNetCDFConfigure function in the <a href="areaDetectorDoxygenHTML/_n_d_file_net_c_d_f_8cpp.html">
NDFileNetCDF.cpp documentation</a> and in the documentation for the constructor
for the <a href="areaDetectorDoxygenHTML/class_n_d_file_net_c_d_f.html">NDFileNetCDF
class</a>.
</p>
<p>
The following is the header contents of a netCDF file produced by this plugin. This
information was produced with the following command:</p>
<pre>ncdump -h test_netCDF_68.nc
netcdf test_netCDF_68 {
dimensions:
numArrays = UNLIMITED ; // (10 currently)
dim0 = 240 ;
dim1 = 320 ;
dim2 = 1 ;
attrStringSize = 256 ;
variables:
int uniqueId(numArrays) ;
double timeStamp(numArrays) ;
float array_data(numArrays, dim0, dim1, dim2) ;
int Attr_colorMode(numArrays) ;
double Attr_AcquireTime(numArrays) ;
double Attr_RingCurrent(numArrays) ;
int Attr_ImageCounter(numArrays) ;
char Attr_CameraModel(numArrays, attrStringSize) ;
int Attr_BinX(numArrays) ;
int Attr_BinY(numArrays) ;
double Attr_AttrTimeStamp(numArrays) ;
double Attr_ROI0Mean(numArrays) ;
double Attr_ROI1Mean(numArrays) ;
char Attr_FilePath(numArrays, attrStringSize) ;
char Attr_FileName(numArrays, attrStringSize) ;
// global attributes:
:dataType = 6 ;
:NDNetCDFFileVersion = 3. ;
:numArrayDims = 3 ;
:dimSize = 1, 320, 240 ;
:dimOffset = 0, 0, 0 ;
:dimBinning = 1, 2, 2 ;
:dimReverse = 0, 0, 0 ;
:Attr_colorMode_DataType = &quot;Int32&quot; ;
:Attr_colorMode_Description = &quot;Color mode&quot; ;
:Attr_colorMode_Source = ;
:Attr_colorMode_SourceType = &quot;Driver&quot; ;
:Attr_AcquireTime_DataType = &quot;Float64&quot; ;
:Attr_AcquireTime_Description = &quot;Camera acquire time&quot; ;
:Attr_AcquireTime_Source = &quot;13SIM1:cam1:AcquireTime&quot; ;
:Attr_AcquireTime_SourceType = &quot;EPICS_PV&quot; ;
:Attr_RingCurrent_DataType = &quot;Float64&quot; ;
:Attr_RingCurrent_Description = &quot;Storage ring current&quot; ;
:Attr_RingCurrent_Source = &quot;S:SRcurrentAI&quot; ;
:Attr_RingCurrent_SourceType = &quot;EPICS_PV&quot; ;
:Attr_ImageCounter_DataType = &quot;Int32&quot; ;
:Attr_ImageCounter_Description = &quot;Image counter&quot; ;
:Attr_ImageCounter_Source = &quot;ARRAY_COUNTER&quot; ;
:Attr_ImageCounter_SourceType = &quot;Param&quot; ;
:Attr_CameraModel_DataType = &quot;String&quot; ;
:Attr_CameraModel_Description = &quot;Camera model&quot; ;
:Attr_CameraModel_Source = &quot;MODEL&quot; ;
:Attr_CameraModel_SourceType = &quot;Param&quot; ;
:Attr_BinX_DataType = &quot;Int32&quot; ;
:Attr_BinX_Description = &quot;X binning&quot; ;
:Attr_BinX_Source = &quot;13SIM1:ROI1:0:BinX_RBV&quot; ;
:Attr_BinX_SourceType = &quot;EPICS_PV&quot; ;
:Attr_BinY_DataType = &quot;Int32&quot; ;
:Attr_BinY_Description = &quot;Y binning&quot; ;
:Attr_BinY_Source = &quot;13SIM1:ROI1:0:BinY_RBV&quot; ;
:Attr_BinY_SourceType = &quot;EPICS_PV&quot; ;
:Attr_AttrTimeStamp_DataType = &quot;Float64&quot; ;
:Attr_AttrTimeStamp_Description = &quot;Time stamp&quot; ;
:Attr_AttrTimeStamp_Source = &quot;TIME_STAMP&quot; ;
:Attr_AttrTimeStamp_SourceType = &quot;Param&quot; ;
:Attr_ROI0Mean_DataType = &quot;Float64&quot; ;
:Attr_ROI0Mean_Description = &quot;Mean value ROI 0&quot; ;
:Attr_ROI0Mean_Source = &quot;MEAN_VALUE&quot; ;
:Attr_ROI0Mean_SourceType = &quot;Param&quot; ;
:Attr_ROI1Mean_DataType = &quot;Float64&quot; ;
:Attr_ROI1Mean_Description = &quot;Mean value ROI 0&quot; ;
:Attr_ROI1Mean_Source = &quot;MEAN_VALUE&quot; ;
:Attr_ROI1Mean_SourceType = &quot;Param&quot; ;
:Attr_FilePath_DataType = &quot;String&quot; ;
:Attr_FilePath_Description = &quot;File path&quot; ;
:Attr_FilePath_Source = &quot;13SIM1:netCDF1:FilePath_RBV&quot; ;
:Attr_FilePath_SourceType = &quot;EPICS_PV&quot; ;
:Attr_FileName_DataType = &quot;String&quot; ;
:Attr_FileName_Description = &quot;File name&quot; ;
:Attr_FileName_Source = &quot;13SIM1:netCDF1:FileName_RBV&quot; ;
:Attr_FileName_SourceType = &quot;EPICS_PV&quot; ;
} </pre>
<p>
ncdump is one of a number of very useful command line utilities that come with the
netCDF package. The -h option to ncdump means to dump only the header information,
not the variable data. This is an explanation of this output:
</p>
<ul>
<li>dimensions: numArrays is the number of arrays in the file. It will be 1 for files
collected in Single mode, and is normally &gt;1 for files collected in Capture or
Stream mode. For each array dim0 is the slowest varying dimension, dim1 the next
slowest, etc. attrStringSize is the maximum string length for string attributes.</li>
<li>variables: There are 15 variables in this netCDF file. uniqueId is the unique
ID number of each array. timeStamp is the timestamp in seconds for each array. array_data
is the array data. Its data type depends on the data type of the NDArray data passed
in the callbacks. It dimensions are [numArrays, dim0, dim1, ...dimN]. This notation
is in the Fortran syntax where the slowest varying dimension comes first in the
list. The remaining variables all have the prefix Attr_ and are the attributes for
the arrays. Each can have its own data type, and all have the numArrays elements.</li>
<li>global attributes. dataType is the NDDataType_t enum value for the array data
type. numArrayDims is the number of dimensions in each array. array_data has 1 more
dimension than this, numArrays, because it contains all of the array callback data.
dimSize is an array[numArrayDims] containing the size of each dimension, with the
fastest varying dimension first. dimOffset, dimBinning, and dimReverse are the values
of the offset, binning and reverse fields in the NDDimension_t structure for each
dimension. The remaining netCDF global attributes all have the prefix Attr_, and
describe the NDArray attribute values. For each NDArray attribute there is information
on the data type, a description, source string and source type. </li>
</ul>
<p>
There is an IDL function, <a href="http://cars.uchicago.edu/software/idl/detector_routines.html#read_nd_netcdf">
read_nd_netcdf</a> that can be used to read the netCDF files created by this plugin.
This routine is contained in the <a href="http://cars.uchicago.edu/software/idl/detectors.html#read_nd_netcdf">
CARS IDL detector package</a>. This function is also contained in the areaDetector
distribution in the Viewers/IDL directory.
</p>
<p>
There is a plugin for the popular <a href="http://rsbweb.nih.gov/ij/">ImageJ</a>
program that can be used to read the netCDF files created by this plugin. This ImageJ
plugin can be downloaded <a href="http://lmb.informatik.uni-freiburg.de/lmbsoft/imagej_plugins/netcdf.en.html">
here</a>. This plugin is also contained in the areaDetector distribution in the
Viewers/ImageJ/EPICS_areaDetector directory.
</p>
<div style="text-align: center">
<h3>
NDFileNetCDF.adl</h3>
<p>
<img alt="NDFileNetCDF.png" src="NDFileNetCDF.png" /></p>
</div>
<h2 id="NeXus">
NeXus (HDF) file plugin
</h2>
<p>
A plugin to write <a href="http://www.nexusformat.org/Main_Page">NeXus</a> files
was written by John Hammonds from the APS. NeXus is a standard format for x-ray
and neutron data based on <a href="http://www.hdfgroup.org">HDF</a>. This is a very
general file format, capable of storing any type of array data and meta-data.</p>
<p>
The <a href="areaDetectorDoxygenHTML/class_n_d_file_nexus.html">NDFileNexus class
documentation </a>describes this class in detail.
</p>
<p>
The NDFileNexus plugin is created with the NDFileNexusConfigure command, either
from C/C++ or from the EPICS IOC shell.</p>
<pre>NDFileNexusConfigure (const char *portName, int queueSize, int blockingCallbacks,
const char *NDArrayPort, int NDArrayAddr, size_t maxMemory,
int priority, int stackSize)
</pre>
<p>
For details on the meaning of the parameters to this function refer to the detailed
documentation on the NDFileNexusConfigure function in the <a href="areaDetectorDoxygenHTML/_n_d_file_nexus_8cpp.html">
NDFileNexus.cpp documentation</a> and in the documentation for the constructor
for the <a href="areaDetectorDoxygenHTML/class_n_d_file_nexus.html">NDFileNexus class</a>.</p>
<p>
NDFileNeXus uses 2 additional parameters to define the location of an XML file that
is read to determine the contents of the NeXus files written by this plugin. These
are described in the following table.</p>
<table border="1" cellpadding="2" cellspacing="2" style="text-align: left">
<tbody>
<tr>
<td align="center" colspan="7,">
<b>Parameter Definitions in NDFileNexus.h and EPICS Record Definitions in NDFileNexus.template</b></td>
</tr>
<tr>
<th>
Parameter index variable</th>
<th>
asyn interface</th>
<th>
Access</th>
<th>
Description</th>
<th>
drvInfo string</th>
<th>
EPICS record name</th>
<th>
EPICS record type</th>
</tr>
<tr>
<td align="center" colspan="7,">
<b>Location of XML file to configure NeXus file contents</b></td>
</tr>
<tr>
<td>
NDFileNexusTemplatePath</td>
<td>
asynOctet</td>
<td>
r/w</td>
<td>
Path to XML template file</td>
<td>
TEMPLATE_FILE_PATH</td>
<td>
$(P)$(R)TemplateFilePath<br />
$(P)$(R)TemplateFilePath_RBV</td>
<td>
waveform<br />
waveform</td>
</tr>
<tr>
<td>
NDFileNexusTemplateFile</td>
<td>
asynOctet</td>
<td>
r/w</td>
<td>
Name of XML template file</td>
<td>
TEMPLATE_FILE_NAME</td>
<td>
$(P)$(R)TemplateFileName<br />
$(P)$(R)TemplateFileName_RBV</td>
<td>
waveform<br />
waveform</td>
</tr>
</tbody>
</table>
<p>
There is currently no documentation on the contents of the XML template file. However,
there are example XML template files in the iocSimDetector and iocPerkinElmer directories.
Documentation on the XML file contents will be written ASAP.</p>
<p>
The prebuilt Linux libraries libhdf5.a and libNeXus.a are built with HDF5 1.6.9.
When they are built with the latest version, 1.8.2, they require GLIBC version 2.7
or higher, i.e. /lib/libc-2.7.so or higher. Since users may want to install areaDetector
on older Linux systems (which predate Fedora Core 8 for example), it was decided
to use this older version of HDF5. Future releases of areaDetector may use HDF5
1.8.2 or later, and hence not work with older Linux systems.</p>
<h2 id="HDF5">
HDF5 file plugin
</h2>
<p>
NDFileHDF5 inherits from NDPluginFile. This plugin uses the HDF5 libraries to store
data. HDF5 file format is a self-describing binary format supported by the <a href="http://www.hdfgroup.org/HDF5">
hdfgroup</a>.
</p>
<p>
The plugin supports all NDArray datatypes and any number of NDArray dimensions (tested
up to 3). It supports storing multiple NDArrays in a single file (in stream or capture
modes) where each NDArray get appended to an extra dimension.
</p>
<p>
NDArray attributes are stored in the HDF5 file. In case of multi-frame files the
attributes are stored in 1D datasets (arrays).
</p>
<p>
The NDFileHDF5 plugin is created with the NDFileHDF5Configure command, either from
C/C++ or from the EPICS IOC shell.</p>
<pre>NDFileHDF5Configure (const char *portName, int queueSize, int blockingCallbacks,
const char *NDArrayPort, int NDArrayAddr, size_t maxMemory,
int priority, int stackSize)
</pre>
<p>
For details on the meaning of the parameters to this function refer to the detailed
documentation on the NDFileNexusConfigure function in the <a href="areaDetectorDoxygenHTML/_n_d_file_h_d_f5_8cpp.html">
NDFileHDF5.cpp documentation</a> and in the documentation for the constructor
for the <a href="areaDetectorDoxygenHTML/class_n_d_file_h_d_f5.html">NDFileHDF5 class</a>.</p>
<h3>
File structure
</h3>
<p>
The HDF5 files comprises a hierachial data structure, similar to a file system structure
with directories (groups) and files (datasets) [<a href="http://www.hdfgroup.org/HDF5/Tutor/fileorg.html">ref</a>]
</p>
<p>
This plugin writes HDF5 files that are compatible with the Nexus file format. This
is achieved by defining a specific hierachial structure of groups and datasets and
by tagging elements in the hierachi with certain &quot;NX_class=nnn&quot; attributes.
Although Nexus libraries are not used to write the data to disk, this file structure
allow Nexus-aware readers to open and read the content of these HDF5 files. This
has been tested with the Nexus reader in the <a href="http://opengda.org/">GDA application</a>.
</p>
<p>
The directory structure of the HDF5 files, generated by this plugin:</p>
<pre>entry &lt;-- NX_class=NXentry
|
+--instrument &lt;-- NX_class=NXinstrument
|
+--NDAttributes
| |
| +--- &lt;-- Any number of EPICS PV based NDAttributes as individual 1D datasets
| +--- The PV attributes are in the 'instrument' group as they are expected
| to originate from the beamline/instrument rather than the detector itself
|
+--detector &lt;-- NX_class=NXdetector
| |
| +--data &lt;-- NX_class=SDS, signal=1
| |
| +--NDAttributes
| |
| +--- &lt;-- Any number of PARAM type NDAttributes as individual 1D datasets
| +--- These type of parameters are in the 'detector' group as they
| originate from the areaDetector system
|
+--performance &lt;-- Performance of the file writing
|
+--timestamp &lt;-- A 2D dataset of different timing measurements taking during file writing
</pre>
<h3>
HDF5 File Viewers
</h3>
<p>
<a href="http://www.hdfgroup.org/hdf-java-html/hdfview/index.html">HDFView</a> is
a simple GUI tool for viewing and browsing HDF files. It has some limited support
for viewing images, plotting graphs and displaying data tables.
</p>
<p>
The <a href="http://www.hdfgroup.org/HDF5/release/obtain5.html">HDF5 libraries</a>
also ships with a number of command-line tools for browsing and dumping data.
</p>
<p>
The screenshot below shows the hdfview application with a datafile open. The datafile
is generated by the plugin and a number of elements are visible:</p>
<ul>
<li>A number of NDAttributes sourced from EPICS PVs during detector acquisition appear
as 1D datasets in the group &quot;/entry/instrument/NDAttributes/&quot;</li>
<li>Several NDAttributes from the areaDetector show up as datasets in the group &quot;/entry/instrument/detector/NDAttributes/&quot;</li>
<li>The image data is in the dataset &quot;/entry/instrument/detector/data&quot;.
The metadata (in HDF known as &quot;attributes&quot;) for that dataset indicate
8bit unsigned char data, 10 frames of 60x40 pixels</li>
<li>Image and table view of the first frame data is open</li>
</ul>
<div style="text-align: center">
<img alt="==== HDFView-screenshot.png ====" src="HDFView-screenshot.png" /></div>
<h3>
Multiple Dimensions
</h3>
<p>
Both areaDetector and the HDF5 file format supports multidimensional datasets. The
dimensions from the NDArray are preserved when writing to the HDF5 file. In multi-frame
files (plugin in Stream or Capture mode) an additional dimension is added to the
dataset to hold the array of frames.
</p>
<p>
In addition to the dimensions of the NDArray it is also possible to specify up to
2 extra &quot;virtual&quot; dimensions to store datasets in the file. This is to
support applications where a sample is scanned in up to two dimensions, say X and
Y. For each scan point a dataset comprising of multiple frames can be stored. The
length of (i.e. number of points in) each of the two virtual dimensions have to
be specified before the plugin opens the file for writing. This feature is only
supported in the Stream and Capture modes.
</p>
<p>
This feature allow for creating very large sets of scan data which matches the dimensions
of the performed scan in one datafile. Depending on the application this can be
a benefit in post processing.
</p>
<p>
The figure below illustrate the use of the two extra &quot;virtual&quot; dimensions
in a 2D (X,Y) raster scan with N frames per point:</p>
<div style="text-align: center">
<img alt="HDFmultiple-dimensions.png" src="HDFmultiple-dimensions.png" /></div>
<p>
Prior to starting a scan like this the user will need to configure the number of
virtual dimensions to use (none, 1 or 2); the number of frames per point; and the
length of each of the virtual dimensions (4 x 2 in the example figure). It is not
possible to change the number or size of dimensions while the file is open.
</p>
<p>
For 2D image (greyscale) formats the dimensions in the multiframe HDF5 file are
organised as follows:</p>
<ul>
<li>For a multiframe file with no use of &quot;virtual&quot; dimension the order is:
{Nth frame, width, height}</li>
<li>For a multiframe file using 1 &quot;virtual&quot; dimension (X) the order is:
{X, Nth frame, width, height}</li>
<li>For a multiframe file using 2 &quot;virtual&quot; dimension (X,Y) the order is:
{Y, X, Nth frame, width, height}</li>
</ul>
<h3>
NDArray attributes
</h3>
<p>
The attributes from NDArrays are stored in the HDF5 files. The list of attributes
is loaded when a file is opened so XML attributes files should not be reloaded during
an acquisition run.
</p>
<p>
The type of attribute is used to determine where in the file structure the attribute
data will end up. All attribute datasets will be named by the NDArray attribute
name. They will also have metadata (known as HDF &quot;attributes&quot;) to indicate
their source type and origin.</p>
<ul>
<li>Attributes from EPICS PVs (type=&quot;EPICS_PV&quot;) will be stored as 1D datasets
in the group &quot;/entry/instrument/NDAttributes/&quot;
<ul>
<li>Special case is attributes of datatype string, which will be stored as HDF5 metadata
attributes to the same group.</li>
</ul>
</li>
<li>All other attributes (type=&quot;PARAM&quot;) will be stored as 1D datasets in
the group &quot;/entry/instrument/detector/NDAttributes/&quot;
<ul>
<li>Special case is attributes of datatype string, which will be stored as HDF5 metadata
attributes to the same group.</li>
</ul>
</li>
</ul>
<h3>
Compression
</h3>
<p>
The HDF5 library supports a number of compression algorithms. When using HDF5 libraries
to write and read files the compression is seemless: it only need to be switched
on when writing and HDF5 enabled applications can read the files without any additional
configuration. Only one compression filter can be applied at the time.
</p>
<p>
The following compression filters are supported in the NDFileHDF5 plugin:</p>
<ul>
<li>Lossless <a href="http://www.hdfgroup.org/doc_resource/SZIP/">SZIP</a> compression
is using a separate library from the hdfgroup. NOTE: The szip library contains the
following in its COPYING license agreement file:<br />
<tt>Revocable (in the event of breach by the user or if required by law), royalty-free,
nonexclusive sublicense to use SZIP compression software routines and underlying
patents for non-commercial, scientific use only is hereby granted by ICs, LLC, to
users of and in conjunction with HDF data storage and retrieval file format and
software library products.</tt><br />
This means that the szip compression should not be used by commercial users without
first obtaining a license.</li>
<li>External libz -also lossless</li>
<li><a href="http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetNbit">N-bit</a>
compression is a bit-packing scheme to be used when a detector provide fewer databits
than standard 8,16,32 bit words. Data width and offset in the word is user configurable</li>
</ul>
<h3>
Parameters and Records
</h3>
<table border="1" cellpadding="2" cellspacing="2" style="text-align: left">
<tbody>
<tr>
<td align="center" colspan="7,">
<b>Parameter Definitions and EPICS Record Definitions in NDFileHDF5.template</b></td>
</tr>
<tr>
<th>
Parameter index variable</th>
<th>
asyn interface</th>
<th>
Access</th>
<th>
Description</th>
<th>
drvInfo string</th>
<th>
EPICS record name</th>
<th>
EPICS record type</th>
</tr>
<tr>
<td>
nRowChunks</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
Configure HDF5 &quot;chunking&quot; to approriate size for the filesystem: sets
number of rows to use per chunk</td>
<td>
HDF5_nRowChunks</td>
<td>
$(P)$(R)NumRowChunks<br />
$(P)$(R)NumRowChunks_RBV</td>
<td>
longout<br />
longin</td>
</tr>
<tr>
<td>
storeAttributes</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
Enable or disable support for storing NDArray attributes in file</td>
<td>
HDF5_storeAttributes</td>
<td>
$(P)$(R)StoreAttr<br />
$(P)$(R)StoreAttr_RBV</td>
<td>
bo<br />
bi</td>
</tr>
<tr>
<td>
storePerformance</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
Enable or disable support for storing file IO timing measurements in file</td>
<td>
HDF5_storePerformance</td>
<td>
$(P)$(R)StorePerform<br />
$(P)$(R)StorePerform_RBV</td>
<td>
bo<br />
bi</td>
</tr>
<tr>
<td align="center" colspan="7,">
<b>Additional Virtual Dimensions</b></td>
</tr>
<tr>
<td>
nExtraDims</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
Number of extra dimensions [0..2]</td>
<td>
HDF5_nExtraDims</td>
<td>
$(P)$(R)NumExtraDims<br />
$(P)$(R)NumExtraDims</td>
<td>
mbbo<br />
mbbi</td>
</tr>
<tr>
<td>
extraDimSizeN</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
Size of extra dimension N (no. of frames per point)</td>
<td>
HDF5_extraDimSizeN</td>
<td>
$(P)$(R)ExtraDimSizeN<br />
$(P)$(R)ExtraDimSizeN_RBV</td>
<td>
</td>
</tr>
<tr>
<td>
extraDimSizeX</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
Size of extra dimension X</td>
<td>
HDF5_extraDimSizeX</td>
<td>
$(P)$(R)ExtraDimSizeX<br />
$(P)$(R)ExtraDimSizeX_RBV</td>
<td>
longout<br />
longin</td>
</tr>
<tr>
<td>
extraDimSizeY</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
Size of extra dimension Y</td>
<td>
HDF5_extraDimSizeY</td>
<td>
$(P)$(R)ExtraDimSizeY<br />
$(P)$(R)ExtraDimSizeY_RBV</td>
<td>
longout<br />
longin</td>
</tr>
<tr>
<td align="center" colspan="7,">
<b>Runtime Statistics</b></td>
</tr>
<tr>
<td>
totalRuntime</td>
<td>
asynFloat64</td>
<td>
r/o</td>
<td>
Total runtime in seconds from first frame to file closed</td>
<td>
HDF5_totalRuntime</td>
<td>
$(P)$(R)Runtime</td>
<td>
ai</td>
</tr>
<tr>
<td>
totalIoSpeed</td>
<td>
asynFloat64</td>
<td>
r/o</td>
<td>
Overall IO write speed in megabit per second from first frame to file closed</td>
<td>
HDF5_totalIoSpeed</td>
<td>
$(P)$(R)IOSpeed</td>
<td>
ai</td>
</tr>
<tr>
<td align="center" colspan="7,">
<b>Compression Filters</b></td>
</tr>
<tr>
<td>
compressionType</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
Select or switch off compression filter</td>
<td>
HDF5_compressionType</td>
<td>
$(P)$(R)Compression<br />
$(P)$(R)Compression_RBV</td>
<td>
mbbo<br />
mbbi</td>
</tr>
<tr>
<td>
nbitsPrecision</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
N-bit compression filter: number of data bits per pixel</td>
<td>
HDF5_nbitsPrecision</td>
<td>
$(P)$(R)NumDataBits<br />
$(P)$(R)NumDataBits_RBV</td>
<td>
longout<br />
longin</td>
</tr>
<tr>
<td>
nbitsOffset</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
N-bit compression filter: dataword bit-offset in pixel</td>
<td>
HDF5_nbitsOffset</td>
<td>
$(P)$(R)DataBitsOffset<br />
$(P)$(R)DataBitsOffset_RBV</td>
<td>
longout<br />
longin</td>
</tr>
<tr>
<td>
szipNumPixels</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
szip compression filter: number of pixels in filter [1..32]</td>
<td>
HDF5_szipNumPixels</td>
<td>
$(P)$(R)SZipNumPixels<br />
$(P)$(R)SZipNumPixels_RBV</td>
<td>
longout<br />
longin</td>
</tr>
<tr>
<td>
zCompressLevel</td>
<td>
asynInt32</td>
<td>
r/w</td>
<td>
zlib compression filter: compression level [1..9]</td>
<td>
HDF5_zCompressLevel</td>
<td>
$(P)$(R)ZLevel<br />
$(P)$(R)ZLevel_RBV</td>
<td>
longout<br />
longin</td>
</tr>
</tbody>
</table>
<p>
<i>The NDFileHDF5 plugin was developed by Ulrik Kofoed Pedersen at Diamond Light Source.</i></p>
<div style="text-align: center">
<h3>
NDFileHDF5.adl</h3>
<p>
<img alt="NDFileHDF5.png" src="NDFileHDF5.png" /></p>
</div>
</body>
</html>