In comparison with the normal writer, the block operations
have been dropped and the buffering of metadata is not sparse
anymore. Direct chunk writing used for data and buffering +
dump at end for metadata.
The difference between the normal and the LiveImageAssembler
is that the live one can process each image individually,
sacrificing storage performance for writing flexibility -
the user can decide live exactly which frames to write.
The live writer will operate ona pulse by pulse basis to
simplify the implementation of random writes (writes of any
pulse_id defined either in advance or from an external source).
All references to folders and files inside buffer were updated.
- Base folder to write detector: detector_folder
- Name of the modules inside detector_folder: module_name
- Data grouping folders based on pulse_id: data_folder
- Data grouping files, based on pulse_id: data_file
Since we are always using this recorders in the context of
Jungfraus, we should start naming things in this context as
well. Instead of root_folder we will be using detector_folder
(as root of the buffer on disk for a specific detector) and
module_name instead of device_name (to specify one module
inside the detector).
The binary reader reads 1 frame at a time from a module - the
difference being loading 1 image instead of 1 block of images.
This will allow to set the offset and pulse_id increment easily
without complicated internal buffer calculations.
When started to use the buffer on GPFS we notices an increase
of metadata access times on GPFS. To try to reduce the number
of metadata updates we set the file size at creation time.
The pulse_id stream can be used to synchronize components.
For example, the writer can listen for the current pulse_id
and write with 1 second delay to a file. This will help us
eliminate the need to wait for the run to complete to start
the data write request.
Since processing of live data can be done at 100Hz it would
be more difficult to stream the full rate with a PUB/SUB mode.
To support distributed live analysis we move to PUSH/PULL
model, since we can support only one live processing at a time
anyway (network constraints when working with large detectors).
babic_a
Dmitry Ozerov