All references to folders and files inside buffer were updated.
- Base folder to write detector: detector_folder
- Name of the modules inside detector_folder: module_name
- Data grouping folders based on pulse_id: data_folder
- Data grouping files, based on pulse_id: data_file
Since we are always using this recorders in the context of
Jungfraus, we should start naming things in this context as
well. Instead of root_folder we will be using detector_folder
(as root of the buffer on disk for a specific detector) and
module_name instead of device_name (to specify one module
inside the detector).
The binary reader reads 1 frame at a time from a module - the
difference being loading 1 image instead of 1 block of images.
This will allow to set the offset and pulse_id increment easily
without complicated internal buffer calculations.
When started to use the buffer on GPFS we notices an increase
of metadata access times on GPFS. To try to reduce the number
of metadata updates we set the file size at creation time.
The pulse_id stream can be used to synchronize components.
For example, the writer can listen for the current pulse_id
and write with 1 second delay to a file. This will help us
eliminate the need to wait for the run to complete to start
the data write request.
Since processing of live data can be done at 100Hz it would
be more difficult to stream the full rate with a PUB/SUB mode.
To support distributed live analysis we move to PUSH/PULL
model, since we can support only one live processing at a time
anyway (network constraints when working with large detectors).
babic_a
Dmitry Ozerov