The live writer will listen to the stream of pulse_ids coming
from the sf-stream in order to determine the writing delay
it needs to apply. (we will wait 1 second after the data was
written to try to read it back)
We do not rely on start_pulse_id and pulse_id_step to
calculate the next image as we don't care which image this
will be - we write whatever we receive. This will allow us
to use an external driver for the pulses we desire to write.
The live writer has been changed to operate on the number of
desired pulses instead of the start and stop pulse_id. This
makes more sense in the context of live retrieval since the
stop pulse id has not been yet observed and the user would
need to calculate it anyway.
In comparison with the normal writer, the block operations
have been dropped and the buffering of metadata is not sparse
anymore. Direct chunk writing used for data and buffering +
dump at end for metadata.
The difference between the normal and the LiveImageAssembler
is that the live one can process each image individually,
sacrificing storage performance for writing flexibility -
the user can decide live exactly which frames to write.
The live writer will operate ona pulse by pulse basis to
simplify the implementation of random writes (writes of any
pulse_id defined either in advance or from an external source).
All references to folders and files inside buffer were updated.
- Base folder to write detector: detector_folder
- Name of the modules inside detector_folder: module_name
- Data grouping folders based on pulse_id: data_folder
- Data grouping files, based on pulse_id: data_file
Since we are always using this recorders in the context of
Jungfraus, we should start naming things in this context as
well. Instead of root_folder we will be using detector_folder
(as root of the buffer on disk for a specific detector) and
module_name instead of device_name (to specify one module
inside the detector).
The binary reader reads 1 frame at a time from a module - the
difference being loading 1 image instead of 1 block of images.
This will allow to set the offset and pulse_id increment easily
without complicated internal buffer calculations.
When started to use the buffer on GPFS we notices an increase
of metadata access times on GPFS. To try to reduce the number
of metadata updates we set the file size at creation time.
The pulse_id stream can be used to synchronize components.
For example, the writer can listen for the current pulse_id
and write with 1 second delay to a file. This will help us
eliminate the need to wait for the run to complete to start
the data write request.
Since processing of live data can be done at 100Hz it would
be more difficult to stream the full rate with a PUB/SUB mode.
To support distributed live analysis we move to PUSH/PULL
model, since we can support only one live processing at a time
anyway (network constraints when working with large detectors).
babic_a
Dmitry Ozerov