Commit Graph

57 Commits

Author SHA1 Message Date
babic_a be206af5e2 Remove useless variables in format 2021-04-21 17:24:34 +02:00
babic_a 63820733ff Rework the state machine interpretation
We do not have separate messages for the state machine anymore
but derive all the information from the stream itself. This
also causes the statistics to change -> it is now run
oriented instead of statistics modulo.
2021-04-21 16:33:37 +02:00
babic_a f562e6ec49 Improve live writer state machine logic 2021-04-21 16:15:20 +02:00
babic_a cc5c3cb64b Added clarification regarding stream behaviour 2021-04-15 15:22:55 +02:00
babic_a 80e562b6ed Fixed typo 2021-04-15 13:17:31 +02:00
babic_a 02bc9eadf1 Improved jf-live-writer README 2021-04-15 13:15:25 +02:00
babic_a afbb952c1f Invert broker format for StoreStream
Place the image metadata first so we can directly receive the
jf_assembler stream into this data structure.
2021-04-15 13:14:58 +02:00
babic_a 28fb3189de Tidy up jf_live_writer 2021-04-14 19:28:15 +02:00
babic_a 139504396f Fix block size in live writer compression 2021-04-14 18:56:36 +02:00
babic_a d339d06c6f Renamed broker agent to writer agent for the writer driver 2021-04-14 17:19:23 +02:00
babic_a 5a46846c01 Add debug output to writer 2021-04-14 12:19:46 +02:00
babic_a c1666d92b2 Fix number of arguments in live writer input 2021-04-14 12:19:30 +02:00
babic_a dfcb968e4b Improved jf-live-writer README 2021-04-08 13:20:20 +02:00
babic_a f9e14cbd78 Add debug docker container for jf-live-writer
This container is meant to run and test the jf-live-writer
locally without the need to install the PHDF5 library in
your local environment. Follow the instructions in
the README file.
2021-04-08 13:18:58 +02:00
babic_a 026114ff37 Separate base phdf5 image
Now the base phdf5 image is separated from the jf-live-writer
build. This is to allow for debug images of jf-live-writer to
be created without the need to go over the base image creation
process.
2021-04-08 12:54:59 +02:00
babic_a 95ec9678fd Add conditional build of jf-live-writer
jf-live-writer requires PHDF5, which is not very popular
on development machines. If you want to enable this, you have
to pass BUILD_JF_LIVE_WRITER=ON to cmake. See the README.
2021-04-08 11:53:05 +02:00
babic_a f662b45c87 Merge remote-tracking branch 'origin/master' 2021-04-08 10:57:37 +02:00
Andrej Babic 25033818dd Add jf-live-writer docker file
Docker container for jf-live-writer. This is needed due to the
phdf5 library needed by this component. If you are not willing
to install it on your machine, please use this.
2021-04-07 17:49:41 +02:00
babic_a c0c2ca3cc5 Finished live writer 2021-03-04 15:57:10 +01:00
babic_a ea81c74a3b Small bug fix in live writer config 2021-03-04 15:56:30 +01:00
babic_a 7a8944b04a Remove unused libraries from live writer 2021-03-04 15:56:00 +01:00
babic_a 09ed3d3b2e Fix stats writer bug for live writer 2021-03-04 15:48:11 +01:00
babic_a ad4f184cf0 Fix statistics for live writer 2021-02-26 15:01:02 +01:00
babic_a 8583597a51 Bits per pixel added to live writer 2021-02-26 12:03:20 +01:00
babic_a 2324ebde4f Adjust main to new signature 2021-02-25 10:51:04 +01:00
babic_a 676e8215d9 Moving image size initialization into each run 2021-02-25 10:49:59 +01:00
babic_a 592741048a Add image details to store stream format 2021-02-25 10:47:46 +01:00
babic_a 9a3df07331 Adjust main in live writer for metadata writes 2021-02-25 10:26:59 +01:00
babic_a 6ad5b9538d Remove unused OP code 2021-02-25 10:22:21 +01:00
babic_a a485736af4 Reimplemented writer 2021-02-25 10:21:49 +01:00
babic_a d8706f65da Updated HDF5 version to latest 2021-02-24 11:39:15 +01:00
babic_a fb9ec376f6 Add readme stub to H5 Writer 2021-02-24 09:01:11 +01:00
babic_a 1ac041f51c Make the live writer an MPI build 2021-02-24 08:57:15 +01:00
babic_a 3c9a9d8bce Add output symlink name to writer config 2021-02-24 08:56:54 +01:00
babic_a 29d3e4c1cb Rewrite the writer main function 2021-02-24 08:56:29 +01:00
babic_a cdfff36824 Broker agent stream format 2021-02-22 10:45:23 +01:00
babic_a 11ba1c399c Moved H5 writer to new project 2021-02-17 12:47:23 +01:00
babic_a d0df2677de Remove old writer 2021-02-17 12:46:58 +01:00
babic_a d0d5776675 Adjust the main for the jf-live-writer to ImageWriter 2021-01-21 10:32:50 +01:00
babic_a 6a1fb79f9f Fix live_writer tests 2021-01-21 10:32:15 +01:00
babic_a 2ed8b0fd1b Add avg throughput to writer stats 2021-01-21 10:29:19 +01:00
babic_a 407b00890d Finish BinaryWriter for Images in buffer 2021-01-21 10:16:38 +01:00
babic_a 94749585d8 Add ImageBinaryWriter implementation 2021-01-19 19:09:19 +01:00
babic_a 2a9e4f2b7c Implementation of WriterStats for the live writer 2021-01-19 18:35:28 +01:00
babic_a 6bbbd734c7 Add jf-live-writer
First implementation of the image buffer writer for the
 Jungfrau
2021-01-19 17:20:06 +01:00
babic_a 3055c5bc60 Add socket to listen for current pulse_id
The live writer will listen to the stream of pulse_ids coming
from the sf-stream in order to determine the writing delay
it needs to apply. (we will wait 1 second after the data was
written to try to read it back)
2020-07-21 13:25:44 +02:00
babic_a 9d9549b868 Remove start_pulse_id and pulse_id_step from live writer
We do not rely on start_pulse_id and pulse_id_step to
calculate the next image as we don't care which image this
will be - we write whatever we receive. This will allow us
to use an external driver for the pulses we desire to write.
2020-07-20 14:59:00 +02:00
babic_a 0441758d96 Change main writer to operate n_pulses instead of stop_pulse_id
The live writer has been changed to operate on the number of
desired pulses instead of the start and stop pulse_id. This
makes more sense in the context of live retrieval since the
stop pulse id has not been yet observed and the user would
need to calculate it anyway.
2020-07-20 14:55:37 +02:00
babic_a c12601f756 Add JFH5LiveWriter
In comparison with the normal writer, the block operations
have been dropped and the buffering of metadata is not sparse
anymore. Direct chunk writing used for data and buffering +
dump at end for metadata.
2020-07-20 14:49:43 +02:00
babic_a b9b8d74276 Add LiveImageAssembler
The difference between the normal and the LiveImageAssembler
is that the live one can process each image individually,
sacrificing storage performance for writing flexibility -
the user can decide live exactly which frames to write.
2020-07-20 13:52:32 +02:00