leonarski_fandClaude Opus 5.5
509a54ef92
So that rugnux can process a collection while it is written, the broker reports each data file once
a writer has closed it and renamed it into place:
- Writer: FileWriter takes a callback, called for every closed data file. StreamWriter uses it on TCP
to send an acknowledgement with ack_for = FILE_CLOSED (new TCPFrameType 10): file number and the
images the file holds. TCP protocol version 5. Files closed at END are reported before the END
acknowledgement.
- LiveCollection (image_pusher): the start message the writer of the master file received, the
closed files, the end. Fed by the TCP pusher (FILE_CLOSED acknowledgements; end once every writer
acknowledged END) and by the in-process HDF5 pusher (the FileWriter callback). The ZeroMQ pusher
has no back channel and offers none.
- Two routes, protected by the dataset's bearer tokens like the other dataset routes:
- GET /live/start.cbor: that start message;
- GET /live/events: server-sent events (collection, file, end, superseded) with the history first
and a keepalive every 10 s. One follower at a time; a second gets 409.
Documented in jfjoch_api.yaml; they add no schema, so the generated C++ model is unchanged.
- The HTTP server's thread pool is set explicitly (16, up to 64): an event stream holds a thread for
the whole collection.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
50 lines
1.9 KiB
C++
50 lines
1.9 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <future>
|
|
|
|
#include "ImagePusher.h"
|
|
#include "LiveCollection.h"
|
|
#include "../writer/FileWriter.h"
|
|
#include "../common/ThreadSafeFIFO.h"
|
|
#include "../common/ZMQWrappers.h"
|
|
#include "../common/Logger.h"
|
|
|
|
class HDF5FilePusher : public ImagePusher {
|
|
std::unique_ptr<FileWriter> writer;
|
|
std::mutex m;
|
|
std::future<void> writer_future;
|
|
ThreadSafeFIFO<ImagePusherQueueElement> writer_queue;
|
|
void WriterThread();
|
|
|
|
std::atomic<uint64_t> images_written = 0;
|
|
|
|
static constexpr uint32_t default_repub_watermark = 220;
|
|
static constexpr auto RepubTimeout = std::chrono::milliseconds(100);
|
|
|
|
std::unique_ptr<ZMQSocket> repub_socket;
|
|
bool repub_active = false;
|
|
Logger logger{"HDF5FilePusher"};
|
|
LiveCollection live;
|
|
public:
|
|
explicit HDF5FilePusher(const std::string &repub_address = "",
|
|
const std::optional<int32_t> &repub_watermark = {});
|
|
// Thread safety: StartDataCollection, EndDataCollection and SendCalibration must run poorly in serial context
|
|
// SendImage can be executed in parallel
|
|
void Preflight(StartMessage &message) override;
|
|
void StartDataCollection(StartMessage &message) override;
|
|
bool EndDataCollection(const EndMessage &message) override;
|
|
bool SendImage(const uint8_t *image_data, size_t image_size, int64_t image_number) override;
|
|
bool SendImage(ZeroCopyReturnValue &z) override;
|
|
bool SendCalibration(const CompressedImage &message) override;
|
|
LiveCollection *GetLiveCollection() override { return &live; }
|
|
|
|
std::string PrintSetup() const override;
|
|
|
|
std::optional<uint64_t> GetImagesWritten() const override;
|
|
size_t GetConnectedWriters() const override;
|
|
ImagePusherType GetType() const override { return ImagePusherType::HDF5; }
|
|
};
|