Docs: broker example config starts with the current schema; TCP stream protocol as implemented

The JFJOCH_BROKER.md example now parses and starts: one geometry per detector, EIGER trim energies alongside the trim files, readout_time_ns, the required azim_int and image_format keys, the pedestal RMS limit under image_format where it is read. module_placement entries are described with all eight required fields. IMAGE_STREAM.md gives TCP protocol version 5, the FILE_CLOSED ack, the 64-bit FIFO occupancy, the liveness-based END wait and the two optional tcp timeouts; the metadata stream also drops reflections. SECURITY.md and the broker page list the bearer-protected /rugnux/model.cif and /rugnux/map.ccp4. The etc/ examples drop the deprecated numa_policy and a misplaced pedestal RMS limit. Co-Authored-By: Claude Fable <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
2026-11-10 12:53:21 +01:00
1 parent bb4f64a010
commit ce78662e97
@@ -70,7 +70,7 @@ If a `START` ACK fails on any connection, Jungfraujoch sends `CANCEL` to all alr

 For each frame:
 1. Read one `TcpFrameHeader` (fixed size, 64-byte aligned).
-2. Validate `magic` (`0x4A464A54` / `"JFJT"`) and `version` (`4`). Both ends reject a frame of any other version, so writer and broker must be of the same release.
+2. Validate `magic` (`0x4A464A54` / `"JFJT"`) and `version` (`5`). Both ends reject a frame of any other version, so writer and broker must be of the same release.
 3. Read `payload_size` bytes (if non-zero).

 #### Pre-flight
@@ -102,13 +102,15 @@ TCP/IP image stream is configured in the broker JSON configuration file under th
 - `image_socket`: listen address in `tcp://<IP>:<port>` format. `0.0.0.0` binds on all interfaces. `*` as port selects a random free port.
 - `nwriters` (optional): maximum number of simultaneous writer connections accepted.
 - `send_buffer_size` (optional): OS-level `SO_SNDBUF` size for accepted connections.
+- `peer_liveness_timeout_ms` (optional, default 15000): how long a send may block with no sign of life from the writer (no `BUSY` heartbeat, no ACK) before the connection is declared dead.
+- `max_backpressure_timeout_ms` (optional, default 60000): hard bound on backpressure - if no bytes can be sent for this long the connection is declared dead even while the writer keeps heartbeating.

 #### ACK handling

 ACK handling is mandatory for correct operation:
 - `PREFLIGHT` **must** be acknowledged (`ack_for=PREFLIGHT`) on each connection within 5 seconds, otherwise the collection is not started. A rejected pre-flight (`OK` clear) carries the reason as error text and does not break the connection.
 - `START` **must** be acknowledged (`ACK` with `ack_for=START`) on each connection within 5 seconds, otherwise collection start fails and a rollback is triggered.
- `END` **must** be acknowledged (`ack_for=END`) on each connection within 10 seconds for successful completion.
+- `END` **must** be acknowledged (`ack_for=END`) on each connection for successful completion. There is no fixed deadline: the writer may still be flushing a backlog, so the broker waits as long as the writer shows signs of life (`BUSY` heartbeats, `DATA` ACKs) and gives up only after `peer_liveness_timeout_ms` (default 15 s) of silence.
 - `CANCEL` should be acknowledged during rollback paths (500ms timeout).
 - `DATA` should be acknowledged for every frame. A `DATA` ACK with `FATAL` flag set reports a downstream error (e.g. disk full) which is propagated to `jfjoch_broker` via `Finalize()`. A failed `DATA` ACK does **not** break the TCP connection on its own — data continues to flow.
 - `CALIBRATION` is not acknowledged at this time.
@@ -135,13 +137,14 @@ On Linux, large payload transmission (`DATA` and `CALIBRATION` frames) can use k
 | 7 | `KEEPALIVE` | Connection liveness probe/pong |
 | 8 | `BUSY` | Writer alive but stalled; carries its FIFO occupancy |
 | 9 | `PREFLIGHT` | Dry run before a collection starts: can this run be written? |
+| 10 | `FILE_CLOSED` | Not a frame of its own: the value of `ack_for` in a writer `ACK` reporting that a data file was closed and renamed into place (`image_number` = file number from 0, `ack_processed_images` = one past the highest image number it holds). The broker uses it to tell a reader following the collection which files it can read. |

 #### TCP frame header (`TcpFrameHeader`)

 | Field                    | Type | Description                                              |
 |--------------------------|---|----------------------------------------------------------|
 | `magic`                  | `uint32_t` | Protocol magic (`0x4A464A54`, `"JFJT"`)                  |
-| `version`                | `uint16_t` | Protocol version (`4`)                                   |
+| `version`                | `uint16_t` | Protocol version (`5`)                                   |
 | `type`                   | `uint16_t` | Frame type (see table above)                             |
 | `image_number`           | `uint64_t` | Image index for `DATA` frames                            |
 | `payload_size`           | `uint64_t` | Number of payload bytes after header                     |
@@ -151,7 +154,7 @@ On Linux, large payload transmission (`DATA` and `CALIBRATION` frames) can use k
 | `ack_processed_images`   | `uint32_t` | In `ACK`: number of images processed by receiver         |
 | `ack_code`               | `uint16_t` | In `ACK`: error/status code                              |
 | `ack_for`                | `uint16_t` | In `ACK`: frame type being acknowledged                  |
-| `ack_fifo_occupancy`     | `uint16_t` | In `ACK`: occupancy of input FIFO in the `jfjoch_writer` |
+| `ack_fifo_occupancy`     | `uint64_t` | In `ACK`: occupancy of input FIFO in the `jfjoch_writer` |
 | `ack_fifo_max_occupancy` | `uint64_t` | In `ACK`: max occupancy of input FIFO                    |

 The header is 64-byte aligned (`alignas(64)`).
@@ -239,7 +242,7 @@ Given PUB socket properties, it is possible to connect multiple viewers to a sin
 ## Metadata stream
 Jungfraujoch can also send pure metadata for the purpose of archiving such information.
 Metadata are serialized as CBOR [metadata message](CBOR.md#metadata-message).
-This is very similar to the image message, but excludes the actual image array and spot positions.
+This is very similar to the image message, but excludes the image array, the spot list and the reflection list.
 As metadata are relatively small, to avoid large number of messages, Jungfraujoch bundles metadata of many images in one message.
 Order of images within bundle, as well as the size of the bundle, are not guaranteed.
 The stream will also include CBOR [start message](CBOR.md#start-message) and [end message](CBOR.md#end-message) with run metadata.
@@ -27,10 +27,12 @@ and answer `401` otherwise; `/statistics` omits its `measurement` block instead.

 `/start` also takes an optional `module_placement`: where the JUNGFRAU modules lie, as `rugnux`
 measures them on an earlier run. rugnux writes it as `<prefix>_module_placement.json`, a fragment of the
-dataset settings (`{"module_placement": [...]}`) to merge into the `/start` body. Each entry names its
-module as `custom_geometry` does - `x0`, `y0` (the image position of its readout pixel 0) and
-`fast_axis`, `slow_axis` - but in the image as stored, after `mirror_y` (see below),
-so the order does not matter, and a list that does not name exactly the detector's modules (one left out,
+dataset settings (`{"module_placement": [...]}`) to merge into the `/start` body. Each entry has eight
+required fields: it names its module as `custom_geometry` does - `x0`, `y0` (the image position of its
+readout pixel 0) and `fast_axis`, `slow_axis` - but in the image as stored, after `mirror_y` (see below),
+and gives where the module lies on the detector plane (`x0_plane`, `y0_plane`, `fast_step`, `slow_step`).
+The first four identify the module, so the order does not matter, and a list that does not name exactly
+the detector's modules (one left out,
 named twice, or not the detector's) is refused with `400`. The module count and the assembly are the
 configuration's (`custom_geometry` below); the placement only moves the modules relative to one another,
 and goes into the start message, the master file's `module_NN` groups and the broker's analysis. See
@@ -47,7 +49,9 @@ on the two differ by the mirror: for an image of height H, `y0` (module_placemen
 (custom_geometry), `x0` is the same, and `Yp` and `Yn` swap in `fast_axis`/`slow_axis`. With `mirror_y`
 off they are equal.

-Example configuration (not every section is shown):
+Example configuration (`etc/broker_local.json`, `broker_eiger.json` and `broker_crmx.json` are
+complete, runnable configurations; a detector takes either `standard_geometry` or `custom_geometry`,
+never both):

 ```json
 {
@@ -87,24 +91,22 @@ Example configuration (not every section is shown):
      "module_sync": true,
      "sensor_thickness_um": 320,
      "calibration_file": [
-        "gainMaps.bin"
+        "/opt/jfjoch/eiger/trim/e1m-01.sn001",
+        "/opt/jfjoch/eiger/trim/e1m-02.sn002"
+      ],
+      "trim_energies_eV": [
+        12400
      ],
      "hostname": [
        "e1m-01",
        "e1m-02"
      ],
-      "readout_time_us": 3,
+      "readout_time_ns": 3000,
      "sensor_material": "Si",
      "tx_delay": [
        0,1
      ],
      "base_data_ipv4_address": "10.10.10.50",
-      "standard_geometry": {
-        "nmodules": 1,
-        "gap_x": 8,
-        "gap_y": 36,
-        "modules_in_row": 1
-      },
      "custom_geometry": [
        {
          "x0": 0,
@@ -127,7 +129,6 @@ Example configuration (not every section is shown):
    "jungfrau_pedestal_g0_frames": 2000,
    "jungfrau_pedestal_g1_frames": 300,
    "jungfrau_pedestal_g2_frames": 300,
-    "jungfrau_pedestal_g0_rms_limit": 100,
    "jungfrau_pedestal_min_image_count": 128,
    "jungfrau_storage_cell_count": 1,
    "jungfrau_storage_cell_delay_ns": 5000,
@@ -135,11 +136,10 @@ Example configuration (not every section is shown):
    "jungfrau_use_gain_hg0": false
  },
  "azim_int": {
-    "polarization_factor": -1,
+    "polarization_corr": true,
    "solid_angle_corr": true,
-    "high_q_recipA": 0,
-    "low_q_recipA": 0,
-    "q_spacing": 0
+    "low_q_recipA": 0.01,
+    "q_spacing": 0.002
  },
  "image_format": {
    "summation": true,
@@ -149,7 +149,9 @@ Example configuration (not every section is shown):
    "bit_depth_image": 16,
    "signed_output": true,
    "mask_module_edges": true,
-    "mask_chip_edges": true
+    "mask_chip_edges": true,
+    "apply_mask": true,
+    "jungfrau_pedestal_g0_rms_limit": 100
  },
  "image_buffer_MiB": 2048,
  "receiver_threads": 64,
@@ -168,6 +170,13 @@ Example configuration (not every section is shown):
 }
 ```

+`standard_geometry` (`nmodules`, `modules_in_row`, `gap_x`, `gap_y`) is the alternative for a regular
+module grid. For an EIGER `hostname` lists two entries per module (one per half-module) and
+`calibration_file` is either a directory of trim files or one trim file per half-module, with the energies
+they were trimmed at in `trim_energies_eV`; for a JUNGFRAU both are one entry per module, the
+calibration being the gain maps. The polarization factor of the azimuthal integration is a dataset
+setting (`/config/dataset`, `polarization_factor`), not part of `azim_int`.
+
 ## Following a collection

 A reader can process a collection while it is being written - rugnux does, given the path the master
@@ -255,7 +264,7 @@ R-free, the bond and angle rmsd, the time it took and the unexplained blobs of t
 state, the images received, the step it is on and how far into it, the steps so far with what each
 found, and once it is over the verdict, the summary, warnings and the time from the last image to
 every result written. The web frontend shows it on the dashboard and in its "Rugnux live" tab, which
-also changes the settings and uploads the model. It needs the dataset's bearer token
+also changes the settings and uploads the model. It, `/rugnux/model.cif` and `/rugnux/map.ccp4` need the dataset's bearer token
 like the dataset routes (`/statistics/data_collection`, `/image_buffer/*`, `/preview/plot*`). `GET /result/scan` carries the rotation lattice, unit cell and Bravais symbol
 rugnux found. A collection in which rugnux finds no lattice still gets the spot count, resolution
 estimate and background of every image in `GET /result/scan` and the master file, marked as not indexed.
@@ -338,7 +347,7 @@ The `jfjoch` package alone is enough.
 In this case - it is necessary to update `etc/broker_local.json` file with frontend path in `/usr/share/jfjoch/frontend`.

 ### Start service
-Start broker:
+Start broker (the HTTP port is optional; the default is 5232, listening on all interfaces):
 ```
 cd build/broker
 ./jfjoch_broker ../../etc/broker_local.json 5232
@@ -113,6 +113,7 @@ the record of its collection - so the broker is left as open and as empty as a f
 | `/preview/plot`, `/preview/plot.bin` (per-image plots, unit cell) | 401 without a token |
 | `/live/events` (the start message and the data files of the collection being written) | 401 without a token |
 | `/rugnux/status` (live rugnux run of the collection: cell, space group, results) | 401 without a token |
+| `/rugnux/model.cif`, `/rugnux/map.ccp4` (the model placed by the live run and its maps) | 401 without a token |
 | `/statistics` (the aggregate the web UI polls) | 200, but the `measurement` block is omitted |
 | everything else (`/status`, `/config/*`, `/start`, `/cancel`, masks, pedestal, ...) | open |

@@ -79,7 +79,6 @@
  ],
  "frontend_directory": "/usr/share/jfjoch/frontend/",
  "image_pusher": "ZeroMQ",
-  "numa_policy": "n2g2",
  "receiver_threads": 64,
  "image_buffer_MiB": 4096,
  "pcie": [
@@ -94,7 +93,6 @@
    "jungfrau_pedestal_g0_frames": 2000,
    "jungfrau_pedestal_g1_frames": 300,
    "jungfrau_pedestal_g2_frames": 300,
-    "jungfrau_pedestal_g0_rms_limit": 100,
    "jungfrau_pedestal_min_image_count": 128,
    "jungfrau_storage_cell_count": 1,
    "jungfrau_storage_cell_delay_ns": 2100,
@@ -68,7 +68,6 @@
  ],
  "frontend_directory": "/usr/share/jfjoch/frontend/",
  "image_pusher": "ZeroMQ",
-  "numa_policy": "n2g2",
  "receiver_threads": 64,
  "image_buffer_MiB": 4096,
  "pcie": [
@@ -88,7 +88,6 @@
    "jungfrau_pedestal_g0_frames": 0,
    "jungfrau_pedestal_g1_frames": 0,
    "jungfrau_pedestal_g2_frames": 0,
-    "jungfrau_pedestal_g0_rms_limit": 100,
    "jungfrau_pedestal_min_image_count": 128,
    "jungfrau_storage_cell_count": 1,
    "jungfrau_storage_cell_delay_ns": 2100,