docs: a diagram of jfjoch_broker's data paths
docs/images/broker_pipeline_fig.py draws broker_pipeline.svg in the style of rugnux_pipeline_fig.py: the inputs (PSI detectors through the FPGA cards, a DECTRIS detector through its stream), the receiver chosen for each run and what it does (pedestal, FPGA, dark mask, Lite, rugnux rotation, rugnux stills, with the Lite fallback for a rotation run the image buffer cannot hold), the outputs (image buffer, pusher, jfjoch_writer, preview and metadata streams, REST) and the configuration keys that switch each path. Embedded at the top of JFJOCH_BROKER's "Online analysis" with a caption. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
This commit is contained in:
1 parent
2feef4b02b
commit
55416762d8
3 files changed
+308
No files matched your search
@@ -190,6 +190,12 @@ back to the broker, so with it the route answers 409.
|
||||
|
||||
## Online analysis
|
||||
|
||||

|
||||
|
||||
*What drives what in `jfjoch_broker`: the inputs on the left, the receiver chosen for each run in the middle, the
|
||||
outputs on the right, with the configuration keys that choose each path (`docs/images/broker_pipeline_fig.py`
|
||||
draws it).*
|
||||
|
||||
Without rugnux live mode, and always in the FPGA workflow, the broker analyses each image on its own: masking,
|
||||
summation, azimuthal and ROI integration, spot finding, and - with indexing on - indexing of the image and
|
||||
refinement of its geometry (`geom_refinement_algorithm`). A rotation collection is indexed image by image, as a serial
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="1600" height="1080" viewBox="0 0 1600 1080" font-family="Helvetica, Arial, sans-serif">
|
||||
<defs><marker id="arr" markerWidth="10" markerHeight="10" refX="8" refY="5" orient="auto" markerUnits="userSpaceOnUse"><path d="M0,0 L10,5 L0,10 z" fill="#555"/></marker></defs>
|
||||
<rect width="1600" height="1080" fill="white"/>
|
||||
<rect x="40" y="198" width="170" height="80" rx="12" ry="12" fill="white" stroke="#222222" stroke-width="3"/>
|
||||
<text x="125.0" y="230" text-anchor="middle" font-size="17" font-weight="bold" fill="#222222">PSI detector</text>
|
||||
<text x="125.0" y="256" text-anchor="middle" font-size="14" fill="#222222">JUNGFRAU, EIGER</text>
|
||||
<rect x="325" y="170" width="225" height="135" rx="12" ry="12" fill="white" stroke="#1A8A7D" stroke-width="3"/>
|
||||
<text x="437.5" y="202" text-anchor="middle" font-size="20" font-weight="bold" fill="#1A8A7D">FPGA cards</text>
|
||||
<text x="437.5" y="228" text-anchor="middle" font-size="14" fill="#222222">UDP packets off the wire;</text>
|
||||
<text x="437.5" y="247" text-anchor="middle" font-size="14" fill="#222222">conversion, summation,</text>
|
||||
<text x="437.5" y="266" text-anchor="middle" font-size="14" fill="#222222">spot finder, azimuthal, ROI</text>
|
||||
<rect x="488" y="277" width="52" height="20" rx="4" fill="#E8740C"/>
|
||||
<text x="514.0" y="292" text-anchor="middle" font-size="13" font-weight="bold" fill="white">FPGA</text>
|
||||
<path d="M210,238 L323,238" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<text x="267.5" y="228" text-anchor="middle" font-size="13" font-family="monospace" fill="#6B6B6B">detector.type</text>
|
||||
<text x="267.5" y="256" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">JUNGFRAU | EIGER</text>
|
||||
<rect x="40" y="645" width="170" height="80" rx="12" ry="12" fill="white" stroke="#222222" stroke-width="3"/>
|
||||
<text x="125.0" y="677" text-anchor="middle" font-size="17" font-weight="bold" fill="#222222">DECTRIS detector</text>
|
||||
<text x="125.0" y="703" text-anchor="middle" font-size="14" fill="#222222">SIMPLON API</text>
|
||||
<rect x="325" y="625" width="225" height="120" rx="12" ry="12" fill="white" stroke="#1A8A7D" stroke-width="3"/>
|
||||
<text x="437.5" y="657" text-anchor="middle" font-size="20" font-weight="bold" fill="#1A8A7D">Stream puller</text>
|
||||
<text x="437.5" y="683" text-anchor="middle" font-size="14" fill="#222222">ZeroMQ Stream2, CBOR:</text>
|
||||
<text x="437.5" y="702" text-anchor="middle" font-size="14" fill="#222222">images as sent by the detector</text>
|
||||
<path d="M210,685 L323,685" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<text x="267.5" y="675" text-anchor="middle" font-size="13" font-family="monospace" fill="#6B6B6B">detector.type</text>
|
||||
<text x="267.5" y="703" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">DECTRIS</text>
|
||||
<rect x="570" y="70" width="540" height="920" rx="18" fill="#EEF3FC" stroke="#B9CBEF" stroke-width="2"/>
|
||||
<text x="600" y="102" font-size="21" font-weight="bold" fill="#2F6FDE">Receiver<tspan dx="8" font-weight="normal" font-size="16" fill="#6B6B6B">(one per run, chosen at start)</tspan></text>
|
||||
<text x="1090" y="102" text-anchor="end" font-size="13" font-family="monospace" fill="#6B6B6B">receiver_threads</text>
|
||||
<rect x="600" y="125" width="480" height="125" rx="12" ry="12" fill="white" stroke="#2F6FDE" stroke-width="3"/>
|
||||
<text x="840.0" y="157" text-anchor="middle" font-size="20" font-weight="bold" fill="#2F6FDE">JFJochReceiverPedestal</text>
|
||||
<text x="840.0" y="179" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">pedestal run (G0, G1, G2)</text>
|
||||
<text x="840.0" y="201" text-anchor="middle" font-size="14" fill="#222222">raw frames into the pedestal and its RMS;</text>
|
||||
<text x="840.0" y="220" text-anchor="middle" font-size="14" fill="#222222">no images, nothing on any stream</text>
|
||||
<text x="612" y="237" font-size="12" font-family="monospace" fill="#6B6B6B">detector_settings.jungfrau_pedestal_*</text>
|
||||
<rect x="600" y="268" width="480" height="125" rx="12" ry="12" fill="white" stroke="#2F6FDE" stroke-width="3"/>
|
||||
<text x="840.0" y="300" text-anchor="middle" font-size="20" font-weight="bold" fill="#2F6FDE">JFJochReceiverFPGA</text>
|
||||
<text x="840.0" y="322" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">standard run, dark-mask sequence</text>
|
||||
<text x="840.0" y="344" text-anchor="middle" font-size="14" fill="#222222">image from the card's output, summation on the host</text>
|
||||
<text x="840.0" y="363" text-anchor="middle" font-size="14" fill="#222222">above the card's; each image indexed</text>
|
||||
<text x="612" y="380" font-size="12" font-family="monospace" fill="#6B6B6B">image_format.summation · azim_int.force_cpu</text>
|
||||
<rect x="1018" y="365" width="52" height="20" rx="4" fill="#2F6FDE"/>
|
||||
<text x="1044.0" y="380" text-anchor="middle" font-size="13" font-weight="bold" fill="white">CUDA</text>
|
||||
<rect x="958" y="365" width="52" height="20" rx="4" fill="#E8740C"/>
|
||||
<text x="984.0" y="380" text-anchor="middle" font-size="13" font-weight="bold" fill="white">FPGA</text>
|
||||
<rect x="600" y="411" width="480" height="125" rx="12" ry="12" fill="white" stroke="#2F6FDE" stroke-width="3"/>
|
||||
<text x="840.0" y="443" text-anchor="middle" font-size="20" font-weight="bold" fill="#2F6FDE">JFJochReceiverDarkMask</text>
|
||||
<text x="840.0" y="465" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">dark-mask sequence</text>
|
||||
<text x="840.0" y="487" text-anchor="middle" font-size="14" fill="#222222">dark frames into the mask;</text>
|
||||
<text x="840.0" y="506" text-anchor="middle" font-size="14" fill="#222222">nothing on any stream</text>
|
||||
<text x="612" y="523" font-size="12" font-family="monospace" fill="#6B6B6B">dark_mask</text>
|
||||
<rect x="600" y="554" width="480" height="125" rx="12" ry="12" fill="white" stroke="#2F6FDE" stroke-width="3"/>
|
||||
<text x="840.0" y="586" text-anchor="middle" font-size="20" font-weight="bold" fill="#2F6FDE">JFJochReceiverLite</text>
|
||||
<text x="840.0" y="608" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">rugnux live off, or a rotation run the image buffer cannot hold</text>
|
||||
<text x="840.0" y="630" text-anchor="middle" font-size="14" fill="#222222">each image on the GPU: masking, azimuthal and</text>
|
||||
<text x="840.0" y="649" text-anchor="middle" font-size="14" fill="#222222">ROI integration, spot finding, indexing</text>
|
||||
<text x="612" y="666" font-size="12" font-family="monospace" fill="#6B6B6B">indexing · spot_finding · azim_int</text>
|
||||
<rect x="1018" y="651" width="52" height="20" rx="4" fill="#2F6FDE"/>
|
||||
<text x="1044.0" y="666" text-anchor="middle" font-size="13" font-weight="bold" fill="white">CUDA</text>
|
||||
<rect x="600" y="697" width="480" height="125" rx="12" ry="12" fill="white" stroke="#2F6FDE" stroke-width="3"/>
|
||||
<text x="840.0" y="729" text-anchor="middle" font-size="20" font-weight="bold" fill="#2F6FDE">JFJochReceiverRugnuxRotation</text>
|
||||
<text x="840.0" y="751" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">rugnux live on, rotation that fits the image buffer</text>
|
||||
<text x="840.0" y="773" text-anchor="middle" font-size="14" fill="#222222">images forwarded as they come; rugnux reads them</text>
|
||||
<text x="840.0" y="792" text-anchor="middle" font-size="14" fill="#222222">back from the image buffer: the whole pipeline</text>
|
||||
<text x="612" y="809" font-size="12" font-family="monospace" fill="#6B6B6B">rugnux.enable · reserved_cpus · deadline_s</text>
|
||||
<rect x="1008" y="794" width="62" height="20" rx="4" fill="#1A8A7D"/>
|
||||
<text x="1039.0" y="809" text-anchor="middle" font-size="13" font-weight="bold" fill="white">rugnux</text>
|
||||
<rect x="948" y="794" width="52" height="20" rx="4" fill="#2F6FDE"/>
|
||||
<text x="974.0" y="809" text-anchor="middle" font-size="13" font-weight="bold" fill="white">CUDA</text>
|
||||
<rect x="600" y="840" width="480" height="125" rx="12" ry="12" fill="white" stroke="#2F6FDE" stroke-width="3"/>
|
||||
<text x="840.0" y="872" text-anchor="middle" font-size="20" font-weight="bold" fill="#2F6FDE">JFJochReceiverRugnuxStills</text>
|
||||
<text x="840.0" y="894" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">rugnux live on, serial (experimental)</text>
|
||||
<text x="840.0" y="916" text-anchor="middle" font-size="14" fill="#222222">rugnux pre-pass on the first frames; then every frame</text>
|
||||
<text x="840.0" y="935" text-anchor="middle" font-size="14" fill="#222222">analysed and integrated on the GPU, merged live</text>
|
||||
<text x="612" y="952" font-size="12" font-family="monospace" fill="#6B6B6B">rugnux.enable · bragg_integration</text>
|
||||
<rect x="1008" y="937" width="62" height="20" rx="4" fill="#1A8A7D"/>
|
||||
<text x="1039.0" y="952" text-anchor="middle" font-size="13" font-weight="bold" fill="white">rugnux</text>
|
||||
<rect x="948" y="937" width="52" height="20" rx="4" fill="#2F6FDE"/>
|
||||
<text x="974.0" y="952" text-anchor="middle" font-size="13" font-weight="bold" fill="white">CUDA</text>
|
||||
<path d="M550,238 L560,238 L560,187.5 L598,187.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M550,238 L560,238 L560,330.5 L598,330.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M550,685 L560,685 L560,473.5 L598,473.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M550,685 L560,685 L560,616.5 L598,616.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M550,685 L560,685 L560,759.5 L598,759.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M550,685 L560,685 L560,902.5 L598,902.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<rect x="1200" y="125" width="350" height="105" rx="12" ry="12" fill="white" stroke="#E8740C" stroke-width="3"/>
|
||||
<text x="1375.0" y="157" text-anchor="middle" font-size="20" font-weight="bold" fill="#E8740C">Image buffer</text>
|
||||
<text x="1375.0" y="183" text-anchor="middle" font-size="14" fill="#222222">compressed images, one slot each;</text>
|
||||
<text x="1375.0" y="202" text-anchor="middle" font-size="14" fill="#222222">rugnux rotation reads them back</text>
|
||||
<text x="1212" y="248" text-anchor="start" font-size="13" font-family="monospace" fill="#6B6B6B">image_buffer_MiB</text>
|
||||
<rect x="1200" y="280" width="350" height="105" rx="12" ry="12" fill="white" stroke="#E8740C" stroke-width="3"/>
|
||||
<text x="1375.0" y="312" text-anchor="middle" font-size="20" font-weight="bold" fill="#E8740C">Image pusher</text>
|
||||
<text x="1375.0" y="338" text-anchor="middle" font-size="14" fill="#222222">ZeroMQ PUSH or TCP/IP stream;</text>
|
||||
<text x="1375.0" y="357" text-anchor="middle" font-size="14" fill="#222222">HDF5 in-process, CBOR files</text>
|
||||
<text x="1212" y="403" text-anchor="start" font-size="13" font-family="monospace" fill="#6B6B6B">image_pusher</text>
|
||||
<rect x="1200" y="435" width="350" height="105" rx="12" ry="12" fill="white" stroke="#E8740C" stroke-width="3"/>
|
||||
<text x="1375.0" y="467" text-anchor="middle" font-size="20" font-weight="bold" fill="#E8740C">jfjoch_writer</text>
|
||||
<text x="1375.0" y="493" text-anchor="middle" font-size="14" fill="#222222">NXmx HDF5: master and data files;</text>
|
||||
<text x="1375.0" y="512" text-anchor="middle" font-size="14" fill="#222222">rugnux results beside the master</text>
|
||||
<text x="1212" y="558" text-anchor="start" font-size="13" font-family="monospace" fill="#6B6B6B">file_writer.format</text>
|
||||
<rect x="1200" y="620" width="350" height="95" rx="12" ry="12" fill="white" stroke="#E8740C" stroke-width="3"/>
|
||||
<text x="1375.0" y="652" text-anchor="middle" font-size="20" font-weight="bold" fill="#E8740C">Preview stream</text>
|
||||
<text x="1375.0" y="678" text-anchor="middle" font-size="14" fill="#222222">ZeroMQ PUB, 1 image/s</text>
|
||||
<text x="1212" y="733" text-anchor="start" font-size="13" font-family="monospace" fill="#6B6B6B">zeromq_preview</text>
|
||||
<rect x="1200" y="750" width="350" height="95" rx="12" ry="12" fill="white" stroke="#E8740C" stroke-width="3"/>
|
||||
<text x="1375.0" y="782" text-anchor="middle" font-size="20" font-weight="bold" fill="#E8740C">Metadata stream</text>
|
||||
<text x="1375.0" y="808" text-anchor="middle" font-size="14" fill="#222222">ZeroMQ PUB, bundles per period</text>
|
||||
<text x="1212" y="863" text-anchor="start" font-size="13" font-family="monospace" fill="#6B6B6B">zeromq_metadata</text>
|
||||
<rect x="1200" y="880" width="350" height="105" rx="12" ry="12" fill="white" stroke="#E8740C" stroke-width="3"/>
|
||||
<text x="1375.0" y="912" text-anchor="middle" font-size="20" font-weight="bold" fill="#E8740C">HTTP/REST, web frontend</text>
|
||||
<text x="1375.0" y="938" text-anchor="middle" font-size="14" fill="#222222">status, plots, scan result,</text>
|
||||
<text x="1375.0" y="957" text-anchor="middle" font-size="14" fill="#222222">image buffer, /rugnux/status</text>
|
||||
<path d="M1375.0,230 L1375.0,278" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M1375.0,385 L1375.0,433" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<text x="1538" y="262" text-anchor="end" font-size="13" fill="#6B6B6B" font-style="italic">START, images, END</text>
|
||||
<path d="M1080,330.5 L1148,330.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M1080,616.5 L1148,616.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M1080,759.5 L1148,759.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M1080,902.5 L1148,902.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M1150,177 L1150,932" fill="none" stroke="#555" stroke-width="2.2"/>
|
||||
<path d="M1150,177.5 L1198,177.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M1150,667.5 L1198,667.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M1150,797.5 L1198,797.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<path d="M1150,932.5 L1198,932.5" fill="none" stroke="#555" stroke-width="2.2" marker-end="url(#arr)"/>
|
||||
<text x="1375.0" y="113" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">images with their per-image results</text>
|
||||
<text x="1375.0" y="1007" text-anchor="middle" font-size="13" fill="#6B6B6B" font-style="italic">fed by every receiver; the pedestal and the mask arrive only here</text>
|
||||
<rect x="60" y="1008" width="52" height="20" rx="4" fill="#E8740C"/>
|
||||
<text x="86.0" y="1023" text-anchor="middle" font-size="13" font-weight="bold" fill="white">FPGA</text>
|
||||
<text x="124" y="1023" font-size="15" fill="#222222">computed on the FPGA card</text>
|
||||
<rect x="350" y="1008" width="52" height="20" rx="4" fill="#2F6FDE"/>
|
||||
<text x="376.0" y="1023" text-anchor="middle" font-size="13" font-weight="bold" fill="white">CUDA</text>
|
||||
<text x="414" y="1023" font-size="15" fill="#222222">GPU analysis where a card is present</text>
|
||||
<rect x="720" y="1008" width="62" height="20" rx="4" fill="#1A8A7D"/>
|
||||
<text x="751.0" y="1023" text-anchor="middle" font-size="13" font-weight="bold" fill="white">rugnux</text>
|
||||
<text x="794" y="1023" font-size="15" fill="#222222">the full rugnux pipeline, in the broker's process</text>
|
||||
<text x="60" y="1048" text-anchor="start" font-size="13" font-family="monospace" fill="#6B6B6B">monospace: keys of the broker configuration (jfjoch_settings) and of the dataset settings</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 15 KiB |
@@ -0,0 +1,165 @@
|
||||
#!/usr/bin/env python3
|
||||
"""jfjoch_broker data paths: inputs, the receiver chosen per run, outputs, and the settings that switch them."""
|
||||
|
||||
# Same style as rugnux_pipeline_fig.py; written next to this script as broker_pipeline.svg.
|
||||
|
||||
import os
|
||||
|
||||
W, H = 1600, 1080
|
||||
BLUE, TEAL, ORANGE, INK = "#2F6FDE", "#1A8A7D", "#E8740C", "#222222"
|
||||
PANEL_FILL, PANEL_STROKE = "#EEF3FC", "#B9CBEF"
|
||||
GPU_BG, FPGA_BG, RUGNUX_BG = "#2F6FDE", "#E8740C", "#1A8A7D"
|
||||
GREY = "#6B6B6B"
|
||||
|
||||
out = []
|
||||
def a(s): out.append(s)
|
||||
|
||||
def start():
|
||||
out.clear()
|
||||
a(f'<svg xmlns="http://www.w3.org/2000/svg" width="{W}" height="{H}" viewBox="0 0 {W} {H}" '
|
||||
f'font-family="Helvetica, Arial, sans-serif">')
|
||||
a('<defs>'
|
||||
'<marker id="arr" markerWidth="10" markerHeight="10" refX="8" refY="5" orient="auto" markerUnits="userSpaceOnUse">'
|
||||
'<path d="M0,0 L10,5 L0,10 z" fill="#555"/></marker>'
|
||||
'</defs>')
|
||||
a(f'<rect width="{W}" height="{H}" fill="white"/>')
|
||||
|
||||
def finish(name):
|
||||
a('</svg>')
|
||||
open(os.path.join(os.path.dirname(os.path.abspath(__file__)), name), "w").write("\n".join(out))
|
||||
|
||||
def badge(x, y, w, bg, text):
|
||||
a(f'<rect x="{x}" y="{y}" width="{w}" height="20" rx="4" fill="{bg}"/>')
|
||||
a(f'<text x="{x+w/2}" y="{y+15}" text-anchor="middle" font-size="13" font-weight="bold" fill="white">{text}</text>')
|
||||
|
||||
BADGES = [("rugnux", 62, RUGNUX_BG), ("CUDA", 52, GPU_BG), ("FPGA", 52, FPGA_BG)]
|
||||
|
||||
def box(x, y, w, h, title, body, color, rule=None, keys=None, gpu=False, fpga=False, rugnux=False, ts=20, bs=15):
|
||||
a(f'<rect x="{x}" y="{y}" width="{w}" height="{h}" rx="12" ry="12" fill="white" stroke="{color}" stroke-width="3"/>')
|
||||
a(f'<text x="{x+w/2}" y="{y+32}" text-anchor="middle" font-size="{ts}" font-weight="bold" fill="{color}">{title}</text>')
|
||||
by = y + 58
|
||||
if rule: # when the run gets this box
|
||||
a(f'<text x="{x+w/2}" y="{y+54}" text-anchor="middle" font-size="13" fill="{GREY}" font-style="italic">{rule}</text>')
|
||||
by = y + 76
|
||||
if body:
|
||||
lines = body if isinstance(body, list) else [body]
|
||||
for i, ln in enumerate(lines):
|
||||
a(f'<text x="{x+w/2}" y="{by+i*19}" text-anchor="middle" font-size="{bs}" fill="{INK}">{ln}</text>')
|
||||
if keys: # the settings that steer the box
|
||||
a(f'<text x="{x+12}" y="{y+h-13}" font-size="12" font-family="monospace" fill="{GREY}">{keys}</text>')
|
||||
bx = x + w - 10
|
||||
for (text, bw, bg), on in zip(BADGES, (rugnux, gpu, fpga)):
|
||||
if on:
|
||||
bx -= bw
|
||||
badge(bx, y+h-28, bw, bg, text)
|
||||
bx -= 8
|
||||
|
||||
def path(d, dashed=False):
|
||||
dash = ' stroke-dasharray="8,6"' if dashed else ''
|
||||
a(f'<path d="{d}" fill="none" stroke="#555" stroke-width="2.2"{dash} marker-end="url(#arr)"/>')
|
||||
|
||||
def label(x, y, text, anchor="middle", size=14):
|
||||
a(f'<text x="{x}" y="{y}" text-anchor="{anchor}" font-size="{size}" fill="{GREY}" font-style="italic">{text}</text>')
|
||||
|
||||
def key(x, y, text, anchor="middle"):
|
||||
a(f'<text x="{x}" y="{y}" text-anchor="{anchor}" font-size="13" font-family="monospace" fill="{GREY}">{text}</text>')
|
||||
|
||||
def panel(x0, x1, y0, y1, tx):
|
||||
a(f'<rect x="{x0}" y="{y0}" width="{x1-x0}" height="{y1-y0}" rx="18" fill="{PANEL_FILL}" stroke="{PANEL_STROKE}" stroke-width="2"/>')
|
||||
a(f'<text x="{tx}" y="{y0+32}" font-size="21" font-weight="bold" fill="{BLUE}">Receiver<tspan dx="8" font-weight="normal" font-size="16" fill="{GREY}">(one per run, chosen at start)</tspan></text>')
|
||||
|
||||
def legend():
|
||||
LY = H-58
|
||||
lx = 60
|
||||
for text, bw, bg, desc, step in [("FPGA", 52, FPGA_BG, "computed on the FPGA card", 290),
|
||||
("CUDA", 52, GPU_BG, "GPU analysis where a card is present", 370),
|
||||
("rugnux", 62, RUGNUX_BG, "the full rugnux pipeline, in the broker's process", 0)]:
|
||||
badge(lx, LY-14, bw, bg, text)
|
||||
a(f'<text x="{lx+bw+12}" y="{LY+1}" font-size="15" fill="{INK}">{desc}</text>')
|
||||
lx += step
|
||||
key(60, LY+26, "monospace: keys of the broker configuration (jfjoch_settings) and of the dataset settings", anchor="start")
|
||||
|
||||
# ---------- geometry
|
||||
IX, IW, IH = 40, 170, 80 # inputs
|
||||
TX, TW = 325, 225 # transport (cards, stream puller)
|
||||
PX0, PX1 = 570, 1110 # receiver panel
|
||||
RX, RW, RH = 600, 480, 125 # receiver boxes
|
||||
R = [125 + i*(RH+18) for i in range(6)] # receiver rows
|
||||
PY0, PY1 = 70, R[-1] + RH + 25
|
||||
BUSL, BUSR = 560, 1150 # arrow buses left and right of the panel
|
||||
OX, OW = 1200, 350 # outputs
|
||||
|
||||
|
||||
def broker():
|
||||
start()
|
||||
|
||||
# ---------- inputs and transport
|
||||
CY, SY = 170, 625 # cards and stream puller rows
|
||||
box(IX, CY+28, IW, IH, "PSI detector", "JUNGFRAU, EIGER", INK, ts=17, bs=14)
|
||||
box(TX, CY, TW, 135, "FPGA cards", ["UDP packets off the wire;", "conversion, summation,", "spot finder, azimuthal, ROI"],
|
||||
TEAL, fpga=True, bs=14)
|
||||
path(f"M{IX+IW},{CY+68} L{TX-2},{CY+68}")
|
||||
key((IX+IW+TX)/2, CY+58, "detector.type")
|
||||
label((IX+IW+TX)/2, CY+86, "JUNGFRAU | EIGER", size=13)
|
||||
|
||||
box(IX, SY+20, IW, IH, "DECTRIS detector", "SIMPLON API", INK, ts=17, bs=14)
|
||||
box(TX, SY, TW, 120, "Stream puller", ["ZeroMQ Stream2, CBOR:", "images as sent by the detector"], TEAL, bs=14)
|
||||
path(f"M{IX+IW},{SY+60} L{TX-2},{SY+60}")
|
||||
key((IX+IW+TX)/2, SY+50, "detector.type")
|
||||
label((IX+IW+TX)/2, SY+78, "DECTRIS", size=13)
|
||||
|
||||
# ---------- receiver panel
|
||||
panel(PX0, PX1, PY0, PY1, RX)
|
||||
key(PX1-20, PY0+32, "receiver_threads", anchor="end")
|
||||
box(RX, R[0], RW, RH, "JFJochReceiverPedestal", ["raw frames into the pedestal and its RMS;", "no images, nothing on any stream"], BLUE,
|
||||
rule="pedestal run (G0, G1, G2)", keys="detector_settings.jungfrau_pedestal_*", bs=14)
|
||||
box(RX, R[1], RW, RH, "JFJochReceiverFPGA", ["image from the card's output, summation on the host", "above the card's; each image indexed"], BLUE,
|
||||
rule="standard run, dark-mask sequence", keys="image_format.summation · azim_int.force_cpu", fpga=True, gpu=True, bs=14)
|
||||
box(RX, R[2], RW, RH, "JFJochReceiverDarkMask", ["dark frames into the mask;", "nothing on any stream"], BLUE,
|
||||
rule="dark-mask sequence", keys="dark_mask", bs=14)
|
||||
box(RX, R[3], RW, RH, "JFJochReceiverLite", ["each image on the GPU: masking, azimuthal and", "ROI integration, spot finding, indexing"], BLUE,
|
||||
rule="rugnux live off, or a rotation run the image buffer cannot hold", keys="indexing · spot_finding · azim_int", gpu=True, bs=14)
|
||||
box(RX, R[4], RW, RH, "JFJochReceiverRugnuxRotation", ["images forwarded as they come; rugnux reads them", "back from the image buffer: the whole pipeline"], BLUE,
|
||||
rule="rugnux live on, rotation that fits the image buffer", keys="rugnux.enable · reserved_cpus · deadline_s", rugnux=True, gpu=True, bs=14)
|
||||
box(RX, R[5], RW, RH, "JFJochReceiverRugnuxStills", ["rugnux pre-pass on the first frames; then every frame", "analysed and integrated on the GPU, merged live"], BLUE,
|
||||
rule="rugnux live on, serial (experimental)", keys="rugnux.enable · bragg_integration", rugnux=True, gpu=True, bs=14)
|
||||
|
||||
# cards feed the first two receivers, the stream puller the other four
|
||||
for r in R[:2]:
|
||||
path(f"M{TX+TW},{CY+68} L{BUSL},{CY+68} L{BUSL},{r+RH/2} L{RX-2},{r+RH/2}")
|
||||
for r in R[2:]:
|
||||
path(f"M{TX+TW},{SY+60} L{BUSL},{SY+60} L{BUSL},{r+RH/2} L{RX-2},{r+RH/2}")
|
||||
|
||||
# ---------- outputs
|
||||
O = {"buffer": 125, "pusher": 280, "writer": 435, "preview": 620, "metadata": 750, "rest": 880}
|
||||
box(OX, O["buffer"], OW, 105, "Image buffer", ["compressed images, one slot each;", "rugnux rotation reads them back"], ORANGE, bs=14)
|
||||
key(OX+12, O["buffer"]+105+18, "image_buffer_MiB", anchor="start")
|
||||
box(OX, O["pusher"], OW, 105, "Image pusher", ["ZeroMQ PUSH or TCP/IP stream;", "HDF5 in-process, CBOR files"], ORANGE, bs=14)
|
||||
key(OX+12, O["pusher"]+105+18, "image_pusher", anchor="start")
|
||||
box(OX, O["writer"], OW, 105, "jfjoch_writer", ["NXmx HDF5: master and data files;", "rugnux results beside the master"], ORANGE, bs=14)
|
||||
key(OX+12, O["writer"]+105+18, "file_writer.format", anchor="start")
|
||||
box(OX, O["preview"], OW, 95, "Preview stream", "ZeroMQ PUB, 1 image/s", ORANGE, bs=14)
|
||||
key(OX+12, O["preview"]+95+18, "zeromq_preview", anchor="start")
|
||||
box(OX, O["metadata"], OW, 95, "Metadata stream", "ZeroMQ PUB, bundles per period", ORANGE, bs=14)
|
||||
key(OX+12, O["metadata"]+95+18, "zeromq_metadata", anchor="start")
|
||||
box(OX, O["rest"], OW, 105, "HTTP/REST, web frontend", ["status, plots, scan result,", "image buffer, /rugnux/status"], ORANGE, bs=14)
|
||||
|
||||
path(f"M{OX+OW/2},{O['buffer']+105} L{OX+OW/2},{O['pusher']-2}")
|
||||
path(f"M{OX+OW/2},{O['pusher']+105} L{OX+OW/2},{O['writer']-2}")
|
||||
label(OX+OW-12, O["pusher"]-18, "START, images, END", anchor="end", size=13)
|
||||
|
||||
# the four receivers that make images feed the right bus; from it the buffer, the two PUB streams and the status
|
||||
for r in (R[1], R[3], R[4], R[5]):
|
||||
path(f"M{RX+RW},{r+RH/2} L{BUSR-2},{r+RH/2}")
|
||||
a(f'<path d="M{BUSR},{O["buffer"]+52} L{BUSR},{O["rest"]+52}" fill="none" stroke="#555" stroke-width="2.2"/>')
|
||||
for name, h in (("buffer", 105), ("preview", 95), ("metadata", 95), ("rest", 105)):
|
||||
path(f"M{BUSR},{O[name]+h/2} L{OX-2},{O[name]+h/2}")
|
||||
label(OX+OW/2, O["buffer"]-12, "images with their per-image results", size=13)
|
||||
label(OX+OW/2, O["rest"]+105+22, "fed by every receiver; the pedestal and the mask arrive only here", size=13)
|
||||
|
||||
legend()
|
||||
finish("broker_pipeline.svg")
|
||||
|
||||
|
||||
broker()
|
||||
print("ok")
|
||||
Reference in new issue
Block a user