diff --git a/docs/IMAGE_STREAM.md b/docs/IMAGE_STREAM.md index 1cddb6ef5..a74acc90d 100644 --- a/docs/IMAGE_STREAM.md +++ b/docs/IMAGE_STREAM.md @@ -11,6 +11,8 @@ These are: ## Image stream Images (with metadata) are serialized as CBOR [image message](CBOR.md#image-message). The stream will also include CBOR [start message](CBOR.md#start-message), [calibration messages](CBOR.md#calibration-message) and [end message](CBOR.md#end-message) with run metadata. +Calibration messages (the JUNGFRAU pedestal and pedestal RMS, with `save_calibration` on) come only from the +FPGA workflow; a DECTRIS stream carries no calibration, so none is forwarded for it. If `file_prefix` is not provided for a data collection, images won't be sent to image stream (or its HDF5/CBOR replacements). @@ -239,12 +241,19 @@ For this reason it is also recommended to use the same option on receiver side. Given PUB socket properties, it is possible to connect multiple viewers to a single socket --- all the viewers should receive all the images sent. +The dark frames of the mask sequence from a DECTRIS detector are no images for a viewer: nothing is sent on +this stream for them, not even a start or end message. + ## Metadata stream Jungfraujoch can also send pure metadata for the purpose of archiving such information. Metadata are serialized as CBOR [metadata message](CBOR.md#metadata-message). This is very similar to the image message, but excludes the image array, the spot list and the reflection list. -As metadata are relatively small, to avoid large number of messages, Jungfraujoch bundles metadata of many images in one message. -Order of images within bundle, as well as the size of the bundle, are not guaranteed. +As metadata are relatively small, to avoid large number of messages, Jungfraujoch bundles metadata of many images in one message: +a bundle is cut every `period_ms` of the `zeromq_metadata` configuration and holds the images analysed since the last one. +Order of images within bundle, as well as the size of the bundle, are not guaranteed, and two bundles cut +close together can leave in either order; the end message is always last. The stream will also include CBOR [start message](CBOR.md#start-message) and [end message](CBOR.md#end-message) with run metadata. +A disabled metadata stream keeps nothing, and the dark frames of the mask sequence from a DECTRIS detector +send nothing on it, as on the preview stream. This is using PUB socket with watermark, so there is some queuing of messages with ZeroMQ. Multiple receivers can be connected. \ No newline at end of file diff --git a/docs/JFJOCH_BROKER.md b/docs/JFJOCH_BROKER.md index 6424b5404..d1313fd37 100644 --- a/docs/JFJOCH_BROKER.md +++ b/docs/JFJOCH_BROKER.md @@ -214,19 +214,42 @@ rugnux processed the data: by rugnux live mode (below) during the collection, or `rotation_indexing_angular_stride_deg` and `spot_finding_settings.quick_integration` are deprecated: accepted, ignored, and returned at their old defaults. `bragg_integration_settings` apply to rugnux live mode. +### The receiver of a run + +Each run is taken by one receiver, chosen when it starts from the detector type, the detector mode, the +goniometer and the rugnux live setting: + +| Input | Run | Receiver | What it does | +|---|---|---|---| +| FPGA cards (PSI JUNGFRAU, PSI EIGER) | every run | `JFJochReceiverFPGA` | the card's converted pixels, spots, azimuthal and ROI sums into the image; summation on the host above what the card sums; indexing of each image; the calibration to the writer; pedestal runs into the pedestal and its RMS; with rugnux live on, a serial run analysed on the GPU after a rugnux pre-pass (see [Serial collections](#serial-collections-experimental)) | +| DECTRIS stream | dark-mask sequence | `JFJochReceiverDarkMask` | the dark frames into the mask; nothing on any stream | +| DECTRIS stream | rugnux live off, or a rotation run the image buffer cannot hold | `JFJochReceiverLite` | each image on the GPU: masking, azimuthal and ROI integration, spot finding, indexing | +| DECTRIS stream | rugnux live on, rotation that fits the image buffer | `JFJochReceiverRugnuxRotation` | images forwarded as they come; rugnux reads the collection back from the image buffer and runs the whole pipeline | +| DECTRIS stream | rugnux live on, serial | `JFJochReceiverRugnuxStills` | a rugnux pre-pass on the first frames, then every frame analysed and integrated on the GPU and merged live | + +`receiver_threads` is the number of image threads. The Lite receiver uses at most 16 of them for a +collection at 100 Hz or slower or of fewer than 256 images (each thread holds a GPU analysis of its +own); the FPGA receiver takes four when it sums images on the host. The GPU FFT indexers and cuFFT +plans the broker's own indexing uses are built once and kept for the life of the process, and the ones +built for earlier indexing settings are released when the settings change. `azim_int.force_cpu` +moves the azimuthal integration of the FPGA workflow from the card to the host, which a profile with +more bins than the card holds needs. + ## Rugnux live mode -With rugnux live mode on, rotation collections from a DECTRIS detector are processed by rugnux inside -the broker while they arrive, and serial (stills) collections in the FPGA workflow are analysed on the GPU -with the geometry and mask a rugnux pre-pass decided (see [Serial collections](#serial-collections-experimental) -below). It is off by default; the configuration file switches it on with +With rugnux live mode on, collections from a DECTRIS detector are processed inside the broker while they +arrive: rotation collections by rugnux, and serial (stills) collections on the GPU with the geometry and mask a +rugnux pre-pass decided (see [Serial collections](#serial-collections-experimental) below). In the FPGA workflow +(PSI JUNGFRAU and EIGER) serial collections are analysed on the GPU after a pre-pass as well, with the differences +given there; rotation collections run image by image, as with the mode off. It is off by default; the +configuration file switches it on with `"rugnux": {"enable": true, "reserved_cpus": 4}`, where `reserved_cpus` is the number of CPU threads left to the receiver and the image pusher. For a rotation collection, rugnux reads the images back from the broker's image buffer, so the buffer must hold the whole -collection - one slot per image. Where it holds fewer, the collection is only forwarded to the writer, -as with the mode off; the live status reports it as skipped and the collection's statistics carry a -warning saying how many images the buffer holds against how many the collection has. +collection - one slot per image. Where it holds fewer, the collection gets the per-image analysis of +the mode off (spots, indexing, scan result); the live status reports it as skipped and the collection's +statistics carry a warning saying how many images the buffer holds against how many the collection has. The results - the merged data, the map coefficients and, with a model, the placed model and its refinement - travel to the writer in the end message, which writes them beside the master file as @@ -251,7 +274,8 @@ they prepare the next collection and expose nothing of the current one. The end message, and with it the end of the collection, waits for the analysis. `deadline_s` (default 60 s; 0 for no cutoff) bounds that wait: this long after the last image the analysis is cut. `POST /cancel` is the other way to cut it: while images still arrive it cancels the collection as with the mode off, -and the analysis with it; after the last image it ends only the analysis. A cut analysis gives what it +and the analysis with it - the live status then ends with verdict `CANCELLED` and the images that came before +the cancel; after the last image it ends only the analysis. A cut analysis gives what it has established: a pass that is past its images still merges and makes its outputs, the canonical pass cut among its images merges those it has read, only the placed model is left out, and any other pass cut short gives way to the last one that ran to its end. The end message goes out at @@ -277,9 +301,9 @@ estimate and background of every image in `GET /result/scan` and the master file ### Serial collections (experimental) Live processing of serial collections is **experimental**. With rugnux live mode on, a serial -(non-rotation) collection in the FPGA workflow, in the standard detector mode, is analysed as frames from a -DECTRIS detector are: the converted frames go to GPU spot finding, indexing and integration, with rugnux's -stills spot finder (adaptive threshold, single-pixel spots kept). Before any frame is analysed, the first +(non-rotation) collection from a DECTRIS detector, in the standard detector mode, is analysed frame by frame: the +frames as the stream delivers them go to GPU spot finding, indexing and integration, with rugnux's stills spot +finder (adaptive threshold, single-pixel spots kept). Before any frame is analysed, the first 500 frames are run through rugnux, in memory, as `rugnux --no-merge` would run on them. That pre-pass decides the beam centre, refines the detector distance (where the dataset settings give a unit cell) and masks the beam-stop shadow and defective pixels; every frame of the collection is then analysed with that @@ -288,14 +312,15 @@ the centre and distance used (`refined_beam_center_x/y`, `refined_detector_dista [CBOR](CBOR.md)). The broker's own distance is taken as the beamline's: unlike offline rugnux, the pre-pass does not search the whole distance range. It refines the centre and distance for a detector square to the beam, as the frames are then analysed. At a serial hit rate the distance needs the 500 frames to be refined at -all: on fewer, too few index for the geometry refinement to run, and the stated distance is kept. The image -buffer has to hold them, and the frames that arrive while the pre-pass runs, in half its slots (for a JUNGFRAU -8M, a slot is the size of an uncompressed frame); a smaller buffer starts the pre-pass on as many as it holds. +all: on fewer, too few index for the geometry refinement to run, and the stated distance is kept. The pre-pass +holds and fits on at most half the slots of the image buffer (a slot is the size of an uncompressed +frame), so a smaller buffer starts it on as many as that. Where a unit cell is given and those frames are too few for the geometry fit (200 indexed frames), frames go on being held and the pre-pass runs again on all of them each time they have doubled, until the fit has its frames, half -the image buffer is full, or the collection ends. The analysis starts when the pre-pass ends and catches up from the -frames held, so it starts later on a low hit rate; a frame finding the buffer full meanwhile is dropped as usual. +the image buffer or 2000 frames are held, or the collection ends; frames arriving during a run do not make the next +one larger than that, so the last run stays short. The analysis starts when the pre-pass ends and catches up from the +frames waiting, so it starts later on a low hit rate. The geometry is fixed when the pre-pass ends and does not change during the collection, so every frame is analysed and merged at it and the radial profile has one binning throughout. Where the pre-pass could not refine it (too few @@ -303,9 +328,10 @@ of its frames indexed, as with a small image buffer, a low hit rate or a chip wh cell given), the distance in the header is used for the whole run: this is a warning in the log and a note of the live status, and a wrong header distance then costs resolution and indexed frames. -Frames arriving during the pre-pass, or faster than the analysis, wait in the image buffer, in at most half of -its slots; a frame that finds no free slot is dropped and counted as for a full image buffer, so analysis -slower than the frame rate costs frames, not memory. A pre-pass that fails leaves the stated geometry and +Frames arriving during the pre-pass, or faster than the analysis, wait in the image buffer, in all its slots but +one per analysis thread, kept for sending; a frame that finds it full is dropped and counted as for a full image +buffer, so analysis slower than the frame rate costs frames, not memory. Where the writer lags behind, the analysis +waits for slots to come back. A pre-pass that fails leaves the stated geometry and mask in use. Where the start message cannot be sent after the pre-pass (no writer listening) or the analysis cannot be set up, the collection ends with that error, reported by `/status` and `/wait_till_done`, and the next `/start` runs as usual. @@ -323,11 +349,44 @@ JUNGFRAU 8M, against about half a minute for the same merge made from nothing. T writer in the end message, as for rotation (`.mtz`), and the run's step "Merge" carries the final statistics. -`GET /rugnux/status` reports such a run as three steps (pre-pass, GPU analysis, merge) and a `stills` block: the +With an uploaded model, the model step runs on every one of these merges, as `rugnux --mode scale --model` +places a model and `rugnux --mode refine` refines it: the model is placed by its rigid body where it fits the +data as given - its indexing ambiguity probed by R-free, and the merge put in the model's indexing - and by +molecular replacement where it does not (or is a prediction); the 2mFo-DFc and mFo-DFc maps are made, and the +placed model is refined, with the blobs of its difference map. Without a space group in the dataset settings +the frames are merged in the model's own, where the model's cell is the consensus cell of the run on the same +axes; otherwise the model is left out, with a warning in the live status saying why. The step on a provisional +merge runs on a thread of its own, on 8 CPU threads and the GPU, beside the analysis and the merge, which do not +wait for it: its result - the placed and the refined model, their maps, the scores - goes to the live status +(`model`, `/rugnux/model.cif`, `/rugnux/map.ccp4`) as soon as it is made, so the density can be looked at while +the collection goes on. A step still running when the next merge is made is cut (a refinement cut short keeps +the weight trials it finished), and each step starts from the placement of the one before: the merge is put in +the same indexing, and the model goes in where it was placed, with no null, indexing probe or search made again. +After the last frame the step on the merge of the collection runs on every thread, and the end message waits +for it until `deadline_s` after the last image; then it is cut as for rotation - the refinement first (cut +short, or left out where it has not started), and a molecular replacement that has not started. The end message +carries the merge in the model's indexing (`.mtz`), the map coefficients, the placed and the +refined model, the refined maps and the blobs, written as for rotation; one that would not fit the pushers' +buffer goes without them. The GPU memory the analysis's indexers hold for the broker's life is counted against +the card when the model step decides what goes on it. `POST /cancel` cuts it +the same way, at any time (`cut` in `/rugnux/status`): the frames that came are still analysed and merged. + +The frames wait in the image buffer compressed, and go on to the writer byte for byte, with their analysis. `efficiency` is the +images received over the images expected; a stream has no module packets, so per-image collection +efficiency, ADU histograms and pulse IDs are absent. The dark frames of the mask sequence stay outside this mode. + +`GET /rugnux/status` reports such a run as three steps (pre-pass, GPU analysis, merge; with a model, the steps of +the model on the merge of the collection after them) and a `stills` block: the pre-pass state and duration, the stated and decided beam centre and distance, the pixels masked, and the images analysed and indexed with the analysis rate. The "Rugnux live" page of the web frontend shows the same. `tools/jfjoch_stills_gpu_replay` replays a stored serial collection through the same analysis. +In the FPGA workflow (JUNGFRAU), a serial collection with rugnux live mode on is analysed by the receiver of the +card (`JFJochReceiverFPGA`) with the pre-pass, the GPU analysis and the live merge described above, with three +differences: the converted frames wait for the analysis in at most half of the image buffer's slots, a frame that +finds them taken is dropped, and there is no model step - an uploaded model is not placed or refined on these +merges. The end message carries the merged data, as for a DECTRIS stream. + ## Setting up a local test for Jungfraujoch For development, it is possible to set up a local installation of Jungfraujoch. This will work without FPGA installed in the computer and allows testing the Jungfraujoch software layer, including