docs: add documentation for event data

docs/source/developer/data_access/data_access.md

@@ -0,0 +1,12 @@
+(developer.data_access)=
+# Data Access
+This section provides information on how to access data in BEC. It is recommended to first read the [architecture section](developer.architecture) to understand the general messaging system through REDIS and the purpose of individual services. 
+Information how to access event based data in BEC can be found in the [event data section](developer.event_data).
+
+```{toctree}
+---
+maxdepth: 2
+hidden: true
+---
+event_data/
+```

docs/source/developer/data_access/event_data.md

@@ -0,0 +1,92 @@
+(developer.event_data)=
+# Event Data
+Understanding the event system in BEC is crucial to benefit from the full capabilities of the system. BEC services are communicating with each other through messages on REDIS. A brief introduction to this concept is given in the previous section (TODO: link to previous section). 
+Information is exchanged through various messages, which are published on specific endpoints in REDIS. 
+Any service (of BEC) can be configured to act upon receiving updates on these endpoints. This is a powerful feature, as it allows you to react to any changes in the system and to trigger custom actions based on the events.
+To give a more specific example, we can look at the preparation for a scan. The *ScanServer* will publish a [*ScanStatusMessage*](/api_reference/_autosummary/bec_lib.messages.ScanStatusMessage) which informs all other services about an upcoming scan. The *ScanStatusMessage* has also relevant information about the scan, which allows the *DeviceServer* to forward this information to all devices. In the end, each device can follow a customised sequence of actions to prepare itself for the upcoming scan based on the *ScanStatusMessage*.
+In the following, we will show how to access commonly used event types in BEC and how to subscribe to them.
+
+## Commonly used event types
+To access the information of the endpoints, you have to be aware of the allowed message operations and the message types for the given endpoint. How to access this information is explained in the message introduction section (TODO: link to message introduction).
+All message types inherit from the [`BECMessage`](/api_reference/_autosummary/bec_lib.messages.BECMessage) class, which ensures that a *content* and *metadata* field exists. Depending on the message type, the content and metadata can be different.
+Below, we give a few examples of commonly used endpoints and how to access them from within the *BECIPythonClient*.
+
+**Scan Status**
+The [`scan_status`](/api_reference/_autosummary/bec_lib.endpoints.MessageEndpoints.scan_status) endpoint allows you to access information about the current/upcoming scan in BEC. We can check the status of the scan, the scan_ID, the scan_type, scan_args and scan_kwargs, as well as information about readoutPriority of devices, i.e. baseline, monitored, async etc. 
+To access this information, we have to check first the message type and allowed operations for this endpoing.
+``` ipython
+[52/371] ❯❯ MessageEndpoints.scan_status().message_type
+Out[52]: bec_lib.messages.ScanStatusMessage
+[53/371] ❯❯ MessageEndpoints.scan_status().message_op
+Out[53]: <MessageOp.SET_PUBLISH: ['register', 'set_and_publish', 'delete', 'get', 'keys']>
+```
+The message type is a [`ScanStatusMessage`](/api_reference/_autosummary/bec_lib.messages.ScanStatusMessage) and the allowed operations are `register`, `set_and_publish`, `delete`, `get` and `keys`. Please check the (TODO section) for more information on the allowed operations.
+
+Let's assume we now want to access this information from the *BECIPythonClient*. We can use the following code:
+```python
+from bec_lib.endpoints import MessageEndpoints
+msg = bec.connector.get(MessageEndpoints.scan_status())
+msg.content # Content of the ScanStatusMessage
+msg.metadata # Metadata of the ScanStatusMessage
+```
+
+**Scan Number**
+The [`scan_number`](/api_reference/_autosummary/bec_lib.endpoints.MessageEndpoints.scan_number) endpoint allows you to access the current scan number. The message type is a [`VariableMessage`](/api_reference/_autosummary/bec_lib.messages.VariableMessage) and the allowed operations are `set`, `get`, `delete` and `keys`.
+To access the current scan number, we can use the following code:
+```python
+from bec_lib.endpoints import MessageEndpoints
+msg = bec.connector.get(MessageEndpoints.scan_number())
+current_scan_number = msg.content["value"]
+```
+
+**Device Read and Read_Configuration**
+The next two common endpoint that we would like to introduce are [`device_read`](/api_reference/_autosummary/bec_lib.endpoints.MessageEndpoints.device_read) and [`device_read_configuration`](/api_reference/_autosummary/bec_lib.endpoints.MessageEndpoints.device_read_configuration). These two endpoints give access to the last updated information of device signals of type *ophyd.Kind.normal/hinted* or *ophyd.Kind.config* respectively. In both cases, the message_type is a [`DeviceMessage`](/api_reference/_autosummary/bec_lib.messages.DeviceMessage) and the allowed operations are `register`, `set_and_publish`, `delete`, `get` and `keys`. 
+To access for example the last *device_read* message, we can use the following code:
+
+```python
+from bec_lib.endpoints import MessageEndpoints
+msg = bec.connector.get(MessageEndpoints.device_read())
+msg.content # Content of the ScanStatusMessage
+msg.metadata # Metadata of the ScanStatusMessage
+```
+
+```{note}
+The *device_read* and *device_read_configuration* endpoints are updated whenever a device signal is updated. From the *BECIPythonClient*, forcing a device to update would be done by calling the `read(cached=False)` or `read_configuration(cached=False)` methods respectively.
+```
+
+**Scan Segment**
+The [`scan_segment`](/api_reference/_autosummary/bec_lib.endpoints.MessageEndpoints.scan_segment) endpoint allows you to access the data of a scan segment, which corresponds to specific readings of devices with `readoutPriority = monitored`. Please check the device_configuration section for more information about the [readout_priority](developer.ophyd_device_config). 
+In a step scan, the scan segment is updated after each step, while in a fly scan, the scan segment is updated based on the procedure defined by the scan. The message type is a [`ScanMessage`](/api_reference/_autosummary/bec_lib.messages.ScanMessage) and the allowed operations are `register`, `send`. As you see from the allowed operations, we can not directly get the last scan segment, but we can subscribe to the endpoint and receive the scan segments as they are published. We will show how to subscribe to an endpoint in the next section.
+
+(developer.event_data.subscription)=
+## Subscribing to events
+Subscribing to events allows you to react to any new message published on a specific endpoint. We can do so by registering a callback function that will be executed whenever the endpoint publishes a new message. This all happens in the same thread, we therefore need to be careful with the execution time of the callback function.
+The callback function will receive a [`bec_lib.connector.MessageObject`](/api_reference/_autosummary/bec_lib.connector.MessageObject) as input, with *topic* as a field for the endpointinfo in REDIS and *value* with the respective BECMessage. Therefore, we need to write our callback function to be capable of handling *ScanMessage*. Let's assume our scan has at least 21 points, we can then use the following callback to print a line once data for point 20 is published:
+``` python
+def my_cb(msg, **kwargs):
+    if msg.value.content["point_id"] ==20:
+        print("Point 20 is done")
+        # My custom api call
+```
+After defining the callback function, we can subscribe to the endpoint with the following code:
+```python
+bec.connector.subscribe(MessageEndpoints.scan_segment(), my_cb)
+```
+Any new message published on the *scan_segment* endpoint will now trigger the callback function. 
+
+```{note}
+It is very important to keep the execution time of the callback function as short as possible. If the callback function takes too long to execute, it will block the thread and may compromise the performance of the system. For secondary operations, we recommend the callback function to trigger actions through an API call to a separate service.
+```
+
+## Accessing event data outside of the BECIPythonClient
+
+If you like to use the event data oustide of the *BECIPythonClient*, you can use the [`RedisConnector`](/api_reference/_autosummary/bec_lib.redis_connector.RedisConnector) to access the REDIS server. You have to provide the correct `'host:port'` to the connector which allows you to connect to REDIS from the system you are running the code. If REDIS runs locally, this would be `'localhost:6379'`.
+Note, at the beamline this would be the hostname of the bec_server. The port `6379` is the default port for REDIS. 
+```python
+from bec_lib.endpoints import MessageEndpoints
+from bec_lib.redis_connector import RedisConnector
+bootstrap = "localhost:6379" 
+connector = RedisConnector(bootstrap)
+connector.get(MessageEndpoints.device_read())
+```
+

docs/source/developer/data_acess/data_access.md

@@ -1,5 +0,0 @@
-(developer.data_access)=
-# Data Access
-Coming soon... 
-
-In the meantime, have a look at the data access documentation in the [user section](user.data_access_and_plotting).

docs/source/developer/data_acess/event_data.md

@@ -1,18 +0,0 @@
-(developer.event_data)=
-# BEC Event Data
-Coming soon...
-
-## Overview
-
-## Most commonly used event data
-
-**Queue status**
-
-**Scan status**
-
-**Scan segments**
-
-**Device readback**
-
-**Device status**
-

docs/source/developer/developer.md

@@ -56,6 +56,17 @@ No matter if you need to add new devices to BEC or modify existing ones, this se
 Discover and learn how to contribute to the command-line tool and graphical user interface for interacting with BEC.
 ```
 
+```{grid-item-card}
+:link: developer.getting_started
+:link-type: ref
+:img-top: /assets/rocket_launch_48dp.svg
+:text-align: center
+
+## Data Access
+
+Discover how to access data in BEC, including the structure of *ScanItems* to being able to get and subscribe to the event system of BEC.
+```
+
 ```{grid-item-card}
 :link: developer.scans
 :link-type: ref

docs/source/developer/glossary.md

@@ -3,26 +3,37 @@
 
 ```{glossary}
 :sorted:
+
 scan_id 
     The ``scan_id`` is the unique identifier for a scan. It is a string, typically uuid4, that is generated automatically by the scan server. It is used to uniquely identify a scan, even across multiple experiments and beamlines.
+
 scan_number
     The ``scan_number`` is an integer that is assigned to a scan by the scan server. It can be used to identify a scan within an experiment but is typically reset to 1 for each new experiment. The ``scan_number`` is also used by the file_writer to name the files that are generated by a scan.
+
 dataset_number
     The ``dataset_number`` is an integer that is assigned to a dataset by the scan server. It is used to group scans into larger, logical units. The ``dataset_number`` is typically reset to 1 for each new experiment.
+
 request_id
     The ``request_id`` is an identifier for a request to the scan server. As of now, it is a uuid4 string that is generated by the bec_lib during the preparation of a new request. A scan can comprise instructions from multiple requests.
+
 queue_id
     The ``queue_id`` is an identifier for an element in a queue. Each queue element can contain multiple scans and thus enforce the sequential execution of those scans. The ``queue_id`` is a uuid4 string that is generated by the bec_lib during the preparation of a new queue element.
+
 DIID
     The ``DIID`` is the device instruction ID. It is a continuously increasing integer that is assigned to each device instruction in a scan. The ``DIID`` is used to uniquely identify each device instruction and its response in the scan. 
+
 point_id
     The ``point_id`` is an integer that is assigned to each point in a scan. It is used to uniquely identify each point in a scan and is used to group the data that is generated by a scan. The ``point_id`` is typically set directly during the scan definition and used by the {term}`Scan Bundler` to logically bundle the data that is generated by a scan. In synchronous fly scans, the ``point_id may`` also be set directly by the device. 
+
 Device Server
     The Device Server provides a core service of BEC and handles the communication to the devices. It is the only process that actually holds the connection to the devices. Other services, like the {term}`Scan Server`, communicate with the Device Server to interact with the devices.
+
 Scan Server
     The Scan Server provides a core service of BEC and handles the execution of scans. It is responsible for receiving and validating scan requests, assembling the scan instructions, queueing the scan and finally executing the scan using a Scan Worker. If necessary, the Scan Server emits DeviceInstructionMessages to the {term}`Device Server` to interact with the devices. 
+
 Scan Bundler
     The Scan Bundler is a core service of BEC that is responsible for bundling the data that is generated by a scan. It subscribes to Redis channels and listens for data that is generated by the devices. The Scan Bundler then bundles the data into logical units, called scan segments, and emits them as ScanMessage back to Redis.
+    
 File Writer
     The File Writer is a core service of BEC that is responsible for writing HDF5 files with Nexus-compatible metadata and data entries to disk. It receives its data from the {term}`Scan Bundler`, and also adds external links to files written by other services, such as data backends for large 2D detector data. The internal structure of the files can be adjusted to the beamline’s needs using customizable plugins to comply with the desired [NeXus application definition](https://manual.nexusformat.org/classes/applications/index.html).
 ```

docs/source/user/command_line_interface.md

@@ -34,7 +34,7 @@ This allows users to use tab-completion for finding devices.
 
 To get a quick glance at all available devices, you can type
     
-```ipython
+``` ipython
 demo [1/3] ❯❯ dev.show_all()
 ```
 
@@ -244,7 +244,7 @@ dev.samx.limits = [-50, 50]
 You may also directly access the low and high limits via `dev.samx.low_limit = -50` and `dev.samx.high_limit=50`.
 Both access patterns are identical.
 Software limits are updated in the device_config, however, when done via command-line this only updates the current device_config session in redis.
-To make sure that limits are stored after reloading the device BEC config, you need to update the deviceConfig on disk, please check [bec.config.save_current_session()](#user.devices.export_device_config).
+To make sure that limits are stored after reloading the device BEC config, you need to update the deviceConfig on disk, please check [bec.config.save_current_session()](#user.devices).
 
 As per default, software limits for motors are set to the values specified in the [BEC device config](#developer.ophyd), subfield device_config. 
 

docs/source/user/devices.md

@@ -82,7 +82,7 @@ To change the readout priority of a device (e.g. samx), use
 dev.samx.readout_priority = "monitored" 
 ```
 
-Possible values are `monitored`, `baseline`, `on_request`, `async` and `continuous`. More details on the readout priority and the different modes can be found in the [developer guide](#developer.ophyd_device).
+Possible values are `monitored`, `baseline`, `on_request`, `async` and `continuous`. More details on the readout priority and the different modes can be found in the [developer guide](#developer.ophyd_device_config).
 
 (user.devices.update_device_config)=
 ### Update the device config
@@ -93,7 +93,7 @@ To update the device config, use
 dev.samx.set_device_config({"tolerance":0.02})
 ```
  which will update the tolerance window for the motor to reach its target position. 
- Keep in mind though, that the parameter exposed through the device_config must be configurable in the [ophyd_device](#developer.ophyd_device) of the bespoken device.
+ Keep in mind though, that the parameter exposed through the device_config must be configurable in the [ophyd_device](#developer.ophyd.ophyd_device) of the bespoken device.
 
 ### Set or update the user parameters