124 lines
4.7 KiB
Markdown
124 lines
4.7 KiB
Markdown
# **Saving multiple sources at SwissFEL: a very serious guide**
|
||
|
||
*Scroll down for the boring code*
|
||
|
||
**Hey, look at my nice data!**
|
||
*What a load of bs!*
|
||
|
||
**Oy, no need to be rude!**
|
||
*I’m not! I mean, what a load of **Beam Synchronous** data!*
|
||
|
||
**Well, thanks... but it’s not all saved.**
|
||
*Bummer. How were you saving it?*
|
||
|
||
**I used this cool command-line script `bs...`**
|
||
*Ooooh... I’m going to stop you right there.*
|
||
|
||
**But my bs command works; there must be a problem with the sources!**
|
||
*Calm down, detective, try saving each source individually with the `bs` command...*
|
||
|
||
**OK, hang on... what the?! They *both* save fine on their own!**
|
||
*Yep, the issue is when you try and save sources from different IOCs/devices with the `bs` command. The data is taken from the **dispatcher**. If the two sources don’t arrive at the dispatcher within a small time window, only the first source is sent in the message. Different sources arrive at the dispatcher at **different times**.*
|
||
|
||
**What type of BS is that?! I can't wait and wait...**
|
||
*Good question. Some bs data comes from **pipelines**, where calculations and moving data around **takes time**. You’re not just saving numbers—you’re saving processed results.*
|
||
|
||
**Pipelines?! I want data, not plumbing problems!**
|
||
*Think of pipelines as hardworking elves doing data analysis behind the scenes. No pipelines, more work for you.*
|
||
|
||
**Alright, I’m sold. But how do I save multiple sources without all this drama?**
|
||
*You need to save from the **data buffer**. The system can handle sources arriving at slightly different times there.*
|
||
|
||
**The data buffer? How?**
|
||
*You’ve got plenty of tools for accessing it:*
|
||
- **[DataHub](https://github.com/paulscherrerinstitute/datahub)** (don't ask about a front-end)
|
||
- **[Data API](https://github.com/paulscherrerinstitute/data_api_python)** (if you speak code)
|
||
- **[Eco](https://github.com/paulscherrerinstitute/eco), [Slic](https://gitlab.psi.ch/slic), Service Now, Concour, Time** (some might not work).
|
||
|
||
**Steady your sources, Doc Brown, I just checked my data and one of the sources isn't running at 100 Hz and missing data, I told you the source was the problem**
|
||
|
||
*Missing data isn't necessarily a problem; the approaches above can handle missing pulse IDs and return you all the data that is in the data buffer*
|
||
|
||
**Pulse IDs? Does my data need to prove its age?**
|
||
|
||
*No, its a way to sort data, with bs data every shot had a unique pulse ID. You can use SwissFEL data analysis packages to match arrays with missing shots in for you*
|
||
|
||
**Cool! Anything else I should know?**
|
||
*Yes: the databuffer can’t clear your desk at PiA*
|
||
|
||
**✅ Do Say:**
|
||
*Pulse IDs rock my world!*
|
||
|
||
**❌ Don’t Say:**
|
||
*Beam synchronous PV*
|
||
|
||
## Example scripts using datahub
|
||
|
||
### Saving historic data for a set time range
|
||
|
||
```python
|
||
from datahub import *
|
||
import matplotlib.pyplot as plt
|
||
from datetime import datetime, timedelta
|
||
|
||
# Set time range: from 6 minutes ago to 5 minutes ago
|
||
now = datetime.now()
|
||
from_time, to_time = [
|
||
(now - timedelta(minutes=m)).strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]
|
||
for m in [6, 5]
|
||
]
|
||
|
||
# Define the channels to monitor
|
||
channels = [
|
||
"SARFE10-PSSS059:SPECTRUM_Y",
|
||
"SAROP21-PBPS133:INTENSITY",
|
||
"SARFE10-PBPG050:FAST-PULSE-ENERGY"
|
||
]
|
||
|
||
# Construct the query with channels and time range
|
||
query = {
|
||
"channels": channels,
|
||
"start": from_time,
|
||
"end": to_time
|
||
}
|
||
|
||
# Connect to the data source and retrieve data
|
||
with Daqbuf(backend="sf-databuffer", cbor=True) as source:
|
||
table = Table()
|
||
source.add_listener(table)
|
||
source.request(query)
|
||
dataframe = table.as_dataframe(index=Table.PULSE_ID)
|
||
|
||
# Iterate through each channel and print the number of pulses
|
||
for channel in channels:
|
||
if channel in dataframe.columns:
|
||
NumShots = dataframe[channel].count()
|
||
print(f"{channel}: {NumShots} pulses")
|
||
else:
|
||
print(f"{channel}: Channel not found in the dataframe.")
|
||
```
|
||
|
||
### Example of a stream of live data
|
||
|
||
```python
|
||
from datahub import Bsread, Table
|
||
|
||
channels = [
|
||
"SARFE10-PSSS059:SPECTRUM_Y",
|
||
"SAROP21-PBPS133:INTENSITY",
|
||
"SARFE10-PBPG050:FAST-PULSE-ENERGY"
|
||
]
|
||
|
||
with Bsread() as source:
|
||
table = Table()
|
||
source.add_listener(table)
|
||
source.req(channels, 0.0, 2.0)
|
||
dataframe = table.as_dataframe(index=Table.PULSE_ID)
|
||
|
||
for channel in channels:
|
||
if channel in dataframe.columns:
|
||
NumShots = dataframe[channel].count()
|
||
print(f"{channel}: {NumShots} pulses")
|
||
else:
|
||
print(f"{channel}: Channel not found in the dataframe.")
|
||
``` |