From 7ccfc30a298fdc8ea0680afdf2ee5d4cecd1115f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20M=C3=A4rki?= Date: Thu, 16 Jun 2016 14:18:43 +0200 Subject: [PATCH] Overwork Readme and add llrf.policies --- Readme.md | 86 +++++++++-- policies/default.policies | 2 +- policies/di.policies | 2 +- policies/llrf.policies | 313 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 385 insertions(+), 18 deletions(-) create mode 100644 policies/llrf.policies diff --git a/Readme.md b/Readme.md index a655d92..1e864d3 100644 --- a/Readme.md +++ b/Readme.md @@ -36,11 +36,11 @@ The JSON structure is defined as follows: # Data Policies -Policies files are JSON formatted text files defining the data reduction scheme to be applied to data points and are stored in the `policies` folder. Matching policies to data points is done using regular expressions applied to channel names (see: [Pattern](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html), more precisely using [Matcher.find()](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#find--)) whereas longer match groups are considered superior to shorter match groups (i.e., pattern 'TRFCB-XY.*' matches channel name 'TRFCB-XYZ' better than pattern 'TRFCB.*'). In case several files define the same pattern, the one defined in the first file (given by the natural ordering of the file system) will be used. +Policies files are JSON formatted text files defining the data reduction scheme to be applied to data points and are stored in the `policies` folder. Assigning policies to data points is done using regular expressions applied to channel names (see: [Pattern](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html), more precisely using [Matcher.find()](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#find--)) whereas a longer match sequence is considered superior to a shorter match sequence. Data points are tagged with the data reduction scheme at the time of their creation/arrival. Accordingly, changes in data policies only effects new data points and has no effect on already persisted data points. -The default data policy is defined in [default.policies](https://github.psi.ch/sf_config/sf_daq_sources/raw/master/policies/default.policies) and is applied to every data point unless there is a data policy providing a superior match. Currently, the default retention time is one day. +The default data policy is defined in [default.policies](https://github.psi.ch/sf_config/sf_daq_sources/raw/master/policies/default.policies) and is applied to every data point unless there is a data policy providing a superior match sequence. Currently, the default retention time is one day. Each group should maintain their own list of data policies in (a) separate file(s). The filename should start with the groups short name (e.g. rf.policies, llrf.policies). A group might maintain more than 1 file. In this case all files should start with the groups short name followed by an underscore and then the rest of the name. The suffix should always be `.policies` (e.g. llrf.policies, llrf_group1.policies, ...). @@ -51,7 +51,7 @@ The JSON structure is defined as follows: ```json { "policies": [{ - "channel": "^SINDG01.*", + "pattern": "^SINDG01", "data_reduction": { "default": [{ "ttl": "P1D", @@ -72,21 +72,75 @@ The JSON structure is defined as follows: ## Explanation - **policies**: List of all data policies defined in this file. -- **channel**: The regular expression applied to channel names. Longer match groups are considered superior to shorter match groups (i.e., Pattern '^SINDG01.*AMPLT$' matches channel name 'SINDG01-RCIR-PUP10:SIG-AMPLT' better than Pattern '^SINDG01.*'). It is also possible to define a channel specific policy using the exact channel name. +- **pattern**: The regular expression applied to channel names. - **data_reduction**: The data reduction applied to the channels matching the above regular expression. This section can contain objects `default` (this reduction scheme is applied to all data points unless a more specific is defined), `scalar` (this reduction scheme is applied to scalar data points overwriting the `default` scheme), `waveform` (this reduction scheme is applied to waveform data points overwriting the `default` scheme), and `image` (this reduction scheme is applied to image data points overwriting the `default` scheme). - **ttl**: The time-to-live of the data point (after that time the data point will be deleted). The ttl is defined as a [ISO-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) duration (e.g., `PT6H` for 6 hours, `P2D` for 2 days etc.). Using a ttl of `-1` will disable recording of channels matching the above pattern. The resolution is in seconds and thus the minimal ttl is 1 second. - **modulo**: Defines the x-th data point (based on the pulse-id) the ttl should be applied to (e.g., `modulo 1`: the ttl is applied to every data point, `modulo 10`: the ttl is applied to ever 10th data point, `modulo 100`: the ttl is applied to ever 100th data point). It is always the maximum ttl applied (pulse-id 1000 matches modulo 1 and 100 and thus "P7D" would be applied). - **offset**: (default: 0) Can be used to define an offset in the pulse-id match (e.g. `modulo: 10, offset: 0` matches pulse-ids 0,10,20... whereas `modulo: 10, offset: 2` matches pulse-ids 2,12,22... -### Examples -#### Example 1 +## Overwriting Policies + +It is possible to overwrite existing data policies. Assigning policies to data points is done using regular expressions applied to channel names (see: [Pattern](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html), more precisely using [Matcher.find()](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#find--)) whereas a longer match sequence is considered superior to a shorter match sequence (i.e., matching channel name 'SINDG01-RCIR-PUP10:SIG-AMPLT' to the pattern '^SINDG' and '^SINDG01' results in match sequences '**SINDG**01-RCIR-PUP10:SIG-AMPLT' vs. '**SINDG01**-RCIR-PUP10:SIG-AMPLT' and thus pattern '^SINDG01' is considered superior). It is also possible define a data policy for a specific channel by using the channels exact name. In case several files define the same pattern, the one defined in the first file (given by the natural ordering of the file system) will be used. + +#### General Rules (Words of Care) + +Using regular expressions to assign data policies to data points should assist users and facilitate their work by just having to define a few rules to handle their full data set. However, users are encouraged not to overstress the power of regular expressions. As a general rule: + +- Do not use wildcards '.*' at the beginning or end of a pattern as this would always result in match sequences covering the complete channel name, making overwrites impossible (Example 3 and 4 explain why). +- Use *or* combinations with care. +- Do not use complex regular expressions (things difficult to understand like '((^|, )(part1|part2|part3))+$'. + + +### Example Overwrites + +#### Example 1 (Dos) + +Lets assume the channel 'SINDG01-RCIR-PUP10:SIG-AMPLT' is applied to pattern '^SINDG' and '^SINDG01'. This results in the match sequences: + +- '^SINDG': '**SINDG**01-RCIR-PUP10:SIG-AMPLT' +- '^SINDG01': '**SINDG01**-RCIR-PUP10:SIG-AMPLT' + +and thus the data policy of '^SINDG01' will be applied to the channel. + +#### Example 2 (Dos with care) + +Lets assume a user is responsible for channels starting with 'SINDG' or 'SINOG' or 'SINUG'. This user defines a base data policy for all these channels by using the pattern '^SINDG|^SINOG|^SINUG'. Lets further assume this user wants to temporary overwrite the data policies for all channels starting with 'SINDG01' (e.g. because the machine shows unexpected behavior and channels starting with 'SINDG01' help to identify the problem). This is possible by defining a new data policy with the pattern '^SINDG01'. Lets apply channels 'SINDG01-RCIR-PUP10:SIG-AMPLT' and 'SINOG01-RCIR-PUP10:SIG-AMPLT' to these patterns: + +- '^SINDG|^SINOG|^SINUG': '**SINDG**01-RCIR-PUP10:SIG-AMPLT' and '**SINOG**01-RCIR-PUP10:SIG-AMPLT' +- '^SINDG01': '**SINDG01**-RCIR-PUP10:SIG-AMPLT' and 'SINOG01-RCIR-PUP10:SIG-AMPLT' + +and thus the data policy of '^SINDG01' will be applied to 'SINDG01-RCIR-PUP10:SIG-AMPLT' and the policy of '^SINDG|^SINOG|^SINUG' to 'SINOG01-RCIR-PUP10:SIG-AMPLT'. + + +#### Example 3 (Don'ts) + +Lets assume the channel 'SINDG01-RCIR-PUP10:SIG-AMPLT' would be applied to pattern '^SINDG.*' and '^SINDG01.*'. This would result in the match sequences: + +- '^SINDG.*': '**SINDG01-RCIR-PUP10:SIG-AMPLT**' +- '^SINDG01.*': '**SINDG01-RCIR-PUP10:SIG-AMPLT**' + +as the pattern matches the full channel name. In this case, ties could be broken by considering the length of the pattern where longer patterns are considered superior to shorter patterns (giving precedence to '^SINDG01.*'). Example 4 shows why this does not work in all cases. + +#### Example 4 (Don'ts) + +Lets assume the channels 'SINDG01-RCIR-PUP10:SIG-AMPLT' and 'SINOG01-RCIR-PUP10:SIG-AMPLT' would be applied to patterns '^SINDG.*|^SINOG.*|^SINUG.*' and 'SINDG01.*'. This would result in the match sequences: + +- '^SINDG.*|^SINOG.*|^SINUG.*': '**SINDG01-RCIR-PUP10:SIG-AMPLT**' and '**SINOG01-RCIR-PUP10:SIG-AMPLT**' +- '^SINDG01.*': '**SINDG01-RCIR-PUP10:SIG-AMPLT**' and 'SINOG01-RCIR-PUP10:SIG-AMPLT' + +as the patterns matches the full channel names. Again, ties could be broken by considering the length of the pattern resulting in '^SINDG.*|^SINOG.*|^SINUG.*' matching both channels as it is longer. Certainly not what the user intended. One could try to overcome this problem by splitting *or* combinations and use separate but equivalent data policies internally (i.e., in a separate data policy for '^SINDG.*', '^SINOG.*', and '^SINUG.*'). However, regex allows for expressions which are difficult to split (not to mention how to understand, e.g., '((^|, )(part1|part2|part3))+$'). Therefore, keep to the general rules of pattern definition. + + +## Example Policies + +### Example 1 Group XYZ defines there default data policy for `SINDG01` to be 2 days. ```json { - "channel":"^SINDG01.*", + "pattern":"^SINDG01", "data_reduction":{ "default":[ { @@ -101,13 +155,13 @@ Group XYZ defines there default data policy for `SINDG01` to be 2 days. All data points (scalar, waveform, image) of channels starting with `SINDG01` (`^` defines 'start with' in regex) have a ttl of 2 days. -#### Example 2 +### Example 2 Lets assume group XYZ wants to track a problem in `SINDG01` and therefore they want to keep waveforms for 3 days giving them time to analyze the problem. ```json { - "channel":"^SINDG01.*", + "pattern":"^SINDG01", "data_reduction":{ "waveform":[ { @@ -128,13 +182,13 @@ Lets assume group XYZ wants to track a problem in `SINDG01` and therefore they w All waveform data points of channels starting with `SINDG01` have a ttl of 3 days. All other data points (scalar, image) have a ttl of 2 days. -#### Example 3 +### Example 3 Lets assume group XYZ was not able to track down the problem using data worth of 3 days. Therefore they decide to extend the time horizon to 10 days but realize there is not enough storage space available. However, they are confident to find the problem with every 100th data point. ```json { - "channel":"^SINDG01.*", + "pattern":"^SINDG01", "data_reduction":{ "waveform":[ { @@ -159,13 +213,13 @@ Lets assume group XYZ was not able to track down the problem using data worth of The default ttl of waveform data points of channels starting with `SINDG01` is 2 days and every 100th of these data points has a ttl of 10 days. All other data points (scalar, image) have a ttl of 2 days. -#### Example 4 +### Example 4 -Lets assume group XYZ could solve the problem and (as exemplary DAQ users) decide to release the additional storage space for other DAQ users. However, they still want to track every 1000th scalar data point of channels starting with `SINDG01` and having `AMPL` in their name for 10 days (as they provide enough information to verify that the problem did not re-appear). +Lets assume group XYZ could solve the problem and (as exemplary DAQ users) decide to release the additional storage space for other DAQ users. However, they still want to track every 1000th scalar data point of channels starting with `SINDG01` and ending with `AMPL` in their name for 10 days (as they provide enough information to verify that the problem did not re-appear). ```json { - "channel":"^SINDG01.*", + "pattern":"^SINDG01", "data_reduction":{ "default":[ { @@ -177,7 +231,7 @@ Lets assume group XYZ could solve the problem and (as exemplary DAQ users) decid } { - "channel":"^SINDG01.*AMPL.*", + "pattern":"^SINDG01.*AMPL$", "data_reduction":{ "scalar":[ { @@ -199,5 +253,5 @@ Lets assume group XYZ could solve the problem and (as exemplary DAQ users) decid } ``` -The default ttl for data points of channels starting with `SINDG01` is 2 days. All scalar data points staring with `SINDG01` and having `AMPL` in their name have a ttl of 2 days and ever 1000th of these data points has a ttl of 10 days. +The default ttl for data points of channels starting with `SINDG01` is 2 days. All scalar data points staring with `SINDG01` and ending with `AMPL` in their name have a ttl of 2 days and ever 1000th of these data points has a ttl of 10 days. diff --git a/policies/default.policies b/policies/default.policies index 55faa23..f14da72 100644 --- a/policies/default.policies +++ b/policies/default.policies @@ -2,7 +2,7 @@ "policies": [ /*all data points last for one day in the DataBuffer unless there is a data policy providing a superior match*/ { - "channel": ".*", + "channel": ".", "data_reduction": { "default": [{ "ttl": "P1D", diff --git a/policies/di.policies b/policies/di.policies index a710452..8202f4c 100644 --- a/policies/di.policies +++ b/policies/di.policies @@ -2,7 +2,7 @@ "policies": [ /*DBLM (T. Šuštar): 3 days everything, 10 days every 100th*/ { - "channel": ".*DBLM.*", + "channel": "DBLM", "data_reduction": { "default": [{ "ttl": "P3D", diff --git a/policies/llrf.policies b/policies/llrf.policies new file mode 100644 index 0000000..1b4b23f --- /dev/null +++ b/policies/llrf.policies @@ -0,0 +1,313 @@ +/* LLRF configuration for data policy */ +/* ================================== */ + +/* IMPORTANT Workflow notice */ +/* After merge request was generated, make an ELOG entry to the LLRF logbook at */ +/* https://elog-gfa.psi.ch/LLRF with */ +/* Section = All */ +/* System = LLRF */ +/* Subsystem = General */ +/* Subject = New Data Buffer TTL policy */ +/* Text = Describe what you have changed and place this link: */ +/* https://git.psi.ch/sf_config/sf_daq_sources/commits/master/policies/llrf.policies */ + + +/* Explanation of bsread data config for LLRF: */ +/* 1. TMEM map Excel document column with bsread will generate a */ +/* # bsread_info header comment in the template channels_uplink_data_ch...template. */ +/* 2. script cfg/AutoStart.sh -> calls cfg/bsread_cfg_extract.sh */ +/* 3. results written to /var/volatile/llrf_bsread.cfg on the IOC ramdisk. */ +/* 4. llrf_rtapp parses this config file llrf_bsread.cfg on startup */ +/* 5. In case GUI the STREAMING-RECONFIG checkbox pressed, this file is reloaded by */ +/* the rtapp. So the way of manually adding new PV's would be to login to the IOC, */ +/* and then edit this file (e.g. with nano or vi), store it and reload it. */ + + + + +/* ------------------------------------------------------------ */ +/* - override settings - */ +/* - - */ +/* - Each default setting will be overridden, when a better - */ +/* - match (exacter match) is made. It is independant of the - */ +/* - sequence in this configuration file. - */ +/* ------------------------------------------------------------ */ + + + + + + + + +{ + "policies": [ + +/* ------------------------------------------------------------ */ +/* - default settings which are generic for all RF stations - */ +/* ------------------------------------------------------------ */ + +/* default setting for status channels of FPGA */ +{ + "pattern":"-RLLE-STA:", + "data_reduction":{ + "default":[ + { + "ttl":"P6M", + "modulo":1 + } + , + { + "ttl":"P2Y", + "modulo":100 + /* 1 point every 1 second */ + } + , + { + "ttl":"P10Y", + "modulo":60000 + /* 1 point every 10 minutes */ + } + ] + } +} +, +/* override for some specific channels, delete fully after ... */ +{ + "pattern":"-RLLE-STA:.*TIMER$", + "data_reduction":{ + "default":[ + { + "ttl":"P6M", + "modulo":1 + } + ] + } +} +, +{ + "pattern":"-RLLE-STA:.*EVRPULSEID$", + "data_reduction":{ + "default":[ + { + "ttl":"P6M", + "modulo":1 + } + ] + } +} +, +{ + "pattern":"-RLLE-STA:.*MSICNT$", + "data_reduction":{ + "default":[ + { + "ttl":"P6M", + "modulo":1 + } + ] + } +} +, + + +/* ------------------------------------------------------------ */ + + +/* default setting for status channels of DSP / RT_APP feedback control */ +{ + "pattern":"-RLLE-DSP:", + "data_reduction":{ + "default":[ + { + "ttl":"P6M", + "modulo":1 + } + , + { + "ttl":"P2Y", + "modulo":100 + /* 1 point every 1 second */ + } + , + { + "ttl":"P10Y", + "modulo":60000 + /* 1 point every 10 minutes */ + } + ] + } +} +, + + +/* ------------------------------------------------------------ */ + + +/* default setting for waveforms (AMPLT / PHASE / DACI/Q */ +{ + "pattern":"-RIQM-STA:DAC", + "data_reduction":{ + "waveform":[ + { + "ttl":"PT2H", + "modulo":1 + } + , + { + "ttl":"P6M", + "modulo":100 + /* 1 point every 1 second */ + } + , + { + "ttl":"P10Y", + "modulo":60000 + /* 1 point every 10 minutes */ + } + ] + } +} +, +{ + "pattern":"-AMPLT$", + "data_reduction":{ + "waveform":[ + { + "ttl":"PT2H", + "modulo":1 + } + , + { + "ttl":"P6M", + "modulo":100 + /* 1 point every 1 second */ + } + , + { + "ttl":"P10Y", + "modulo":60000 + /* 1 point every 10 minutes */ + } + ] + } +} +, +{ + "pattern":"-PHASE$", + "data_reduction":{ + "waveform":[ + { + "ttl":"PT2H", + "modulo":1 + } + , + { + "ttl":"P6M", + "modulo":100 + /* 1 point every 1 second */ + } + , + { + "ttl":"P10Y", + "modulo":60000 + /* 1 point every 10 minutes */ + } + ] + } +} +, +/* default setting for ADCRAW, but typically not sent (extracted in cfg/bsread_cfg_extract.sh */ +{ + "pattern":"-ADCRAW$", + "data_reduction":{ + "waveform":[ + { + "ttl":"PT2H", + "modulo":1 + } + ] + } +} +, +/* override for klystron output amplitude waveform 1 week with 100 Hz */ +{ + "pattern":"RKLY-DCP10:FOR-AMPLT$", + "data_reduction":{ + "waveform":[ + { + "ttl":"P1W", + "modulo":1 + } + , + { + "ttl":"P6M", + "modulo":100 + /* 1 point every 1 second */ + } + , + { + "ttl":"P10Y", + "modulo":60000 + /* 1 point every 10 minutes */ + } + ] + } +} +, + +/* ------------------------------------------------------------ */ + + +/* default setting for 100 Hz scalars AMPLT-AVG and PHASE-AVG */ +{ + "pattern":"-AMPLT-AVG$", + "data_reduction":{ + "scalar":[ + { + "ttl":"P6M", + "modulo":1 + } + , + { + "ttl":"P2Y", + "modulo":100 + /* 1 point every 1 second */ + } + , + { + "ttl":"P10Y", + "modulo":60000 + /* 1 point every 10 minutes */ + } + ] + } +} +, +{ + "pattern":"-PHASE-AVG$", + "data_reduction":{ + "scalar":[ + { + "ttl":"P6M", + "modulo":1 + } + , + { + "ttl":"P2Y", + "modulo":100 + /* 1 point every 1 second */ + } + , + { + "ttl":"P10Y", + "modulo":60000 + /* 1 point every 10 minutes */ + } + ] + } +} + ] +} + +