diff --git a/Readme.md b/Readme.md index ba4262c..fff7b9f 100644 --- a/Readme.md +++ b/Readme.md @@ -1,23 +1,201 @@ # Overview -This repository holds the requested bsread data sources of the SwissFEL Data Buffer. -# Configuration Files -Config files are text files holding one IOC/source per line. If the IOC is not reachable by the "standard" bsread port 9999 the port needs to be appended to the IOC name via `:`. - -The file might contain comments starting with `#` which are not interpreted. - -Each group should maintain their own list of IOCs in (a) separate file(s). The filename should start with the groups short name (e.g. rf.sources, llrf.sources). A group might maintain more than 1 file. In this case all files should start with the groups short name followed by an underscore and then the rest of the name. The suffix should always be `.sources` (e.g. llrf.sources, llrf_group1.sources, ...) - -Example: -``` -# Goup A IOCs -SOME_IOC_01 -SOME_IOC_02 - -# Goup B IOCs -# Special test IOC reachable on port 20000 -TEST_IOC_01:20000 -``` +This repository holds the requested bsread data sources and their data policies of the SwissFEL Data Buffer. # Configuration Management + The configuration change workflow is described in following Memorandum: http://i.psi.ch/czF7 + +# Data Sources + +Data sources files are JSON formatted text files defining IOCs/sources and are stored in the `sources` folder. + +Each group should maintain their own list of IOCs in (a) separate file(s). The filename should start with the group's short name (e.g. rf.sources, llrf.sources). A group might maintain more than 1 file. In this case all files should start with the groups short name followed by an underscore and then the rest of the name. The suffix should always be `.sources` (e.g. llrf.sources, llrf_group1.sources, ...). + +Sources files might contain comments like `/* my comment */` which are not interpreted. + +The JSON structure is defined as follows: + +```json +{ + "sources": [{ + /* IOC using default port 9999 */ + "stream": "tcp://sf-ioc-xyz.psi.ch:9999" + }, { + /* IOC using non default port 20000 */ + "stream": "tcp://sf-ioc-abc.psi.ch:20000" + }] +} +``` + +## Explanation + +- **sources**: List of all IOCs/sources defined in this file. +- **stream**: The IOC/source to be recorded (`tcp://:` - the default bsread port is 9999). + + +# Data Policies + +Policies files are JSON formatted text files defining the data reduction scheme to be applied to data points and are stored in the `policies` folder. Matching policies to data points is done using regular expressions applied to channel names (see: [Pattern](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html), more precisely using [Matcher.find()](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#find--)) whereas longer match groups are considered superior to shorter match groups (i.e., pattern 'TRFCB-XY.*' matches channel name 'TRFCB-XYZ' better than pattern 'TRFCB.*'). In case several files define the same pattern, the one defined in the first file (given by the natural ordering of the file system) will be used. + +The default data policy is defined in [default.policies](https://github.psi.ch/sf_config/sf_daq_sources/raw/master/policies/default.policies) and is applied to every data point unless there is a data policy providing a superior match. Currently, the default retention time is one day. + +Each group should maintain their own list of data policies in (a) separate file(s). The filename should start with the groups short name (e.g. rf.policies, llrf.policies). A group might maintain more than 1 file. In this case all files should start with the groups short name followed by an underscore and then the rest of the name. The suffix should always be `.policies` (e.g. llrf.policies, llrf_group1.policies, ...). + +Policies files might contain comments like `/* my comment */` which are not interpreted. + +The JSON structure is defined as follows: + +```json +{ + "policies": [{ + "channel": "^SINDG01.*", + "data_reduction": { + "default": [{ + "ttl": "P1D", + "modulo": 1 + }], + "scalar": [{ + "ttl": "P2D", + "modulo": 1 + }, { + "ttl": "P7D", + "modulo": 100 + }] + } + }] +} +``` + +## Explanation + +- **policies**: List of all data policies defined in this file. +- **channel**: The regular expression applied to channel names. Longer match groups are considered superior to shorter match groups (i.e., Pattern '^SINDG01.*AMPLT$' matches channel name 'SINEG01-RCIR-PUP10:SIG-AMPLT' better than Pattern '^SINDG01.*'). It is also possible to define a channel specific policy using the exact channel name. +- **data_reduction**: The data reduction applied to the channels matching the above regular expression. This section can contain objects `default` (this reduction scheme is applied to all data points unless a more specific is defined), `scalar` (this reduction scheme is applied to scalar data points overwriting the `default` scheme), `waveform` (this reduction scheme is applied to waveform data points overwriting the `default` scheme), and `image` (this reduction scheme is applied to image data points overwriting the `default` scheme). +- **ttl**: The time-to-live of the data point (after that time the data point will be deleted). The ttl is defined as a [ISO-8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) duration (e.g., `PT6H` for 6 hours, `P2D` for 2 days etc.). Using a ttl of `-1` will disable recording of channels matching the above pattern. +- **modulo**: Defines the x-th data point (based on the pulse-id) the ttl should be applied to (e.g., `modulo 1`: the ttl is applied to every data point, `modulo 10`: the ttl is applied to ever 10th data point, `modulo 100`: the ttl is applied to ever 100th data point). It is always the maximum ttl applied (pulse-id 1000 matches modulo 1 and 100 and thus "P7D" would be applied). +- **offset**: (default: 0) Can be used to define an offset in the pulse-id match (e.g. `modulo: 10, offset: 0` matches pulse-ids 0,10,20... whereas `modulo: 10, offset: 2` matches pulse-ids 2,12,22... + +### Examples + +#### Example 1 + +Group XYZ defines there default data policy for `SINDG01` to be 2 days. + +```json +{ + "channel":"^SINEG01.*", + "data_reduction":{ + "default":[ + { + "ttl":"P2D", + "modulo":1 + } + ] + } +} +``` + +All data points (scalar, waveform, image) of channels starting with `SINDG01` (`^` defines 'start with' in regex) have a ttl of 2 days. + + +#### Example 2 + +Lets assume group XYZ wants to track a problem in `SINDG01` and therefore they want to keep waveforms for 3 days giving them time to analyze the problem. + +```json +{ + "channel":"^SINDG01.*", + "data_reduction":{ + "waveform":[ + { + "ttl":"P3D", + "modulo":1 + } + ], + "default":[ + { + "ttl":"P2D", + "modulo":1 + } + ] + } +} +``` + +All waveform data points of channels starting with `SINDG01` have a ttl of 3 days. All other data points (scalar, image) have a ttl of 2 days. + + +#### Example 3 + +Lets assume group XYZ was not able to track down the problem using data worth of 3 days. Therefore they decide to extend the time horizon to 10 days but realize there is not enough storage space available. However, they are confident to find the problem with every 100th data point. + +```json +{ + "channel":"^SINDG01.*", + "data_reduction":{ + "waveform":[ + { + "ttl":"P2D", + "modulo":1 + }, + { + "ttl":"P10D", + "modulo":100 + } + ], + "default":[ + { + "ttl":"P2D", + "modulo":1 + } + ] + } +} +``` + +The default ttl of waveform data points of channels starting with `SINDG01` is 2 days and every 100th of these data points has a ttl of 10 days. All other data points (scalar, image) have a ttl of 2 days. + + +#### Example 4 + +Lets assume group XYZ could solve the problem and (as exemplary DAQ users) decide to release the additional storage space for other DAQ users. However, they still want to track every 1000th scalar data point of channels starting with `SINDG01` and having `AMPL` in their name for 10 days (as they provide enough information to verify that the problem did not re-appear). + +```json +{ + "channel":"^SINDG01.*", + "data_reduction":{ + "default":[ + { + "ttl":"P2D", + "modulo":1 + } + ] + } +} + +{ + "channel":"^SINDG01.*AMPL.*", + "data_reduction":{ + "scalar":[ + { + "ttl":"P2D", + "modulo":1 + }, + { + "ttl":"P10D", + "modulo":1000 + } + ], + "default":[ + { + "ttl":"P2D", + "modulo":1 + } + ] + } +} +``` + +The default ttl for data points of channels starting with `SINDG01` is 2 days. All scalar data points staring with `SINDG01` and having `AMPL` in their name have a ttl of 2 days and ever 1000th of these data points has a ttl of 10 days. + diff --git a/llrf.sources b/llrf.sources deleted file mode 100644 index a4597b5..0000000 --- a/llrf.sources +++ /dev/null @@ -1,77 +0,0 @@ -# LLRF IOC's with bsdata streaming from real time application (=port:20000) - -# injector -SINEG01-CVME-LLRF1:20000 -SINEG01-CVME-LLRF2:20000 -SINSB01-CVME-LLRF1:20000 -SINSB01-CVME-LLRF2:20000 -SINSB02-CVME-LLRF1:20000 -SINSB02-CVME-LLRF2:20000 -#SINSB03-CVME-LLRF1:20000 -#SINSB03-CVME-LLRF2:20000 -#SINSB04-CVME-LLRF1:20000 -#SINSB04-CVME-LLRF2:20000 -#SINXB01-CVME-LLRF1:20000 -#SINXB01-CVME-LLRF2:20000 -#SINDI01-CVME-LLRF1:20000 -#SINDI01-CVME-LLRF2:20000 - -# linac 1 -#S10CB01-CVME-LLRF1:20000 -#S10CB01-CVME-LLRF2:20000 -#S10CB02-CVME-LLRF1:20000 -#S10CB02-CVME-LLRF2:20000 -#S10CB03-CVME-LLRF1:20000 -#S10CB03-CVME-LLRF2:20000 -#S10CB04-CVME-LLRF1:20000 -#S10CB04-CVME-LLRF2:20000 -#S10CB05-CVME-LLRF1:20000 -#S10CB05-CVME-LLRF2:20000 -#S10CB06-CVME-LLRF1:20000 -#S10CB06-CVME-LLRF2:20000 -#S10CB07-CVME-LLRF1:20000 -#S10CB07-CVME-LLRF2:20000 -#S10CB08-CVME-LLRF1:20000 -#S10CB08-CVME-LLRF2:20000 -#S10CB09-CVME-LLRF1:20000 -#S10CB09-CVME-LLRF2:20000 - -# linac 2 -#S20CB01-CVME-LLRF1:20000 -#S20CB01-CVME-LLRF2:20000 -#S20CB02-CVME-LLRF1:20000 -#S20CB02-CVME-LLRF2:20000 -#S20CB03-CVME-LLRF1:20000 -#S20CB03-CVME-LLRF2:20000 -#S20CB04-CVME-LLRF1:20000 -#S20CB04-CVME-LLRF2:20000 - -# linac 3 -#S30CB01-CVME-LLRF1:20000 -#S30CB01-CVME-LLRF2:20000 -#S30CB02-CVME-LLRF1:20000 -#S30CB02-CVME-LLRF2:20000 -#S30CB03-CVME-LLRF1:20000 -#S30CB03-CVME-LLRF2:20000 -#S30CB04-CVME-LLRF1:20000 -#S30CB04-CVME-LLRF2:20000 -#S30CB05-CVME-LLRF1:20000 -#S30CB05-CVME-LLRF2:20000 -#S30CB06-CVME-LLRF1:20000 -#S30CB06-CVME-LLRF2:20000 -#S30CB07-CVME-LLRF1:20000 -#S30CB07-CVME-LLRF2:20000 -#S30CB08-CVME-LLRF1:20000 -#S30CB08-CVME-LLRF2:20000 -#S30CB09-CVME-LLRF1:20000 -#S30CB09-CVME-LLRF2:20000 -#S30CB10-CVME-LLRF1:20000 -#S30CB10-CVME-LLRF2:20000 -#S30CB11-CVME-LLRF1:20000 -#S30CB11-CVME-LLRF2:20000 -#S30CB12-CVME-LLRF1:20000 -#S30CB12-CVME-LLRF2:20000 -#S30CB13-CVME-LLRF1:20000 -#S30CB13-CVME-LLRF2:20000 -#S30CB14-CVME-LLRF1:20000 -#S30CB14-CVME-LLRF2:20000 diff --git a/policies/default.policies b/policies/default.policies new file mode 100644 index 0000000..55faa23 --- /dev/null +++ b/policies/default.policies @@ -0,0 +1,14 @@ +{ + "policies": [ + /*all data points last for one day in the DataBuffer unless there is a data policy providing a superior match*/ + { + "channel": ".*", + "data_reduction": { + "default": [{ + "ttl": "P1D", + "modulo": 1 + }] + } + } + ] +} diff --git a/policies/di.policies_tmp b/policies/di.policies_tmp new file mode 100644 index 0000000..a710452 --- /dev/null +++ b/policies/di.policies_tmp @@ -0,0 +1,17 @@ +{ + "policies": [ + /*DBLM (T. Šuštar): 3 days everything, 10 days every 100th*/ + { + "channel": ".*DBLM.*", + "data_reduction": { + "default": [{ + "ttl": "P3D", + "modulo": 1 + }, { + "ttl": "P10D", + "modulo": 100 + }] + } + } + ] +} diff --git a/sources/di.sources_tmp b/sources/di.sources_tmp new file mode 100644 index 0000000..3317b43 --- /dev/null +++ b/sources/di.sources_tmp @@ -0,0 +1,9 @@ +{ + /*Diagnostic IOC's*/ + "sources": [ + /*DBLM (T. Šuštar)*/ + { + "stream": "tcp://SINLH02-CVME-DBLM018:9999" + } + ] +} diff --git a/sources/llrf.sources b/sources/llrf.sources new file mode 100644 index 0000000..ba2fc99 --- /dev/null +++ b/sources/llrf.sources @@ -0,0 +1,77 @@ +{ +/*LLRF IOC's with bsdata streaming from real time application (=port:20000)*/ + "sources": [ +/*injector*/ + {"stream": "tcp://SINEG01-CVME-LLRF1:20000"} + ,{"stream": "tcp://SINEG01-CVME-LLRF2:20000"} + ,{"stream": "tcp://SINSB01-CVME-LLRF1:20000"} + ,{"stream": "tcp://SINSB01-CVME-LLRF2:20000"} + ,{"stream": "tcp://SINSB02-CVME-LLRF1:20000"} + ,{"stream": "tcp://SINSB02-CVME-LLRF2:20000"} + /*,{"stream": "tcp://SINSB03-CVME-LLRF1:20000"} + ,{"stream": "tcp://SINSB03-CVME-LLRF2:20000"} + ,{"stream": "tcp://SINSB04-CVME-LLRF1:20000"} + ,{"stream": "tcp://SINSB04-CVME-LLRF2:20000"} + ,{"stream": "tcp://SINXB01-CVME-LLRF1:20000"} + ,{"stream": "tcp://SINXB01-CVME-LLRF2:20000"} + ,{"stream": "tcp://SINDI01-CVME-LLRF1:20000"} + ,{"stream": "tcp://SINDI01-CVME-LLRF2:20000"}*/ +/*linac 1*/ + /*,{"stream": "tcp://S10CB01-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB01-CVME-LLRF2:20000"} + ,{"stream": "tcp://S10CB02-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB02-CVME-LLRF2:20000"} + ,{"stream": "tcp://S10CB03-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB03-CVME-LLRF2:20000"} + ,{"stream": "tcp://S10CB04-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB04-CVME-LLRF2:20000"} + ,{"stream": "tcp://S10CB05-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB05-CVME-LLRF2:20000"} + ,{"stream": "tcp://S10CB06-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB06-CVME-LLRF2:20000"} + ,{"stream": "tcp://S10CB07-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB07-CVME-LLRF2:20000"} + ,{"stream": "tcp://S10CB08-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB08-CVME-LLRF2:20000"} + ,{"stream": "tcp://S10CB09-CVME-LLRF1:20000"} + ,{"stream": "tcp://S10CB09-CVME-LLRF2:20000"}*/ +/*linac 2*/ + /*,{"stream": "tcp://S20CB01-CVME-LLRF1:20000"} + ,{"stream": "tcp://S20CB01-CVME-LLRF2:20000"} + ,{"stream": "tcp://S20CB02-CVME-LLRF1:20000"} + ,{"stream": "tcp://S20CB02-CVME-LLRF2:20000"} + ,{"stream": "tcp://S20CB03-CVME-LLRF1:20000"} + ,{"stream": "tcp://S20CB03-CVME-LLRF2:20000"} + ,{"stream": "tcp://S20CB04-CVME-LLRF1:20000"} + ,{"stream": "tcp://S20CB04-CVME-LLRF2:20000"}*/ +/*linac 3*/ + /*,{"stream": "tcp://S30CB01-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB01-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB02-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB02-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB03-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB03-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB04-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB04-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB05-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB05-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB06-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB06-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB07-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB07-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB08-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB08-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB09-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB09-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB10-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB10-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB11-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB11-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB12-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB12-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB13-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB13-CVME-LLRF2:20000"} + ,{"stream": "tcp://S30CB14-CVME-LLRF1:20000"} + ,{"stream": "tcp://S30CB14-CVME-LLRF2:20000"}*/ + ] +} diff --git a/validate.sh b/validate.sh new file mode 100755 index 0000000..7cc80b5 --- /dev/null +++ b/validate.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +HOST="http://sf-dispatcher-api.psi.ch" +# HOST="http://localhost:8080" +HTTP_ADDRESS="/configuration/validate" +RESPONSE=response.json +# see http://stackoverflow.com/questions/18258804/how-to-catch-timeout-errors-in-a-curl-shell-script +COMMAND="curl -s -w %{http_code} -X POST" + +while read -r -d '' filename; do + COMMAND+=" -F files=@${filename}" +done < <(find . \( -name "*.sources" -o -name "*.policies" \) -print0) + +COMMAND+=" ${HOST}${HTTP_ADDRESS} -o ${RESPONSE}" +# echo "Execute command: "${COMMAND} +STATUS=$(${COMMAND}) + +cat ${RESPONSE} | python -m json.tool +rm -f ${RESPONSE} +#echo ${STATUS}