ITQC/gls_docs
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sf_daq/ch.psi.daq.queryrest#1

Browse Source 
- implementing compression of result streams
- allowing for different responseFormats (JSON, CSV)
This commit is contained in:
Zellweger Christof Ralf
2015-12-21 10:09:26 +01:00
parent 04583789cb
commit 01a74f9e6b
10 changed files with 364 additions and 212 deletions
Download Patch File Download Diff File Expand all files Collapse all files

129
Readme.md
View File

@ -48,6 +48,7 @@ The REST interface is accessible through `http://data-api.psi.ch/sf`.
<a name="query_channel_names"/>
## Query Channel Names
### Request
@ -71,14 +72,14 @@ POST http://<host>:<port>/channels
### Example
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"regex": "TRFCA|TRFCB"}' http://data-api.psi.ch/sf/channels
curl -H "Content-Type: application/json" -X POST -d '{"regex": "TRFCA|TRFCB"}' http://data-api.psi.ch/sf/channels
```
<a name="query_range"/>
## Query Range
Queries are applied to a range. Following ranges are supported.
Queries are applied to a range. The following types of ranges ranges are supported.
### By Pulse-Id
@ -131,17 +132,68 @@ Queries are applied to a range. Following ranges are supported.
## Query Data
### Request
### `compressed`: data is compressed by default
To save bandwidth, all data transferred from the server to the client is compressed (gzipped) by default. In case compressing the data is too processor-intense, it can be disabled by specifying `compressed=false` in the body of the request.
Because of this, we have to tell `curl` that the data is compressed so that it is being decompressed automatically. `curl` decompresses the response when the `--compressed` parameter is set:
#### Example
```bash
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
If we want the raw data uncompressed from the server, we have to specify this in the query body parameter with by specifying `compressed=false`:
```bash
curl -H "Content-Type: application/json" -X POST -d '{"compressed":false,"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
### `responseFormat`: data is in JSON by default
Responses can be formatted as CSV or JSON using the `responseFormat` field. The returned data is JSON-formatted per default.
CSV export does not support `index` and `extrema` aggregations.
#### Example
```bash
curl --compressed -H "Content-Type: application/json" -X POST -d '{"responseFormat":"csv","range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"],"fields":["channel","pulseId","iocMillis","iocNanos","globalMillis","globalNanos","shape","eventCount","value"]}' http://data-api.psi.ch/sf/query
```
#### Response example
The response is in CSV.
```text
channel;pulseId;iocMillis;iocNanos;globalMillis;globalNanos;shape;eventCount;value
testChannel1;0;0;0;0;0;[1];1;0
testChannel1;1;10;0;10;0;[1];1;1
testChannel1;2;20;0;20;0;[1];1;2
testChannel1;3;30;0;30;0;[1];1;3
testChannel1;4;40;0;40;0;[1];1;4
testChannel2;0;0;0;0;0;[1];1;0
testChannel2;1;10;0;10;0;[1];1;1
testChannel2;2;20;0;20;0;[1];1;2
testChannel2;3;30;0;30;0;[1];1;3
testChannel2;4;40;0;40;0;[1];1;4
```
### Query request endpoint
```
GET http://<host>:<port>/query
```
#### Data
#### Request body
A request is performed using JSON. The JSON query defines the channels to be queried, the range, and how the data should be aggregated (this is optional but highly recommended).
A request is performed by sending a valid JSON object in the HTTP request body. The JSON query defines the channels to be queried, the range, and how the data should be aggregated (this is optional but highly recommended).
There exist following fields:
The following attributes can be specified:
- **channels**: Array of channel names to be queried.
- **range**: The range of the query (see [Query Range](Readme.md#query_range)).
@ -153,12 +205,21 @@ There exist following fields:
- **aggregationType**: Specifies the type of aggregation (see [here](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/AggregationType.java)). The default type is *value* aggregation (e.g., sum([1,2,3])=6). Alternatively, it is possible to define *index* aggregation for multiple arrays in combination with binning (e.g., sum([1,2,3], [3,2,1]) = [4,4,4]).
- **aggregateChannels**: Specifies whether the data of the requested channels should be combined together using the defined aggregation (values: true|**false**)
- **dbMode**: Defines the database to access (values: **databuffer**|archiverappliance)
- **compressed**: Defines whether the response should be compressed or not (values: **true**|false)
- **responseFormat**: Specifies the format the response of the requested data is in, either in JSON or CSV format (values: **json**|csv)
### Example
Compressed data but uncompressed by `curl`:
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"]}' http://data-api.psi.ch/sf/query
```
Raw, uncompressed data (returns non-human-readable data):
```bash
curl -H "Content-Type: application/json" -X POST -d '{"compressed": false,"range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"]}' http://data-api.psi.ch/sf/query
```
### Response example
@ -222,7 +283,7 @@ The response is in JSON.
### Example Queries
Following examples build on a waveform data (see below). They also work for scalars (consider it as a waveform of length = 1) and images (waveform of length = dimX * dimY).
The following examples build on waveform data (see below). They also work for scalars (consider it as a waveform of length = 1) and images (waveform of length = dimX * dimY).
![Data Visualization](doc/images/Data_Visualization.png)
@ -299,7 +360,7 @@ Following examples build on a waveform data (see below). They also work for scal
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -327,7 +388,7 @@ See JSON representation of the data above.
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"range":{"startMillis":0,"startNanos":0,"endMillis":30,"endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startMillis":0,"startNanos":0,"endMillis":30,"endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -357,7 +418,7 @@ Supported format is ISO8601 *YYYY-MM-DDThh:mm:ss.sTZD* (e.g. *1997-07-16T19:20:3
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"range":{"startDate":"1970-01-01T01:00:00.000","startNanos":0,"endDate":"1970-01-01T01:00:00.030","endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startDate":"1970-01-01T01:00:00.000","startNanos":0,"endDate":"1970-01-01T01:00:00.030","endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -389,7 +450,7 @@ Archiver Appliance supports queries by *time range* and *date range* only (as it
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"dbmode":"archiverappliance","range":{"startMillis":0,"startNanos":0,"endMillis":30,"endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"dbmode":"archiverappliance","range":{"startMillis":0,"startNanos":0,"endMillis":30,"endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -418,7 +479,7 @@ Allows for server side optimizations since not all data needs to be retrieved.
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -471,7 +532,7 @@ Use **none** in case ordering does not matter (allows for server side optimizati
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"ordering":"desc","fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"ordering":"desc","fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -524,7 +585,7 @@ curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -598,7 +659,7 @@ Array value [aggregations](https://github.psi.ch/projects/ST/repos/ch.psi.daq.qu
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"nrOfBins":2,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"nrOfBins":2,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -657,7 +718,7 @@ Array value [aggregations](https://github.psi.ch/projects/ST/repos/ch.psi.daq.qu
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"binSize":10,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["globalMillis","value"],"range":{"globalMillis":0,"globalMillis":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"binSize":10,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["globalMillis","value"],"range":{"globalMillis":0,"globalMillis":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -714,7 +775,7 @@ Array value [aggregations](https://github.psi.ch/projects/ST/repos/ch.psi.daq.qu
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"nrOfBins":1,"aggregationType":"index","aggregations":["min","max","mean","sum"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"nrOfBins":1,"aggregationType":"index","aggregations":["min","max","mean","sum"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -783,7 +844,7 @@ curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -
###### Command
```bash
curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"aggregationType":"extrema","aggregations":["min","max","sum"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
curl --compressed -H "Content-Type: application/json" -X POST -d '{"aggregationType":"extrema","aggregations":["min","max","sum"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```
###### Response
@ -830,31 +891,3 @@ curl -H "Accept: application/json" -H "Content-Type: application/json" -X POST -
]
```
### CSV Export
Responses can be formatted as csv by requesting `text/csv`.
CSV export does not support `index` and `extrema` aggregations.
### Example
```bash
curl -H "Accept: text/csv" -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"],"fields":["channel","pulseId","iocMillis","iocNanos","globalMillis","globalNanos","shape","eventCount","value"]}' http://data-api.psi.ch/sf/query
```
### Response example
The response is in CSV.
```text
channel;pulseId;iocMillis;iocNanos;globalMillis;globalNanos;shape;eventCount;value
testChannel1;0;0;0;0;0;[1];1;0
testChannel1;1;10;0;10;0;[1];1;1
testChannel1;2;20;0;20;0;[1];1;2
testChannel1;3;30;0;30;0;[1];1;3
testChannel1;4;40;0;40;0;[1];1;4
testChannel2;0;0;0;0;0;[1];1;0
testChannel2;1;10;0;10;0;[1];1;1
testChannel2;2;20;0;20;0;[1];1;2
testChannel2;3;30;0;30;0;[1];1;3
testChannel2;4;40;0;40;0;[1];1;4
```