01a74f9e6b
- implementing compression of result streams - allowing for different responseFormats (JSON, CSV)
#Overview
This project provides a REST interface to execute queries on the databuffer.
Requirements
This project requires Java 8 or greater.
Deployment
Use the instructions provided by ch.psi.daq.install to install the application on a server.
Application Properties
Following files define and describe application properties:
- Cassandra specific properties.
- Query specific properties..
- Query REST specific properties.
It is possible to overwrite properties by defining new values in ${HOME}/.config/daq/queryrest.properties
Maven
Upload jar to the Maven repository (from ch.psi.daq.buildall):
./gradlew ch.psi.daq.queryrest:uploadArchives
DropIt
Upload jar DropIt (from ch.psi.daq.buildall):
./gradlew ch.psi.daq.queryrest:dropIt -x test
Local Instance
DAQLocal provides a local instance of the DAQ system for testing purposes (allowing users/developers to verify their code before they come to PSI to do their research and interact with the DAQ cluster).
REST Interface
The REST interface is accessible through http://data-api.psi.ch/sf.
Query Channel Names
Request
POST http://<host>:<port>/channels
Data
{"regex": "TRFCA|TRFCB", "dbMode": "databuffer"}
Explanation
- regex: Reqular expression used to filter channel names (default: no filtering). Filtering is done using JAVA's Pattern, more precisely Matcher.find()).
- dbMode: Defines the database to access (values: databuffer|archiverappliance)
Example
curl -H "Content-Type: application/json" -X POST -d '{"regex": "TRFCA|TRFCB"}' http://data-api.psi.ch/sf/channels
Query Range
Queries are applied to a range. The following types of ranges ranges are supported.
By Pulse-Id
"range":{
"startPulseId":0,
"endPulseId":100
}
- startPulseId: The start pulse-id of the range request.
- endPulseId: The end pulse-id of the range request.
By Date
"range":{
"startDate":"2015-08-06T18:00:00.000",
"startNanos":0,
"endDate":"2015-08-06T18:59:59.999",
"endNanos":999999
}
- startDate: The start date of the time range (ISO8601 format (YYYY-MM-DDThh:mm:ss.sTZD e.g. 1997-07-16T19:20:30.475+02:00 (omitting +02:00 falls back to the local time zone)).
- [startNanos]: The optional nanosecond offset to the milliseconds (range [0..999999]).
- endDate: The end date of the time range.
- [endNanos]: The optional nanosecond offset.
By Time
"range":{
"startMillis":0,
"startNanos":0,
"endMillis":100,
"endNanos":999999
}
- startMillis: The start time of the range in milliseconds since January 1, 1970 (the UNIX/JAVA epoch).
- [startNanos]: The optional nanosecond offset to the milliseconds (range [0..999999]).
- endMillis: The end time of the range.
- [endNanos]: The optional nanosecond offset.
Query Data
compressed: data is compressed by default
To save bandwidth, all data transferred from the server to the client is compressed (gzipped) by default. In case compressing the data is too processor-intense, it can be disabled by specifying compressed=false in the body of the request.
Because of this, we have to tell curl that the data is compressed so that it is being decompressed automatically. curl decompresses the response when the --compressed parameter is set:
Example
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
If we want the raw data uncompressed from the server, we have to specify this in the query body parameter with by specifying compressed=false:
curl -H "Content-Type: application/json" -X POST -d '{"compressed":false,"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
responseFormat: data is in JSON by default
Responses can be formatted as CSV or JSON using the responseFormat field. The returned data is JSON-formatted per default.
CSV export does not support index and extrema aggregations.
Example
curl --compressed -H "Content-Type: application/json" -X POST -d '{"responseFormat":"csv","range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"],"fields":["channel","pulseId","iocMillis","iocNanos","globalMillis","globalNanos","shape","eventCount","value"]}' http://data-api.psi.ch/sf/query
Response example
The response is in CSV.
channel;pulseId;iocMillis;iocNanos;globalMillis;globalNanos;shape;eventCount;value
testChannel1;0;0;0;0;0;[1];1;0
testChannel1;1;10;0;10;0;[1];1;1
testChannel1;2;20;0;20;0;[1];1;2
testChannel1;3;30;0;30;0;[1];1;3
testChannel1;4;40;0;40;0;[1];1;4
testChannel2;0;0;0;0;0;[1];1;0
testChannel2;1;10;0;10;0;[1];1;1
testChannel2;2;20;0;20;0;[1];1;2
testChannel2;3;30;0;30;0;[1];1;3
testChannel2;4;40;0;40;0;[1];1;4
Query request endpoint
GET http://<host>:<port>/query
Request body
A request is performed by sending a valid JSON object in the HTTP request body. The JSON query defines the channels to be queried, the range, and how the data should be aggregated (this is optional but highly recommended).
The following attributes can be specified:
- channels: Array of channel names to be queried.
- range: The range of the query (see Query Range).
- ordering: The ordering of the data (see here for possible values).
- fields: The requested fields (see here for possible values).
- nrOfBins: Activates data binning. Specifies the number of bins the pulse/time range should be divided into.
- binSize: Activates data binning. Specifies the number of pulses per bin for pulse-range queries or the number of milliseconds per bin for time-range queries (using number of pulses and number of milliseconds makes this binning strategy consistent between channel with different update frequencies).
- aggregations: Activates data aggregation. Array of requested aggregations (see here for possible values). These values will be added to the data array response.
- aggregationType: Specifies the type of aggregation (see here). The default type is value aggregation (e.g., sum([1,2,3])=6). Alternatively, it is possible to define index aggregation for multiple arrays in combination with binning (e.g., sum([1,2,3], [3,2,1]) = [4,4,4]).
- aggregateChannels: Specifies whether the data of the requested channels should be combined together using the defined aggregation (values: true|false)
- dbMode: Defines the database to access (values: databuffer|archiverappliance)
- compressed: Defines whether the response should be compressed or not (values: true|false)
- responseFormat: Specifies the format the response of the requested data is in, either in JSON or CSV format (values: json|csv)
Example
Compressed data but uncompressed by curl:
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"]}' http://data-api.psi.ch/sf/query
Raw, uncompressed data (returns non-human-readable data):
curl -H "Content-Type: application/json" -X POST -d '{"compressed": false,"range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"]}' http://data-api.psi.ch/sf/query
Response example
The response is in JSON.
[
{
"channel":"channel1",
"data":[
{
"pulseId":0,
"iocMillis":0,
"iocNanos":0,
"globalMillis":0,
"globalNanos":0,
"value":0
},
{
"pulseId":2,
"iocMillis":2,
"iocNanos":2,
"globalMillis":2,
"globalNanos":2,
"value":2
},
{
"pulseId":4,
"iocMillis":4,
"iocNanos":4,
"globalMillis":4,
"globalNanos":4,
"value":4
}
]
},
{
"channel":"channel2",
"data":[
{
"pulseId":1,
"iocMillis":1,
"iocNanos":1,
"globalMillis":1,
"globalNanos":1,
"value":1
},
{
"pulseId":3,
"iocMillis":3,
"iocNanos":3,
"globalMillis":3,
"globalNanos":3,
"value":3
}
]
}
]
Example Queries
The following examples build on waveform data (see below). They also work for scalars (consider it as a waveform of length = 1) and images (waveform of length = dimX * dimY).
[
{
"channel":"Channel_01",
"data":[
{
"iocMillis":0,
"iocNanos":0,
"pulseId":0,
"globalMillis":0,
"globalNanos":0,
"shape":[
4
],
"value":[1,2,3,4]
},
{
"iocMillis":10,
"iocNanos":0,
"pulseId":1,
"globalMillis":10,
"globalNanos":0,
"shape":[
4
],
"value":[2,3,4,5]
},
{
"iocMillis":20,
"iocNanos":0,
"pulseId":2,
"globalMillis":20,
"globalNanos":0,
"shape":[
4
],
"value":[3,4,5,6]
},
{
"iocMillis":30,
"iocNanos":0,
"pulseId":3,
"globalMillis":30,
"globalNanos":0,
"shape":[
4
],
"value":[4,5,6,7]
}
]
}
]
Query Examples
Query by Pulse-Id Range
{
"range":{
"startPulseId":0,
"endPulseId":3
},
"channels":[
"Channel_01"
]
}
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
See JSON representation of the data above.
Query by Time Range
{
"range":{
"startMillis":0,
"startNanos":0,
"endMillis":30,
"endNanos":999999
},
"channels":[
"Channel_01"
]
}
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startMillis":0,"startNanos":0,"endMillis":30,"endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
See JSON representation of the data above.
Query by Date Range
{
"range":{
"startDate":"1970-01-01T01:00:00.000",
"startNanos":0,
"endDate":"1970-01-01T01:00:00.030",
"endNanos":999999
},
"channels":[
"Channel_01"
]
}
Supported format is ISO8601 YYYY-MM-DDThh:mm:ss.sTZD (e.g. 1997-07-16T19:20:30.475+02:00).
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"range":{"startDate":"1970-01-01T01:00:00.000","startNanos":0,"endDate":"1970-01-01T01:00:00.030","endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
See JSON representation of the data above.
Querying Archiver Appliance
{
"dbMode":"archiverappliance",
"range":{
"startMillis":0,
"startNanos":0,
"endMillis":30,
"endNanos":999999
},
"channels":[
"Channel_01"
]
}
Archiver Appliance supports queries by time range and date range only (as it has no notion about pulse-id).
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"dbmode":"archiverappliance","range":{"startMillis":0,"startNanos":0,"endMillis":30,"endNanos":999999},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
See JSON representation of the data above.
Querying for Specific Fields
Allows for server side optimizations since not all data needs to be retrieved.
{
"fields":["pulseId","value"],
"range":{
"startPulseId":0,
"endPulseId":3
},
"channels":[
"Channel_01"
]
}
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
[
{
"channel":"Channel_01",
"data":[
{
"pulseId":0,
"value":[1,2,3,4]
},
{
"pulseId":1,
"value":[2,3,4,5]
},
{
"pulseId":2,
"value":[3,4,5,6]
},
{
"pulseId":3,
"value":[4,5,6,7]
}
]
}
]
Data Ordering
{
"ordering":"desc",
"fields":["pulseId","value"],
"range":{
"startPulseId":0,
"endPulseId":3
},
"channels":[
"Channel_01"
]
}
Use none in case ordering does not matter (allows for server side optimizations).
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"ordering":"desc","fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
[
{
"channel":"Channel_01",
"data":[
{
"pulseId":3,
"value":[4,5,6,7]
},
{
"pulseId":2,
"value":[3,4,5,6]
},
{
"pulseId":1,
"value":[2,3,4,5]
},
{
"pulseId":0,
"value":[1,2,3,4]
}
]
}
]
Value Aggregation
{
"aggregationType":"value",
"aggregations":["min","max","mean"],
"fields":["pulseId","value"],
"range":{
"startPulseId":0,
"endPulseId":3
},
"channels":[
"Channel_01"
]
}
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
[
{
"channel":"Channel_01",
"data":[
{
"pulseId":0,
"value":{
"min":1.0,
"max":4.0,
"mean":2.5
}
},
{
"pulseId":1,
"value":{
"min":2.0,
"max":5.0,
"mean":3.5
}
},
{
"pulseId":2,
"value":{
"min":3.0,
"max":6.0,
"mean":4.5
}
},
{
"pulseId":3,
"value":{
"min":4.0,
"max":7.0,
"mean":5.5
}
}
]
}
]
Array value aggregations:
Value Aggregation with Binning (nrOfBins)
{
"nrOfBins":2,
"aggregationType":"value",
"aggregations":["min","max","mean"],
"fields":["pulseId","value"],
"range":{
"startPulseId":0,
"endPulseId":3
},
"channels":[
"Channel_01"
]
}
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"nrOfBins":2,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
[
{
"channel":"Channel_01",
"data":[
{
"pulseId":0,
"value":{
"min":1.0,
"max":5.0,
"mean":3.0
}
},
{
"pulseId":2,
"value":{
"min":3.0,
"max":7.0,
"mean":5.0
}
}
]
}
]
Array value aggregations with additional binning:
Value Aggregation with Binning (binSize)
binSize specifies the number of pulses per bin for pulse-range queries or the number of milliseconds per bin for time-range queries (using number of pulses and number of milliseconds makes this binning strategy consistent between channel with different update frequencies).
{
"binSize":10,
"aggregationType":"value",
"aggregations":["min","max","mean"],
"fields":["globalMillis","value"],
"range":{
"globalMillis":0,
"globalMillis":3
},
"channels":[
"Channel_01"
]
}
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"binSize":10,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["globalMillis","value"],"range":{"globalMillis":0,"globalMillis":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
[
{
"channel":"Channel_01",
"data":[
{
"globalMillis":0,
"value":{
"min":1.0,
"max":5.0,
"mean":3.0
}
},
{
"globalMillis":20,
"value":{
"min":3.0,
"max":7.0,
"mean":5.0
}
}
]
}
]
Array value aggregations with additional binning:
Index Aggregation
{
"nrOfBins":1,
"aggregationType":"index",
"aggregations":["min","max","mean","sum"],
"fields":["pulseId","value"],
"range":{
"startPulseId":0,
"endPulseId":3
},
"channels":[
"Channel_01"
]
}
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"nrOfBins":1,"aggregationType":"index","aggregations":["min","max","mean","sum"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
[
{
"channel":"Channel_01",
"data":[
{
"pulseId":0,
"value":[
{
"min":1.0,
"max":4.0,
"mean":2.5,
"sum":10.0
},
{
"min":2.0,
"max":5.0,
"mean":3.5,
"sum":14.0
},
{
"min":3.0,
"max":6.0,
"mean":4.5,
"sum":18.0
},
{
"min":4.0,
"max":7.0,
"mean":5.5,
"sum":22.0
}
]
}
]
}
]
Aggregation of array indices with binning (several nrOfBins are also possible).
Extrema Search
{
"aggregationType":"extrema",
"aggregations":["min","max","sum"],
"fields":["pulseId","value"],
"range":{
"startPulseId":0,
"endPulseId":3
},
"channels":[
"Channel_01"
]
}
Command
curl --compressed -H "Content-Type: application/json" -X POST -d '{"aggregationType":"extrema","aggregations":["min","max","sum"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
Response
[
{
"channel":"Channel_01",
"data":{
"minima":{
"min":{
"value":1.0,
"event":{
"pulseId":0,
"value":[1,2,3,4]
}
},
"sum":{
"value":10.0,
"event":{
"pulseId":0,
"value":[1,2,3,4]
}
}
},
"maxima":{
"max":{
"value":7.0,
"event":{
"pulseId":3,
"value":[4,5,6,7]
}
},
"sum":{
"value":22.0,
"event":{
"pulseId":3,
"value":[4,5,6,7]
}
}
}
}
}
]