# Overview

This project provides a REST interface to execute queries on the databuffer.

# Requirements

This project requires Java 8 or greater.

# Deployment

Use the instructions provided by [ch.psi.daq.install](https://git.psi.ch/sf_daq/ch.psi.daq.install#query_rest) to install the application on a server.

## Application Properties

Following files define and describe application properties:

- [Cassandra](https://github.psi.ch/projects/ST/repos/ch.psi.daq.cassandra/browse/src/main/resources/cassandra.properties) specific properties.
- [Query](https://github.psi.ch/projects/ST/repos/ch.psi.daq.dispatcher/browse/src/main/resources/query.properties) specific properties.. 
- [Query REST](https://github.psi.ch/projects/ST/repos/ch.psi.daq.queryrest/browse/src/main/resources/queryrest.properties) specific properties.

It is possible to overwrite properties by defining new values in `${HOME}/.config/daq/queryrest.properties`

## Maven

Upload jar to the Maven repository (from ch.psi.daq.buildall):

```bash
./gradlew ch.psi.daq.queryrest:uploadArchives
```

## DropIt

Upload jar DropIt (from ch.psi.daq.buildall):

```bash
./gradlew ch.psi.daq.queryrest:dropIt -x test
```

## Local Instance

[DAQLocal](https://github.psi.ch/projects/ST/repos/ch.psi.daq.daqlocal/browse) provides a local instance of the DAQ system for testing purposes (allowing users/developers to verify their code before they come to PSI to do their research and interact with the DAQ cluster).

<a name="rest_interface"/>

# REST Interface

The REST interface is accessible through `http://data-api.psi.ch/sf`.

<a name="query_channel_names"/>


## Query Channel Names 

### Request

```
POST http://<host>:<port>/channels
```

#### Data

```json
{"regex": "TRFCA|TRFCB","backends": ["databuffer"],"ordering":"asc","reload":true}
```

##### Explanation

- **regex**: Reqular expression used to filter channel names. In case this value is undefined, no filter will be applied. Filtering is done using JAVA's [Pattern](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html), more precisely [Matcher.find()](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#find--)).
- **backends**: Array of backends to access (values: databuffer|archiverappliance|filestorage). In case this value is undefined, all backends will be queried for their channels.
- **ordering**: The ordering of the channel names (values: **none**|asc|desc).
- **reload**: Forces the server to reload cached channel names (values: **false**|true).

### Example

#### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"regex": "TRFCA|TRFCB"}' http://data-api.psi.ch/sf/channels
```

#### Response

```json
[
   {
      "backend":"databuffer",
      "channels":[
         "Channel_01",
         "Channel_02",
         "Channel_03"
      ]
   },
   {
      "backend":"archiverappliance",
      "channels":[
         "Channel_01",
         "Channel_04",
         "Channel_05"
      ]
   }
]
```

<a name="query_range"/>

## Query Range 

Queries are applied to a range. The following types of ranges ranges are supported. 

### By Pulse-Id

```json
"range":{
   "startPulseId":0,
   "endPulseId":100
}
```

- **startPulseId**: The start pulse-id of the range request.
- **endPulseId**: The end pulse-id of the range request.


### By Date

```json
"range":{
   "startDate":"2015-08-06T18:00:00.000",
   "endDate":"2015-08-06T18:59:59.999",
}
```

- **startDate**: The start date of the time range (ISO8601 format (YYYY-MM-DDThh:mm:ss.sTZD e.g. 1997-07-16T19:20:30.123+02:00 (omitting +02:00 falls back to the local time zone)). Additionally, ISO8601 format up to nano seconds is also supported (e.g. 1997-07-16T19:20:30.123456789+02:00).
- **endDate**: The end date of the time range.


### By Time

```json
"range":{
   "startSeconds":"0.0",
   "endSeconds":"1.000999999"
}
```

- **startSeconds**: The start time of the range in seconds since January 1, 1970 (the UNIX epoch) as a decimal value including fractional seconds.
- **endSeconds**: The end time of the range in seconds since January 1, 1970 (the UNIX epoch) as a decimal value including fractional seconds.

<a name="query_channel_names"/>

## Query Channel Names

The simplest way to define channels is to use an array of channel name Strings.

```json
"channels":[
	"Channel_02",
	"Channel_04"
]
```

The query interface will automatically select the backend which contains the channel (e.g., *databuffer* for *Channel_02* and *archiverappliance* for *Channel_04*. In case name clashes exist, the query interface will use following order of priority: *databuffer*, *archiverappliance* and *filestorage*.

It is also possible to explicitly define the backend to overcome name clashes.   

```json
"channels":[
   {
      "name":"Channel_01",
      "backend":"archiverappliance"
   },
   {
      "name":"Channel_01",
      "backend":"databuffer"
   }
]
```


<a name="query_data"/>

## Query Data 

### Query request endpoint

```
GET http://<host>:<port>/query
```

#### Request body

A request is performed by sending a valid JSON object in the HTTP request body. The JSON query defines the channels to be queried, the range, and how the data should be aggregated (this is optional but highly recommended).

The following attributes can be specified:

- **channels**: Array of channels to be queried (see [Query Range](Readme.md#query_channel_names)).
- **range**: The range of the query (see [Query Range](Readme.md#query_range)).
- **ordering**: The ordering of the data (see [here](https://git.psi.ch/sf_daq/ch.psi.daq.common/blob/master/src/main/java/ch/psi/daq/common/ordering/Ordering.java) for possible values).
- **fields**: The requested fields (see [here](https://git.psi.ch/sf_daq/ch.psi.daq.query/blob/master/src/main/java/ch/psi/daq/query/model/QueryField.java) for possible values).
- **nrOfBins**: Activates data binning. Specifies the number of bins the pulse/time range should be divided into.
- **binSize**: Activates data binning. Specifies the number of pulses per bin for pulse-range queries or the number of milliseconds per bin for time-range queries (using number of pulses and number of milliseconds makes this binning strategy consistent between channel with different update frequencies). 
- **aggregations**: Activates data aggregation. Array of requested aggregations (see [here](https://git.psi.ch/sf_daq/ch.psi.daq.query/blob/master/src/main/java/ch/psi/daq/query/model/Aggregation.java) for possible values). These values will be added to the *data* array response.
- **aggregationType**: Specifies the type of aggregation (see [here](https://git.psi.ch/sf_daq/ch.psi.daq.query/blob/master/src/main/java/ch/psi/daq/query/model/AggregationType.java)). The default type is *value* aggregation (e.g., sum([1,2,3])=6). Alternatively, it is possible to define *index* aggregation for multiple arrays in combination with binning (e.g., sum([1,2,3], [3,2,1]) = [4,4,4]).
- **compression**: Defines the compression algorithm to use, default value is **none**, see all values [here](https://github.psi.ch/sf_daq/ch.psi.daq.query/blob/master/src/main/java/ch/psi/daq/query/model/Compression.java))
- **responseFormat**: Specifies the format the response of the requested data is in, either in JSON or CSV format, default value **JSON**, see all values [here](https://github.psi.ch/sf_daq/ch.psi.daq.query/blob/master/src/main/java/ch/psi/daq/query/model/ResponseFormat.java))

### `compression`: compression of data can be enabled

By default, no data is compressed when transferred from the server to the client. However, compression can be enabled by setting the `compression` attribute to a value other than `none`, i.e. to `gzip` or `deflate`.

If compression is enabled, we have to tell `curl` that the data is compressed so that it is being decompressed automatically. `curl` decompresses the response when the `--compressed` parameter is set.

### `responseFormat`: data is in JSON by default

Responses can be formatted as CSV or JSON using the `responseFormat` field. The returned data is JSON-formatted by default. 

CSV export does not support `index` and `extrema` aggregations.

### Example Queries

The following examples build on waveform data (see below). They also work for scalars (consider it as a waveform of length = 1) and images (waveform of length = dimX * dimY). 

![Data Visualization](doc/images/Data_Visualization.png)

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "iocSeconds":"0.000000000",
            "pulseId":0,
            "globalSeconds":"0.000000000",
            "shape":[
               4
            ],
            "value":[1,2,3,4]
         },
         {
            "iocSeconds":"0.010000000",
            "pulseId":1,
            "globalSeconds":"0.010000000",
            "shape":[
               4
            ],
            "value":[2,3,4,5]
         },
         {
            "iocSeconds":"0.020000000",
            "pulseId":2,
            "globalSeconds":"0.020000000",
            "shape":[
               4
            ],
            "value":[3,4,5,6]
         },
         {
            "iocSeconds":"0.030000000",
            "pulseId":3,
            "globalSeconds":"0.030000000",
            "shape":[
               4
            ],
            "value":[4,5,6,7]
         }
      ]
   }
]
```

### Query Examples

##### Query by Pulse-Id Range

```json
{
   "range":{
      "startPulseId":0,
      "endPulseId":3
   },
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

See JSON representation of the data above. 



##### Query by Time Range

```json
{
   "range":{
      "startSeconds":"0.0",
      "endSeconds":"0.030999999"
   },
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"range":{"startSeconds":"0.0","endSeconds":"0.030999999"},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

See JSON representation of the data above. 



##### Query by Date Range

```json
{
   "range":{
      "startDate":"1970-01-01T01:00:00.000",
      "endDate":"1970-01-01T01:00:00.030"
   },
   "channels":[
      "Channel_01"
   ]
}
```

Supported format is ISO8601 *YYYY-MM-DDThh:mm:ss.sTZD* (e.g. *1997-07-16T19:20:30.475+02:00*).

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"range":{"startDate":"1970-01-01T01:00:00.000","endDate":"1970-01-01T01:00:00.030"},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

See JSON representation of the data above. 



##### Querying Archiver Appliance

```json
{
   "range":{
      "startSeconds":"0.0",
      "endSeconds":"0.030999999"
   },
   "channels":[
      {
	      "name": "Channel_01",
	      "backend":"archiverappliance"
      },
      {
	      "name": "Channel_02",
	      "backend":"archiverappliance"
      }
   ]
}
```

Archiver Appliance supports queries by *time range* and *date range* only (as it has no notion about pulse-id).


###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"range":{"startSeconds":"0.0","endSeconds":"0.030999999"},"channels":[{"name": "Channel_01","backend":"archiverappliance"}]}' http://data-api.psi.ch/sf/query
```

###### Response

See JSON representation of the data above. 

##### Query using compression

```json
{
    "compression":"gzip",
    "range":{
        "startPulseId":0,
        "endPulseId":3
     },
     "channels":[
        "Channel_01"
     ]
}
```

or `deflate` can be used too:

```json
{
    "compression":"deflate",
    "range":{
        "startPulseId":0,
        "endPulseId":3
     },
     "channels":[
        "Channel_01"
     ]
}
```

###### Command (gzip)

The `curl` command has a `--compressed` option to decompress data automatically.

```bash
curl --compressed -H "Content-Type: application/json" -X POST -d '{"compression":"gzip","range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

##### Query setting CSV response format

```json
{
    "responseFormat":"csv",
    "range":{
        "startPulseId":0,
        "endPulseId":4
    },
    "channels":[
        "channel1",
        "channel2"
    ],
    "fields":[
        "channel",
        "pulseId",
        "iocSeconds",
        "globalSeconds",
        "shape",
        "eventCount",
        "value"
    ]
}
```

It is possible to request the time in seconds (since January 1, 1970 (the UNIX epoch) as a decimal value including fractional seconds - using fields *globalSeconds* and *iocSeconds*) or in milliseconds (since January 1, 1970 (the JAVA epoch) - using fields *globalMillis* and *iocMillis*)

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"responseFormat":"csv","range":{"startPulseId":0,"endPulseId":4},"channels":["channel1","channel2"],"fields":["channel","pulseId","iocSeconds","globalSeconds","shape","eventCount","value"]}' http://data-api.psi.ch/sf/query
```

###### Response

The response is in CSV.

```text
channel;pulseId;iocSeconds;globalSeconds;shape;eventCount;value
testChannel1;0;0.000000000;0.000000000;[1];1;0
testChannel1;1;0.010000000;0.010000000;[1];1;1
testChannel1;2;0.020000000;0.020000000;[1];1;2
testChannel1;3;0.030000000;0.030000000;[1];1;3
testChannel1;4;0.040000000;0.040000000;[1];1;4
testChannel2;0;0.000000000;0.000000000;[1];1;0
testChannel2;1;0.010000000;0.010000000;[1];1;1
testChannel2;2;0.020000000;0.020000000;[1];1;2
testChannel2;3;0.030000000;0.030000000;[1];1;3
testChannel2;4;0.040000000;0.040000000;[1];1;4
``` 

##### Querying for Specific Fields

Allows for server side optimizations since not all data needs to be retrieved.

```json
{
   "fields":["pulseId","value"],
   "range":{
      "startPulseId":0,
      "endPulseId":3
   },
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":0,
            "value":[1,2,3,4]
         },
         {
            "pulseId":1,
            "value":[2,3,4,5]
         },
         {
            "pulseId":2,
            "value":[3,4,5,6]
         },
         {
            "pulseId":3,
            "value":[4,5,6,7]
         }
      ]
   }
]
```

##### Data Ordering

```json
{
   "ordering":"desc",
   "fields":["pulseId","value"],
   "range":{
      "startPulseId":0,
      "endPulseId":3
   },
   "channels":[
      "Channel_01"
   ]
}
```

Use **none** in case ordering does not matter (allows for server side optimizations).


###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"ordering":"desc","fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":3,
            "value":[4,5,6,7]
         },
         {
            "pulseId":2,
            "value":[3,4,5,6]
         },
         {
            "pulseId":1,
            "value":[2,3,4,5]
         },
         {
            "pulseId":0,
            "value":[1,2,3,4]
         }
      ]
   }
]

```


##### Value Aggregation

```json
{
   "aggregationType":"value",
   "aggregations":["min","max","mean"],
   "fields":["pulseId","value"],
   "range":{
      "startPulseId":0,
      "endPulseId":3
   },
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":0,
            "value":{
               "min":1.0,
               "max":4.0,
               "mean":2.5
            }
         },
         {
            "pulseId":1,
            "value":{
               "min":2.0,
               "max":5.0,
               "mean":3.5
            }
         },
         {
            "pulseId":2,
            "value":{
               "min":3.0,
               "max":6.0,
               "mean":4.5
            }
         },
         {
            "pulseId":3,
            "value":{
               "min":4.0,
               "max":7.0,
               "mean":5.5
            }
         }
      ]
   }
]

```

Array value [aggregations](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/Aggregation.java): 

![Value Aggregation](doc/images/Value_Aggregation.png)


##### Value Aggregation with Binning (nrOfBins)

```json
{
   "nrOfBins":2,
   "aggregationType":"value",
   "aggregations":["min","max","mean"],
   "fields":["pulseId","value"],
   "range":{
      "startPulseId":0,
      "endPulseId":3
   },
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"nrOfBins":2,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":0,
            "value":{
               "min":1.0,
               "max":5.0,
               "mean":3.0
            }
         },
         {
            "pulseId":2,
            "value":{
               "min":3.0,
               "max":7.0,
               "mean":5.0
            }
         }
      ]
   }
]

```

Array value [aggregations](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/Aggregation.java) with additional binning: 

![Value Aggregation with Binning](doc/images/Value_Binning_NrOfBins.png)

##### Value Aggregation with Binning (binSize)

**binSize** specifies the number of pulses per bin for pulse-range queries or the number of milliseconds per bin for time-range queries (using number of pulses and number of milliseconds makes this binning strategy consistent between channel with different update frequencies). 

```json
{
   "binSize":10,
   "aggregationType":"value",
   "aggregations":["min","max","mean"],
   "fields":["globalMillis","value"],
   "range":{
      "startSeconds":"0.0",
      "endSeconds":"0.030000000"
   },
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"binSize":10,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["globalMillis","value"],"range":{"startSeconds":"0.0","endSeconds":"0.030000000"},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "globalMillis":0,
            "value":{
               "min":1.0,
               "max":5.0,
               "mean":3.0
            }
         },
         {
            "globalMillis":20,
            "value":{
               "min":3.0,
               "max":7.0,
               "mean":5.0
            }
         }
      ]
   }
]

```

Array value [aggregations](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/Aggregation.java) with additional binning: 

![Value Aggregation with Binning](doc/images/Value_Binning_BinSize.png)

##### Index Aggregation

```json
{
   "nrOfBins":1,
   "aggregationType":"index",
   "aggregations":["min","max","mean","sum"],
   "fields":["pulseId","value"],
   "range":{
      "startPulseId":0,
      "endPulseId":3
   },
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"nrOfBins":1,"aggregationType":"index","aggregations":["min","max","mean","sum"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":0,
            "value":[
               {
                  "min":1.0,
                  "max":4.0,
                  "mean":2.5,
                  "sum":10.0
               },
               {
                  "min":2.0,
                  "max":5.0,
                  "mean":3.5,
                  "sum":14.0
               },
               {
                  "min":3.0,
                  "max":6.0,
                  "mean":4.5,
                  "sum":18.0
               },
               {
                  "min":4.0,
                  "max":7.0,
                  "mean":5.5,
                  "sum":22.0
               }
            ]
         }
      ]
   }
]
```

[Aggregation](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/Aggregation.java) of array indices with binning (several nrOfBins are also possible). 

![Index Aggregation with Binning](doc/images/Index_Binning.png)


##### Extrema Search

```json
{
   "aggregationType":"extrema",
   "aggregations":["min","max","sum"],
   "fields":["pulseId","value"],
   "range":{
      "startPulseId":0,
      "endPulseId":3
   },
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl  -H "Content-Type: application/json" -X POST -d '{"aggregationType":"extrema","aggregations":["min","max","sum"],"fields":["pulseId","value"],"range":{"startPulseId":0,"endPulseId":3},"channels":["Channel_01"]}' http://data-api.psi.ch/sf/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":{
         "minima":{
            "min":{
               "value":1.0,
               "event":{
                  "pulseId":0,
                  "value":[1,2,3,4]
               }
            },
            "sum":{
               "value":10.0,
               "event":{
                  "pulseId":0,
                  "value":[1,2,3,4]
               }
            }
         },
         "maxima":{
            "max":{
               "value":7.0,
               "event":{
                  "pulseId":3,
                  "value":[4,5,6,7]
               }
            },
            "sum":{
               "value":22.0,
               "event":{
                  "pulseId":3,
                  "value":[4,5,6,7]
               }
            }
         }
      }
   }
]
```