#Overview

This project provides a REST interface to execute queries on the databuffer.

# Requirements

This project requires Java 8 or greater.

# Deployment

Use the instructions provided by [ch.psi.daq.install](https://github.psi.ch/projects/ST/repos/ch.psi.daq.install/browse#query_rest) to install the application on a server.

## Application Properties

Following files define and describe application properties:

- [Cassandra](https://github.psi.ch/projects/ST/repos/ch.psi.daq.cassandra/browse/src/main/resources/cassandra.properties) specific properties.
- [Query](https://github.psi.ch/projects/ST/repos/ch.psi.daq.dispatcher/browse/src/main/resources/query.properties) specific properties.. 
- [Query REST](https://github.psi.ch/projects/ST/repos/ch.psi.daq.queryrest/browse/src/main/resources/queryrest.properties) specific properties.

It is possible to overwrite properties by defining new values in `${HOME}/.config/daq/queryrest.properties`

<a name="rest_interface"/>

# REST Interface

<a name="query_channel_names"/>

## Query Channel Names 

### Request

```
POST http://<host>:<port>/channels
```

#### Data

```json
{"regex": "TRFCA|TRFCB", "dbMode": "databuffer"}
```

##### Explanation

- **regex**: Reqular expression used to filter channel names (default: no filtering). Filtering is done using JAVA's [Pattern](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html), more precisely [Matcher.find()](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#find--)). 
- **dbMode**: Defines the database to access (values: **databuffer**|archiverappliance)


### Example

```bash
curl -H "Content-Type: application/json" -X POST -d '{"regex": "TRFCA|TRFCB"}' http://sf-nube-14.psi.ch:8080/channels
```

<a name="query_data"/>

## Query Data 

### Request

```
GET http://<host>:<port>/query
```

#### Data

A request is performed using JSON. The JSON query defines the channels to be queried, the range, and how the data should be aggregated (this is optional but highly recommended).

There exist following fields:

- **channels**: Array of channel names to be queried.
- **startPulseId** and **endPulseId** : A pulse-id range request with start and end pulse-id.
- **startMillis/[startNanos]** and **endMillis/[endNanos]**: A time range request with start and end milliseconds since January 1, 1970 (the UNIX/JAVA epoch) and optionally supplemented with the nanosecond offset to the milliseconds (range [0..999999]).
- **startDate/[startNanos]** and **endDate/[endNanos]**: A time range request with start and end date (format yyyy/MM/dd HH:mm:ss.SSS or dd.MM.yyyy HH:mm:ss.SSS) and optionally supplemented with the nanosecond offset to the milliseconds (range [0..999999]).
- **ordering**: The ordering of the data (see [here](https://github.psi.ch/projects/ST/repos/ch.psi.daq.common/browse/src/main/java/ch/psi/daq/common/ordering/Ordering.java) for possible values).
- **fields**: The requested fields (see [here](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/QueryField.java) for possible values).
- **nrOfBins**: Activates data binning. Specifies the number of bins the pulse/time range should be devided into.
- **binSize**: Activates data binning. Specifies the number of pulses per bin for pulse-range queries or the number of milliseconds per bin for time-range queries. 
- **aggregations**: Activates data aggregation. Array of requested aggregations (see [here](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/Aggregation.java) for possible values). These values will be added to the *data* array response.
- **aggregationType**: Specifies the type of aggregation (see [here](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/AggregationType.java)). The default type is *value* aggregation (e.g., sum([1,2,3])=6). Alternatively, it is possible to define *index* aggregation for multiple arrays in combination with binning (e.g., sum([1,2,3], [3,2,1]) = [4,4,4]).
- **aggregateChannels**: Specifies whether the data of the requested channels should be combined together using the defined aggregation (values: true|**false**) 
- **dbMode**: Defines the database to access (values: **databuffer**|archiverappliance)


### Example

```bash
curl -H "Content-Type: application/json" -X POST -d '{"channels":["channel1","channel2"],"startPulseId":0,"endPulseId":4}' http://sf-nube-14.psi.ch:8080/query
```

### Response example

The response is in JSON. 

```json
[
   {
      "channel":"channel1",
      "data":[
         {
            "pulseId":0,
            "iocMillis":0,
            "iocNanos":0,
            "globalMillis":0,
            "globalNanos":0,
            "value":0
         },
         {
            "pulseId":2,
            "iocMillis":2,
            "iocNanos":2,
            "globalMillis":2,
            "globalNanos":2,
            "value":2
         },
         {
            "pulseId":4,
            "iocMillis":4,
            "iocNanos":4,
            "globalMillis":4,
            "globalNanos":4,
            "value":4
         }
      ]
   },
   {
      "channel":"channel2",
      "data":[
         {
            "pulseId":1,
            "iocMillis":1,
            "iocNanos":1,
            "globalMillis":1,
            "globalNanos":1,
            "value":1
         },
         {
            "pulseId":3,
            "iocMillis":3,
            "iocNanos":3,
            "globalMillis":3,
            "globalNanos":3,
            "value":3
         }
      ]
   }
]
```

### Example Queries

Following examples build on a waveform data (see below). They also work for scalars (consider it as a waveform of length = 1) and images (waveform of length = dimX * dimY). 

![Data Visualization](doc/images/Data_Visualization.png)

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "iocMillis":0,
            "iocNanos":0,
            "pulseId":0,
            "globalMillis":0,
            "globalNanos":0,
            "shape":[
               4
            ],
            "value":[1,2,3,4]
         },
         {
            "iocMillis":10,
            "iocNanos":0,
            "pulseId":1,
            "globalMillis":10,
            "globalNanos":0,
            "shape":[
               4
            ],
            "value":[2,3,4,5]
         },
         {
            "iocMillis":20,
            "iocNanos":0,
            "pulseId":2,
            "globalMillis":20,
            "globalNanos":0,
            "shape":[
               4
            ],
            "value":[3,4,5,6]
         },
         {
            "iocMillis":30,
            "iocNanos":0,
            "pulseId":3,
            "globalMillis":30,
            "globalNanos":0,
            "shape":[
               4
            ],
            "value":[4,5,6,7]
         }
      ]
   }
]
```

### Query Examples

##### Query by Pulse-Id Range

```json
{
   "startPulseId":0,
   "endPulseId":3,
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"startPulseId":0,"endPulseId":3,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

See JSON representation of the data above. 



##### Query by Time Range

```json
{
   "startMillis":0,
   "startNanos":0,
   "endMillis":30,
   "endNanos":999999,
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"startMillis":0,"startNanos":0,"endMillis":30,"endNanos":999999,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

See JSON representation of the data above. 



##### Query by Date Range

```json
{
   "startDate":"1970/01/01 01:00:00.000",
   "startNanos":0,
   "endDate":"1970/01/01 01:00:00.030",
   "endNanos":999999,
   "channels":[
      "Channel_01"
   ]
}
```

Supported formats: *yyyy/MM/dd HH:mm:ss.SSS* and *dd.MM.yyyy HH:mm:ss.SSS*

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"startDate":"1970/01/01 01:00:00.000","startNanos":0,"endDate":"1970/01/01 01:00:00.030","endNanos":999999,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

See JSON representation of the data above. 



##### Querying Archiver Appliance

```json
{
   "dbmode":"archiverappliance",
   "startMillis":0,
   "startNanos":0,
   "endMillis":30,
   "endNanos":999999,
   "channels":[
      "Channel_01"
   ]
}
```

Archiver Appliance supports queries by *time range* and *date range* only (as it has no notion about pulse-id).


###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"dbmode":"archiverappliance","startMillis":0,"startNanos":0,"endMillis":30,"endNanos":999999,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

See JSON representation of the data above. 



##### Querying for Specific Fields

Allows for server side optimizations since not all data needs to be retrieved.

```json
{
   "fields":["pulseId","value"]
   "startPulseId":0,
   "endPulseId":3,
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"fields":["pulseId","value"],"startPulseId":0,"endPulseId":3,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":0,
            "value":[1,2,3,4]
         },
         {
            "pulseId":1,
            "value":[2,3,4,5]
         },
         {
            "pulseId":2,
            "value":[3,4,5,6]
         },
         {
            "pulseId":3,
            "value":[4,5,6,7]
         }
      ]
   }
]
```

##### Data Ordering

```json
{
   "ordering":"desc",
   "fields":["pulseId","value"]
   "startPulseId":0,
   "endPulseId":3,
   "channels":[
      "Channel_01"
   ]
}
```

Use **none** in case ordering does not matter (allows for server side optimizations).


###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"ordering":"desc","fields":["pulseId","value"],"startPulseId":0,"endPulseId":3,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":3,
            "value":[4,5,6,7]
         },
         {
            "pulseId":2,
            "value":[3,4,5,6]
         },
         {
            "pulseId":1,
            "value":[2,3,4,5]
         },
         {
            "pulseId":0,
            "value":[1,2,3,4]
         }
      ]
   }
]

```


##### Value Aggregation

```json
{
   "aggregationType":"value",
   "aggregations":["min","max","mean"],
   "fields":["pulseId","value"]
   "startPulseId":0,
   "endPulseId":3,
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"startPulseId":0,"endPulseId":3,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":0,
            "value":{
               "min":1.0,
               "max":4.0,
               "mean":2.5
            }
         },
         {
            "pulseId":1,
            "value":{
               "min":2.0,
               "max":5.0,
               "mean":3.5
            }
         },
         {
            "pulseId":2,
            "value":{
               "min":3.0,
               "max":6.0,
               "mean":4.5
            }
         },
         {
            "pulseId":3,
            "value":{
               "min":4.0,
               "max":7.0,
               "mean":5.5
            }
         }
      ]
   }
]

```

Array value [aggregations](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/Aggregation.java). 

![Value Aggregation](doc/images/Value_Aggregation.png)


##### Value Aggregation with Binning

```json
{
   "nrOfBins":2,
   "aggregationType":"value",
   "aggregations":["min","max","mean"],
   "fields":["pulseId","value"]
   "startPulseId":0,
   "endPulseId":3,
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"nrOfBins":2,"aggregationType":"value","aggregations":["min","max","mean"],"fields":["pulseId","value"],"startPulseId":0,"endPulseId":3,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":0,
            "value":{
               "min":1.0,
               "max":5.0,
               "mean":3.0
            }
         },
         {
            "pulseId":2,
            "value":{
               "min":3.0,
               "max":7.0,
               "mean":5.0
            }
         }
      ]
   }
]

```

Array value [aggregation](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/Aggregation.java) with additional binning. 

![Value Aggregation with Binning](doc/images/Value_Binning.png)

##### Index Aggregation

```json
{
   "nrOfBins":1,
   "aggregationType":"index",
   "aggregations":["min","max","mean","sum"],
   "fields":["pulseId","value"]
   "startPulseId":0,
   "endPulseId":3,
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"nrOfBins":1,"aggregationType":"index","aggregations":["min","max","mean","sum"],"fields":["pulseId","value"],"startPulseId":0,"endPulseId":3,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":[
         {
            "pulseId":0,
            "value":[
               {
                  "min":1.0,
                  "max":4.0,
                  "mean":2.5,
                  "sum":10.0
               },
               {
                  "min":2.0,
                  "max":5.0,
                  "mean":3.5,
                  "sum":14.0
               },
               {
                  "min":3.0,
                  "max":6.0,
                  "mean":4.5,
                  "sum":18.0
               },
               {
                  "min":4.0,
                  "max":7.0,
                  "mean":5.5,
                  "sum":22.0
               }
            ]
         }
      ]
   }
]
```

[Aggregation](https://github.psi.ch/projects/ST/repos/ch.psi.daq.query/browse/src/main/java/ch/psi/daq/query/model/Aggregation.java) of array indices with binning (several nrOfBins are also possible). 

![Index Aggregation with Binning](doc/images/Index_Binning.png)


##### Extrema Search

```json
{
   "aggregationType":"extrema",
   "aggregations":["min","max","sum"],
   "fields":["pulseId","value"]
   "startPulseId":0,
   "endPulseId":3,
   "channels":[
      "Channel_01"
   ]
}
```

###### Command

```bash
curl -H "Content-Type: application/json" -X POST -d '{"aggregationType":"extrema","aggregations":["min","max","sum"],"fields":["pulseId","value"],"startPulseId":0,"endPulseId":3,"channels":["Channel_01"]}' http://sf-nube-14.psi.ch:8080/query
```

###### Response

```json
[
   {
      "channel":"Channel_01",
      "data":{
         "minima":{
            "min":{
               "value":1.0,
               "event":{
                  "pulseId":0,
                  "value":[1,2,3,4]
               }
            },
            "sum":{
               "value":10.0,
               "event":{
                  "pulseId":0,
                  "value":[1,2,3,4]
               }
            }
         },
         "maxima":{
            "max":{
               "value":7.0,
               "event":{
                  "pulseId":3,
                  "value":[4,5,6,7]
               }
            },
            "sum":{
               "value":22.0,
               "event":{
                  "pulseId":3,
                  "value":[4,5,6,7]
               }
            }
         }
      }
   }
]
```