motorDriverTests/README.md

# Motor Driver Tests

This repository contains various tests for EPICS motor drivers against real
hardware on the "sinqtest" test instrument. Alternatively, it can also be used
to test the hardware itself by using it with drivers which are known to pass
the tests.

The general architecture of the framework is as follows:
- `ioc` contains all files used to create the IOC used in the tests.
- `setup` contains the motor classes, the EPICS communication interface and generalized tests (which are not tied to one specific axis). These tests are arranged in a hierarchical order:
    - `tests/.` contains tests and utility functions applicable for any motor record-based
    driver.
    - `tests/sinqMotor/.` contains tests and utility functions applicable for drivers 
    based on https://gitea.psi.ch/lin-epics-modules/sinqMotor
    - `tests/sinqMotor/turboPmac/.` contains tests and utility functions applicable for the
    sinqMotor-based Turbo PMAC driver https://gitea.psi.ch/lin-epics-modules/turboPmac
    - `tests/sinqMotor/masterMacs/.` contains tests and utility functions applicable for the
    sinqMotor-based MasterMACS driver https://gitea.psi.ch/lin-epics-modules/mastermacs
- `tests` contains the tests for the individual axes within subfolders of their respective controller folders. For example, the tests for axis "ax1" of the controller "turboPmac1" configured in `ioc/motors/turboPmac1` should be in `tests/turboPmac1/ax1`, the tests for axis "ax2" of the same controller should be in `tests/turboPmac1/ax2` and so on. If an axis folder has a prepended underscore, it is ignored by `runtests` (e.g. the tests in `tests/turboPmac1/_ax3` would be ignored). This can be used to disable tests when e.g. the axis is not available temporarily.
The classes are used to initialize the individual motors within the `tests` directory.
- `config.yaml` is the test configuration (see [Configuration](#configuration))
- `maketestenv` can be used to create a Python virtual environment for running the tests.
- `runtests` is a parallelizing `pytest` wrapper. See section for more information [Parallelizing tests over motors](#parallelizing-tests-over-motors).

## Installation

The following prerequisites need to be fulfilled:
1) EPICS 7 and the IOC shell must be installed.
2) The PSI EPICS drive `/sq_epics/ioc` must be mounted as `/ioc`.
The repository can then be simply cloned anywhere. Depending on the available hardware, the [configuration](#configuration) needs to be adjusted.

## Configuration

The test setup is defined in `config.yaml`. This file contains information which
needs to be shared between the IOC and the tests.
- `pvprefix`: EPICS PV prefix of the IOC records. This information is used to create the full
identifier of the records created by the motor drivers. 
- `versions`: Driver versions used in the IOC
- `controllers`: Configuration of the different motor controllers (IP + port and poll periods)

### Adding / removing a controller to the IOC

The controller name and its configuration needs to added under `controllers` in the following format:

```yaml
turboPmac1:
  ip: "172.28.101.24" # Controller IP adress 
  port: 1025 # Controller port 
  busypoll: 0.05 # Time in seconds
  idlepoll: 1 # Time in seconds
```

Additionally, two files need to be added under `ioc/motors`: `ioc/motors/turboPmac1.cmd` and `ioc/motors/turboPmac1.substitutions`.
These files need to be configured according to the corresponding motor driver documentation (e.g. according to https://gitea.psi.ch/lin-epics-modules/turboPmac for a TurboPMAC driver).
Lastly, `ioc/st.cmd` needs to run the `.cmd` file under motors and to import the driver. See the documentation of [sinqMotor](https://gitea.psi.ch/lin-epics-modules/sinqMotor) for details.

## Starting the IOC

It is recommended to start the IOC via `ioc/startioc`:

```bash
ioc/startioc.py
```

This file uses `config.yaml` to generate / overwrite a file `ioc/config.cmd`. It then starts `ioc/st.cmd`, which
in turn imports `ioc/config.cmd` to get the current configuration.

## Running the tests

### General

Running tests requires the following three steps:
- Creating (if not done previously) and activating a suitable virtual environment:
```bash
./maketestenv
source testenv/bin/activate
```
- Starting the IOC via `ioc/startioc` (see [Configuration](#starting-the-ioc))
- Running the desired test(s) via pytest. For example:
```bash
pytest tests/turboPmac1/
```
This runs all Turbo PMAC tests within the directory.

To run a specific test file:
```bash
pytest tests/turboPmac1/ax1/test_common.py
```

### Configuring pytest

And to run a specific test "test_something" within this file:
```bash
pytest tests/turboPmac1/ax1/test_common.py -k 'test_something'
```

Pytest normally suppresses stdout (which is where Pythons `print` writes by default).
To show it, use the `-s` flag:
```bash
pytest -s tests/turboPmac1/ax1/test_common.py
```

Two custom flags exist in this test framework: `--stresstest ` and `--log`:

#### --stresstest

Adding this flags enables some stress tests, which are essentially loops over
"normal" tests. In the source code, they are marked with
`@pytest.mark.stresstest`. Since these tests greatly increase the total runtime,
it is recommended to only run them after the "normal" tests pass.
```bash
pytest --stresstest tests/turboPmac1/ax1/test_common.py
```

#### --log

Each motor has its own dedicated logger, whose loglevel can be set with the flag
`--log=LEVEL`. `LEVEL` should be one of the `logging` - native levels - i.e.
`DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL` (see 
https://docs.python.org/3/library/logging.html#logging-levels).

In the context of this test framework, the log levels mean the following:
- `DEBUG`: Show all raw caput and caget commands
- `INFO`: Write high-level commands (e.g. move, stop, home etc.)
- `WARNING`: Not used
- `ERROR`: Serious error (e.g. motor could not be enabled / disabled)
- `CRITICAL`: Not used
All error levels higher than the defined one are forwarded to the Pytest
logger.

The default log level is `CRITICAL`.

To specify the log level for a test run:
```bash
pytest --log=INFO tests/turboPmac1/ax1/test_common.py
```

To see the log in stdout:

```bash
pytest --log=INFO -s tests/turboPmac1/ax1/test_common.py
```

### Parallelizing tests over motors

Tests which don't run on the same motor can be parallelized in order to both save on runtime
and to test controller and driver performance while handling multiple motors. To do so,
the wrapper script `runtests` is provided which starts a pytest process for each motor. 
The individual tests for each motor are run sequentially.

`runtests` accepts any arguments and forwards them to the pytest call.
If a folder is given as an argument, all tests which are not within this folder are ignored.

- `runtests` will run all tests parallelized per motor
- `runtests tests/turboPmac1` will run all Turbo PMAC tests parallelized per motor
- `runtests tests/turboPmac1/ax1` will only run the tests for motor `ax1`. These tests are run sequentially.

In addition to the parallelization feature, `runtests` also starts the IOC if it is not running already.

## Running custom scripts

The test framework can also be used to run custom scripts. The file 
`example_custom_script.py` contains a simple example, which can be run by activating
the virtual environment and then simply executing it:
```console
bash
source testenv/bin/activate
./example_custom_script.py
```
## Developer notes

### EPICS integration

The EPICS integration is currently done via `setup/caproto.py`. This module is taken from NICOS (https://github.com/mlz-ictrl/nicos/blob/master/nicos/devices/epics/pva/caproto.py)
and has been slightly modified (mainly removal of NICOS-specific decorators and the conversion of the EPICS to the NICOS status). The docstring of the `setup.classes.Motor`
class contains further information.