detectors/slsDetectorPackage
25
0
Fork 0
mirror of https://github.com/slsdetectorgroup/slsDetectorPackage.git synced 2025-06-23 10:07:59 +02:00
Code Issues Actions Packages Releases Activity

commands code generation (#871)

Browse Source 
* commands code generation  (#803)

* commands code generation for only frames command

* fix cmake file and add Caller files

* working exptime, fully extended commands file and its variants

* start adding template commands

* add INT_CMD_VEC_ID template

* add list command, generate multiple bins, format code

* reach 208 commands using the cpp macros

* add tests for command parser

* start adding tests for commands parser

* fix typo to use commands.yaml

* add more tests for command_parser

* add all template functions (up to 218 commands)

* finish template functions and add more CmdProxy.cpp functions (250+)

* 257 commands

* 300 commands the rest are very special commands

* add special commands without generation

* separate special functions from generated c++ file

* implementing one command for put and get (buggy)

* add infer action in a separate file

* generate header for special commands from yaml

* allow only 0 or 1 for bool inputs

* group all commands in gen_commands.py

* add help to gen_commands.py

* add autocomplete bash script

* autocompletion: add support for module levels and help

* remove debugging line

* add autocompletion, help to commands, change int [0,1] to bool

* copy tests for Caller.cpp. Tests pass

* update with the new developer branch changes

* fix errors after merging (there is problems with tests)

* fixed port/stopport in yaml (intput typo), added '_caller' to the test dac and test on chip dac command in global test for cmdcaller

* undo previous test simulator debug change

* add documentation for the generated code

* reducing the comment to be replaced in length so formatting does not split into 2 lines

* removed formatting specific style of C++11 in gen_commands.py to keep with the top level clang format of the project
* regeneratign code for commands

* automation generated

* Redirect deprecated commands (#872)

* working implementation, need to fix dac

* fixed deprecation redirect for dac command

* Detector specific autocomplete (#873)

* working implementation, need to fix dac

* fixed deprecation redirect for dac command

* detector specific completion for dac

* added autocomplete using detector specific

* fixed error when autocompleting partial words

* Generate commands/fix commands (#875)

* fix vm_a, im_a etc have deg Celsius suffix, also help missing or changed in some places

* dac: require det id for all, arg0 to be printed at output, help for onchip dac and dac, onchipdac: spacing

* getscan detid and blocking trigger help

* udp_Dstlist det_id fixed, but rx_id invalid

* cmdApp in line with cmdLineApp (missing version, receiver_id, not creating det object in help action

* added set_command to differentiate between check_det_id and require_det_id (mixed up), args: -1 needs to check for at least one argument

* reordering

* reordering and checked till integer_command_hex

* fixed a lot more commands

* fix caller tests for eiger

* changes to tests after Bechir left

* changing .cmd to .cmdcall for the caller commands

* fixed tests for caller, still warning for setexptime about cast input

* autocomplete ran

* add moench test

* regenerating autocomplete and commands

* fixing other things from merge conflicts (renaming slsDetectorDefs to defs in commands.yaml)

* formatting

* added code injection to help (#876)

* updated 3 commands to have get output that can be put into put (#877)

* updated some commands to have get output that can be put into put

* fix tests for clkdiv

* adding help to free (#878)

* removing old commands and renaming them, (also making it work for parameters command as it was still calling cmdproxy) (#879)

* More helpful error messages for unsupported actions (#880)

* removing old commands and renaming them, (also making it work for parameters command as it was still calling cmdproxy)

* Added specific help for unsupported actions

* fixed a vetofile get special exception message. more specific warning for special exception message instead of no function warning

* added condition checking true in exceptions for special message

---------
Co-authored-by: Bechir Brahem <bachbrahem@gmail.com>
Co-authored-by: Erik Frojdh <erik.frojdh@gmail.com>
Co-authored-by: Dhanya Thattil <dhanya.thattil@psi.ch>
This commit is contained in:
maliakal_d
2023-12-13 14:43:38 +01:00
committed by GitHub
parent d72c9e29a4
commit ce7270e8a2
63 changed files with 180123 additions and 9134 deletions
Download Patch File Download Diff File Expand all files Collapse all files

288
slsDetectorSoftware/generator/readme.md Normal file
View File

@ -0,0 +1,288 @@
# Generator
used to generate C++ cli commands. and bash autocompletion scripts.
## Autocomplete
### Overview
Looks through the `dump.json` file for the different values of an enum and stores them in the dictionary `type_values`.
```sh
# To print the different values for enums
python gen_commands.py -a
```
also the autocomplete.py generates shell autocompletion scripts for both bash and zsh. It uses the template file `bash_autocomplete.in.sh` and adds the necessary code in an output file `bash_autocomplete.sh` (same for zsh).
To use the bash autocompletion the `bash_autocomplete.sh` must be sourced.
```sh
source bash_autocomplete.sh
# g <Tab><Tab> will give the list of all commands
# p 0:<Tab><Tab> will also give the list of all commands
# g exp<Tab> will autocomplete to g exptime
# g exptime <Tab><Tab> will return "s ms us ns"
# p timing <Tab><Tab> will return "auto,burst_trigger,gating..."
```
**Note:**
The dump.json is the AST of the file `slsDetectorPackage/slsSupportLib/src/ToString.cpp`.
```sh
# to generate the dump.json file
cd slsSupportLib/src/ToString.cpp
clang++ -Xclang -ast-dump=json -Xclang -ast-dump-filter -Xclang StringTo -c ToString.cpp -I ../include/ -std=gnu++11
# clang version used: 14.0.0-1ubuntu1.1
```
the `dump.json` file produced by clang is not a correct json file because we used the `-ast-dump-filter`. autocomplete.py can be used to fix the format of `dump.json` and produce a new file called `fixed.json` that is json format.
```
# to convert dump.json into correct json format.
python autocomplete.py -f # produces the file fixed.json
```
### Code components
- `type_values` is a dictionary that contains the different values for commands args. It is populated with enum values that stard with defs:: or slsDetectorDefs:: (eg. defs::burstMode). Also it contains values added manually such as "mv,mV" and those start with special::
- `generate_type_values` parses the AST file to find the part that contains if statements. and extracts the different possible values for an enum.
- `generate_bash_autocomplete` generates autocompletion scripts for bash or zsh. the difference between zsh and bash scripts can be found in the template files. (bash handles 0:<Tab><Tab> completion differently than zsh more details can be found in the comments of the template files )
- `fix_json` fixes the file 'autocomplete/dump.json' and outputs a new corrected file in 'autocomplete/fixed.json'
## Command Parser
Definitely the most important component of all the generator module.
command_parser exist to keep the commands.yaml file concise and easy to read and produce a complete version of the commands.yaml for the code generator to work on.
The goal is that the code generator works on a version of commands.yaml that is easy to iterate over and generate code.
```
# complete version
some_command:
help: "do this"
infer_action: true
actions:
GET:
args:
- argc: 0
function: "getCommand"
...
- argc: 1
function: "getCommand"
...
detectors:
MYTHEN3:
- argc: 0
function: "getCommandMythen"
...
PUT:
args:
- argc: 2
function: "setCommand"
...
- argc: 3
function: "setCommand"
...
```
the complete vesion can only have `args` or `detectors` field inside an action (GET or PUT). **Each element in the args array have a different argc**. and in each element in the args array we can find all the information needed to generate the code for that one argc case. for example in the code `if(action == 'GET')` and `if (args.size() == 1)` then we can generate the code for that one case independetly.
commands.yaml has a lot on ~inheritance~. examples show best what it is:
> fields insides an action will be passed to args and detectors
> the extended args for default actions will be used for detectors
> any field can be overriden
```
resetdacs:
help: "[(optional) hard] ..."
actions:
PUT:
function: resetToDefaultDacs
require_det_id: true
output: [ '"successful"' ]
input_types: [ bool ]
args:
- argc: 1
arg_types: [ special::hard ]
input: [ '"1"' ]
- argc: 0
input: [ '"0"' ]
# this will be converted to
resetdacs:
actions:
PUT:
args:
- arg_types:
- special::hard
argc: 1
cast_input:
- false
check_det_id: false
convert_det_id: true
function: resetToDefaultDacs
input:
- '"1"'
input_types:
- bool
output:
- '"successful"'
require_det_id: true
store_result_in_t: false
- arg_types: []
argc: 0
cast_input:
- false
check_det_id: false
convert_det_id: true
function: resetToDefaultDacs
input:
- '"0"'
input_types:
- bool
output:
- '"successful"'
require_det_id: true
store_result_in_t: false
command_name: resetdacs
function_alias: resetdacs
help: "[(optional) hard] ..."
infer_action: true
```
command_parser does not have a specific schema for the commands.yaml this is by design so it can be very extensible and future-proof. This also can have problems when there is typos (writing intput instead of input...)
command_parser first verifies the commands.yaml and checks if there's some obvious problems in it.
templates found in commands.yaml were taken from the CmdProxy code they were added for debugging purposes when writing the generator.
tricky things:
--
- if input array has n elements and cast_input array is empty. command_parser will fill it with n false values.
- store_result_in_t will be added by default as true to GET action. but as false to PUT action. (unless it is written in the action)
- infer_action by default is true
- commands that have is_description true won't be verified
- function_alias is the name of the function in the c++ code. by default it is the command name. errors came up with the command virtual as virtual is a reserved keyword in c++
- command_name is the string of the command that will be typed in cli. (frames, exptime, ...). by default it is the command name. pattern is a special keyword in yaml. problems came up with the command pattern
- arg_types is by default input_types unless otherwise specified
- when the parent has specific detector behaviour and the child does not. writing an empty detector section in the action would not inherit any detector specific fields (check exptime1)
- commands can inherit other commands (check exptime1 and exptime2)
- argc: -1 means that the command has an unknown number of arguments
### Code Walkthrough
the code is well commented it is well explained in the script
### Tests
tests for command_parser can be found in `generator/tests/command_parser/`
```
pip install -r requirements.txt
python -m pytests
```
verification is not well tested
## codegen
Now for C++ code generation. After parsing the commands.yaml file and producing the extended_commands.yaml `gen_commands.py` will iterate over the commands and generate `Caller.h`, `Caller.cpp`, `inferAction.cpp` and `inferAction.h` .
### infer action
the generated code will produce 5 new targets: "detg detp deta deth det"
`detg` will set the action as GET
`detp` will the action as PUT
`det` will guess the action depending on the number of arguments
the codegen module will generate a function for every command that will return the action based on the number of arguments
```cpp
int InferAction::activate() {
if (args.size() == 0) {
return slsDetectorDefs::GET_ACTION;
}
if (args.size() == 1) {
return slsDetectorDefs::PUT_ACTION;
} else {
throw RuntimeError("Could not infer action: Wrong number of arguments");
}
}
```
the `inferAction` class will be called from `CmdApp.cpp` to infer the action and the command function will be called with the appropriate action.
some commands have the same number of argument count for both get and put. These commands can be found using the the `check_infer.py` script. in the generated code it will say that "det is disabled"
```bash
# to see these commands
python infer_action/check_infer.py
```
### Caller.cpp code
in this level we only use the extended_commands.yaml file.
the `generate()` function in `gen_commands.py` will iterate over all of the commands and :
- write the function signature
- write the help
- write c++ code to check the inputs: check argument count and check if we are able to convert the arguments into the required types
- iterate over actions and arguments
- iterate over the detectors and write code for each one of them (if mythen3 ... if eiger ... else default code... ) and call `codegen.write_arg()` to write the argument for a single argument
codegen.write_arg()
-
write_arg in codegen reads the argument fields and generate c++ code accordingly.
## fields explanations
- arg_types:[array of types] it is only used for autocompletion no C++ code is dependent on it
- is_description:[boolean] same as above
- template:[boolean] only used in commands.yaml and it won't present in extended_commands.yaml. it is inspired by the CmdProxy.h code
- help:[string] command help
- input:[array of variable names] the input arguments that will be passed to the function
- input_types:[array of types] the types of the input arguments given to the function
- cast_input:[array of boolean] if true it will cast the corresponding input to the type in input_types
- output:[array] outputs that will be printed (eg. ["123", "'a'"] will be os<<123<<'a')
- function: the function that will be called
- function_alias: the name of the function in the c++ code (more on it in tricky things)
- command_name: the string of the command that will be typed in cli. (more on it in tricky things)
- require_det_id: if true it will require a detector id to be passed as the last argument
- check_det_id: if true it will check the detector id and throw an error if it is not valid
- convert_det_id: if true it will convert the detector id to the correct type `std::vector<int>{ det_id }`
- store_result_in_t: if true it will store the result of the function in the variable t (more on it in tricky things)
- infer_action: if true it will infer the action (only if det is used)
- detectors: the detectors that have specific behaviour
- args: the arguments of the command
- argc: the number of arguments
- extra_variables[array]: each element takes three parameters: value, name, type and creates that variable in the beginning of the argument code
- exceptions[array]: each element takes two parameters: condition, message
- pattern_command: takes three arguments: nGetArgs, nPutArgs and command_name and it will write this code
```cpp
int level = -1, iArg = 0, nGetArgs = $nGetArgs$, nPutArgs = $nPutArgs$;
GetLevelAndUpdateArgIndex(action, $command_name$, level, iArg, nGetArgs,nPutArgs);
```
- separate_time_units: takes three parameters: input, output[0], output[1] each one is a variable name
```cpp
std::string tmp_time($input$);
std::string $output[1]$ = RemoveUnit(tmp_time);
auto $output[0]$ = StringTo<time::ns>(tmp_time, $output[1]$);
```
- convert_to_time: takes three parameters: input[0], input[1], output
```cpp
auto output = StringTo<time::ns>(input[0], input[1]);
```
- ctb_output_list: **maybe it should be removed?** takes 5 parameters: GETFCNLIST, GETFCNNAME, GETFCN, suffix, printable_name