slsDetectorPackage/slsDetectorSoftware/generator

Generator

used to generate C++ cli commands. and bash autocompletion scripts.

Autocomplete

Overview

If any changes to enums in slsDetectorDefs

# to generate the dump.json file
cd slsSupportLib/src
clang++ -Xclang -ast-dump=json -Xclang -ast-dump-filter -Xclang StringTo -c ToString.cpp -I ../include/ -std=gnu++11 > ../../slsDetectorSoftware/generator/autocomplete/dump.json 
cd ../../slsDetectorSoftware/generator/autocomplete
python autocomplete.py -f

To Generate command line commands

python gen_commands.py -pgf

To generate autocomplete

python gen_commands.py -agf
source bash_autocomplete.sh

Always clang-format at the end

cd slsDetectorPackage/build
make format

Details

Looks through the dump.json file for the different values of an enum and stores them in the dictionary type_values.

# To print the different values for enums 
python gen_commands.py -a

also the autocomplete.py generates shell autocompletion scripts for both bash and zsh. It uses the template file bash_autocomplete.in.sh and adds the necessary code in an output file bash_autocomplete.sh (same for zsh).
To use the bash autocompletion the bash_autocomplete.sh must be sourced.

source bash_autocomplete.sh
# g <Tab><Tab> will give the list of all commands
# p 0:<Tab><Tab> will also give the list of all commands 
# g exp<Tab> will autocomplete to g exptime
# g exptime <Tab><Tab> will return "s ms us ns"
# p timing <Tab><Tab> will return "auto,burst_trigger,gating..."

Note:
The dump.json is the AST of the file slsDetectorPackage/slsSupportLib/src/ToString.cpp.

# to generate the dump.json file
cd slsSupportLib/src
clang++ -Xclang -ast-dump=json -Xclang -ast-dump-filter -Xclang StringTo -c ToString.cpp -I ../include/ -std=gnu++11 > ../../slsDetectorSoftware/generator/autocomplete/dump.json 
cd ../../slsDetectorSoftware/generator/autocomplete
python autocomplete.py -f
# clang version used: 14.0.0-1ubuntu1.1

the dump.json file produced by clang is not a correct json file because we used the -ast-dump-filter. autocomplete.py can be used to fix the format of dump.json and produce a new file called fixed.json that is json format.

# to convert dump.json into correct json format. 
python autocomplete.py -f # produces/updates the file fixed.json

Code components

• type_values is a dictionary that contains the different values for commands args. It is populated with enum values that stard with defs:: or slsDetectorDefs:: (eg. defs::burstMode). Also it contains values added manually such as "mv,mV" and those start with special::
• generate_type_values parses the AST file to find the part that contains if statements. and extracts the different possible values for an enum.
• generate_bash_autocomplete generates autocompletion scripts for bash or zsh. the difference between zsh and bash scripts can be found in the template files. (bash handles 0: completion differently than zsh more details can be found in the comments of the template files )
• fix_json fixes the file 'autocomplete/dump.json' and outputs a new corrected file in 'autocomplete/fixed.json'

Command Parser

Definitely the most important component of all the generator module.

command_parser exist to keep the commands.yaml file concise and easy to read and produce a complete version of the commands.yaml for the code generator to work on.

The goal is that the code generator works on a version of commands.yaml that is easy to iterate over and generate code.

# complete version
some_command:
    help: "do this"
    infer_action: true
    actions:
        GET:
            args:
                - argc: 0
                  function: "getCommand"
                  ...
                - argc: 1
                  function: "getCommand"
                  ...
            detectors:
                MYTHEN3:
                    - argc: 0
                      function: "getCommandMythen"
                      ...
        PUT:
            args:
                - argc: 2
                  function: "setCommand"
                  ...
                - argc: 3
                  function: "setCommand"
                  ...
    

the complete vesion can only have args or detectors field inside an action (GET or PUT). Each element in the args array have a different argc. and in each element in the args array we can find all the information needed to generate the code for that one argc case. for example in the code if(action == 'GET') and if (args.size() == 1) then we can generate the code for that one case independetly.

commands.yaml has a lot on inheritance. examples show best what it is:

fields insides an action will be passed to args and detectors

the extended args for default actions will be used for detectors

any field can be overriden

resetdacs:
  help: "[(optional) hard] ..."
  actions:
    PUT:
      function: resetToDefaultDacs
      require_det_id: true
      output: [ '"successful"' ]
      input_types: [ bool ]
      args:
        - argc: 1
          arg_types: [ special::hard ]
          input: [ '"1"' ]
        - argc: 0
          input: [ '"0"' ]

# this will be converted to 

resetdacs:
  actions:
    PUT:
      args:
      - arg_types:
        - special::hard
        argc: 1
        cast_input:
        - false
        check_det_id: false
        convert_det_id: true
        function: resetToDefaultDacs
        input:
        - '"1"'
        input_types:
        - bool
        output:
        - '"successful"'
        require_det_id: true
        store_result_in_t: false
      - arg_types: []
        argc: 0
        cast_input:
        - false
        check_det_id: false
        convert_det_id: true
        function: resetToDefaultDacs
        input:
        - '"0"'
        input_types:
        - bool
        output:
        - '"successful"'
        require_det_id: true
        store_result_in_t: false
  command_name: resetdacs
  function_alias: resetdacs
  help: "[(optional) hard] ..."
  infer_action: true

command_parser does not have a specific schema for the commands.yaml this is by design so it can be very extensible and future-proof. This also can have problems when there is typos (writing intput instead of input...)

command_parser first verifies the commands.yaml and checks if there's some obvious problems in it.

templates found in commands.yaml were taken from the CmdProxy code they were added for debugging purposes when writing the generator.

tricky things:

• if input array has n elements and cast_input array is empty. command_parser will fill it with n false values.
• store_result_in_t will be added by default as true to GET action. but as false to PUT action. (unless it is written in the action)
• infer_action by default is true
• commands that have is_description true won't be verified
• function_alias is the name of the function in the c++ code. by default it is the command name. errors came up with the command virtual as virtual is a reserved keyword in c++
• command_name is the string of the command that will be typed in cli. (frames, exptime, ...). by default it is the command name. pattern is a special keyword in yaml. problems came up with the command pattern
• arg_types is by default input_types unless otherwise specified
• when the parent has specific detector behaviour and the child does not. writing an empty detector section in the action would not inherit any detector specific fields (check exptime1)
• commands can inherit other commands (check exptime1 and exptime2)
• argc: -1 means that the command has an unknown number of arguments

Code Walkthrough

the code is well commented it is well explained in the script

Tests

tests for command_parser can be found in generator/tests/command_parser/

pip install -r requirements.txt
python -m pytests

verification is not well tested

codegen

Now for C++ code generation. After parsing the commands.yaml file and producing the extended_commands.yaml gen_commands.py will iterate over the commands and generate Caller.h, Caller.cpp, inferAction.cpp and inferAction.h .

infer action

the generated code will produce 5 new targets: "sls_detector_get sls_detector_put sls_detector_acquire sls_detector_help sls_detector"

sls_detector_get will set the action as GET
sls_detector_put will the action as PUT

sls_detector will guess the action depending on the number of arguments

the codegen module will generate a function for every command that will return the action based on the number of arguments

int InferAction::activate() {

  if (args.size() == 0) {
    return slsDetectorDefs::GET_ACTION;
  }

  if (args.size() == 1) {
    return slsDetectorDefs::PUT_ACTION;
  } else {

    throw RuntimeError("Could not infer action: Wrong number of arguments");
  }
}

the inferAction class will be called from CmdApp.cpp to infer the action and the command function will be called with the appropriate action.

some commands have the same number of argument count for both get and put. These commands can be found using the the check_infer.py script. in the generated code it will say that "sls_detector is disabled"

# to see these commands
python infer_action/check_infer.py 

Caller.cpp code

in this level we only use the extended_commands.yaml file. the generate() function in gen_commands.py will iterate over all of the commands and :

• write the function signature
• write the help
• write c++ code to check the inputs: check argument count and check if we are able to convert the arguments into the required types
• iterate over actions and arguments
• iterate over the detectors and write code for each one of them (if mythen3 ... if eiger ... else default code... ) and call codegen.write_arg() to write the argument for a single argument

codegen.write_arg()

write_arg in codegen reads the argument fields and generate c++ code accordingly.

fields explanations

• arg_types:[array of types] it is only used for autocompletion no C++ code is dependent on it
• is_description:[boolean] same as above
• template:[boolean] only used in commands.yaml and it won't present in extended_commands.yaml. it is inspired by the CmdProxy.h code
• help:[string] command help
• input:[array of variable names] the input arguments that will be passed to the function
• input_types:[array of types] the types of the input arguments given to the function
• cast_input:[array of boolean] if true it will cast the corresponding input to the type in input_types
• output:[array] outputs that will be printed (eg. ["123", "'a'"] will be os<<123<<'a')
• function: the function that will be called
• function_alias: the name of the function in the c++ code (more on it in tricky things)
• command_name: the string of the command that will be typed in cli. (more on it in tricky things)
• require_det_id: if true it will require a detector id to be passed as the last argument
• check_det_id: if true it will check the detector id and throw an error if it is not valid
• convert_det_id: if true it will convert the detector id to the correct type std::vector<int>{ det_id }
• store_result_in_t: if true it will store the result of the function in the variable t (more on it in tricky things)
• infer_action: if true it will infer the action (only if sls_detector is used)
• detectors: the detectors that have specific behaviour
• args: the arguments of the command
• argc: the number of arguments
• extra_variables[array]: each element takes three parameters: value, name, type and creates that variable in the beginning of the argument code
• exceptions[array]: each element takes two parameters: condition, message
• pattern_command: takes three arguments: nGetArgs, nPutArgs and command_name and it will write this code

  int level = -1, iArg = 0, nGetArgs = $nGetArgs$, nPutArgs = $nPutArgs$;
  GetLevelAndUpdateArgIndex(action, $command_name$, level, iArg, nGetArgs,nPutArgs);
  

• separate_time_units: takes three parameters: input, output[0], output[1] each one is a variable name

  std::string tmp_time($input$);
  std::string $output[1]$ = RemoveUnit(tmp_time);
  auto $output[0]$ = StringTo<time::ns>(tmp_time, $output[1]$);
  

• convert_to_time: takes three parameters: input[0], input[1], output

  auto output = StringTo<time::ns>(input[0], input[1]);
  

• ctb_output_list: maybe it should be removed? takes 5 parameters: GETFCNLIST, GETFCNNAME, GETFCN, suffix, printable_name