commands code generation (#871)

* commands code generation  (#803)

* commands code generation for only frames command

* fix cmake file and add Caller files

* working exptime, fully extended commands file and its variants

* start adding template commands

* add INT_CMD_VEC_ID template

* add list command, generate multiple bins, format code

* reach 208 commands using the cpp macros

* add tests for command parser

* start adding tests for commands parser

* fix typo to use commands.yaml

* add more tests for command_parser

* add all template functions (up to 218 commands)

* finish template functions and add more CmdProxy.cpp functions (250+)

* 257 commands

* 300 commands the rest are very special commands

* add special commands without generation

* separate special functions from generated c++ file

* implementing one command for put and get (buggy)

* add infer action in a separate file

* generate header for special commands from yaml

* allow only 0 or 1 for bool inputs

* group all commands in gen_commands.py

* add help to gen_commands.py

* add autocomplete bash script

* autocompletion: add support for module levels and help

* remove debugging line

* add autocompletion, help to commands, change int [0,1] to bool

* copy tests for Caller.cpp. Tests pass

* update with the new developer branch changes

* fix errors after merging (there is problems with tests)

* fixed port/stopport in yaml (intput typo), added '_caller' to the test dac and test on chip dac command in global test for cmdcaller

* undo previous test simulator debug change

* add documentation for the generated code

* reducing the comment to be replaced in length so formatting does not split into 2 lines

* removed formatting specific style of C++11 in gen_commands.py to keep with the top level clang format of the project
* regeneratign code for commands

* automation generated

* Redirect deprecated commands (#872)

* working implementation, need to fix dac

* fixed deprecation redirect for dac command

* Detector specific autocomplete (#873)

* working implementation, need to fix dac

* fixed deprecation redirect for dac command

* detector specific completion for dac

* added autocomplete using detector specific

* fixed error when autocompleting partial words

* Generate commands/fix commands (#875)

* fix vm_a, im_a etc have deg Celsius suffix, also help missing or changed in some places

* dac: require det id for all, arg0 to be printed at output, help for onchip dac and dac, onchipdac: spacing

* getscan detid and blocking trigger help

* udp_Dstlist det_id fixed, but rx_id invalid

* cmdApp in line with cmdLineApp (missing version, receiver_id, not creating det object in help action

* added set_command to differentiate between check_det_id and require_det_id (mixed up), args: -1 needs to check for at least one argument

* reordering

* reordering and checked till integer_command_hex

* fixed a lot more commands

* fix caller tests for eiger

* changes to tests after Bechir left

* changing .cmd to .cmdcall for the caller commands

* fixed tests for caller, still warning for setexptime about cast input

* autocomplete ran

* add moench test

* regenerating autocomplete and commands

* fixing other things from merge conflicts (renaming slsDetectorDefs to defs in commands.yaml)

* formatting

* added code injection to help (#876)

* updated 3 commands to have get output that can be put into put (#877)

* updated some commands to have get output that can be put into put

* fix tests for clkdiv

* adding help to free (#878)

* removing old commands and renaming them, (also making it work for parameters command as it was still calling cmdproxy) (#879)

* More helpful error messages for unsupported actions (#880)

* removing old commands and renaming them, (also making it work for parameters command as it was still calling cmdproxy)

* Added specific help for unsupported actions

* fixed a vetofile get special exception message. more specific warning for special exception message instead of no function warning

* added condition checking true in exceptions for special message

---------
Co-authored-by: Bechir Brahem <bachbrahem@gmail.com>
Co-authored-by: Erik Frojdh <erik.frojdh@gmail.com>
Co-authored-by: Dhanya Thattil <dhanya.thattil@psi.ch>
This commit is contained in:
2023-12-13 14:43:38 +01:00
committed by GitHub
parent d72c9e29a4
commit ce7270e8a2
63 changed files with 180123 additions and 9134 deletions

View File

@ -0,0 +1,236 @@
import argparse
import json
from pathlib import Path
# command to generate the ast
# clang version: 14.0.0-1ubuntu1.1
# clang++ -Xclang -ast-dump=json -Xclang -ast-dump-filter -Xclang StringTo -c ToString.cpp -I ../include/ -std=gnu++11
#
import yaml
AUTOCOMPLETE_PATH = Path(__file__).parent
DUMP_PATH = AUTOCOMPLETE_PATH / 'dump.json'
FIXED_PATH = AUTOCOMPLETE_PATH / 'fixed.json'
type_values = {
'special::mv': ["mv", "mV"],
"special::deg": ["deg"],
"special::time_unit": ["s", "ms", "us", "ns"],
"special::hard": ["hard"],
"special::force-delete-normal-file": ["--force-delete-normal-file"],
"special::currentSourceFix": ["fix", "nofix"],
"special::currentSourceLow": ["normal", "low"],
"special::path": [],
"special::pedestal_parameters" : ["", "0"]
}
def get_types(arg_types):
ret = set()
for arg_type in arg_types:
if type_info(arg_type) == 'base':
if arg_type == 'bool':
ret = ret.union(["0", "1"])
else:
tmp = [not_list for not_list in type_values[arg_type] if not isinstance(not_list, list)]
ret = ret.union(tmp)
#Intercept the options and in case detector specific options appear replace the
#list of options with a command line call that fetches them
#TODO! Rename detg
if "defs::dacIndex" in arg_types:
return "`detg daclist | sed -e 's/.*\[\(.*\)\].*/\\1/' | sed 's/,//g'`"
elif "defs::detectorSettings" in arg_types:
return "`detg settingslist | sed -e 's/.*\[\(.*\)\].*/\\1/' | sed 's/,//g'`"
elif "defs::timingMode" in arg_types:
return "`detg timinglist | sed -e 's/.*\[\(.*\)\].*/\\1/' | sed 's/,//g'`"
return ret
def type_info(type_name):
if type_name.startswith('defs::') or type_name.startswith('slsDetectorDefs::'):
return 'enum'
if type_name.startswith('special::'):
return 'special'
return 'base'
def get_enum(function):
return function['type']['qualType'].split(' ')[0]
def get_literal(ifstmt):
stringliteral = []
expression = ifstmt['inner'][0]
if expression['kind'] == 'BinaryOperator':
if expression['opcode'] == '!=':
return None, None
for cxxOperatorCall in expression['inner']:
if cxxOperatorCall['kind'] == 'CXXOperatorCallExpr':
implicitCastExpr = cxxOperatorCall['inner'][2]
stringliteral.append(implicitCastExpr['inner'][0]['value'][1:-1])
else:
cxxOperatorCall = expression
implicitCastExpr = cxxOperatorCall['inner'][2]
stringliteral = implicitCastExpr['inner'][0]['value'][1:-1]
retstmt = get_object_by_kind(ifstmt['inner'], 'ReturnStmt')
declrefexpt = get_object_by_kind(retstmt['inner'], 'DeclRefExpr')
enum_val = declrefexpt["referencedDecl"]["name"]
return enum_val, stringliteral
def get_object_by_kind(inner, kind, position=1):
for obj in inner:
if obj['kind'] == kind:
position -= 1
if position == 0:
return obj
return None
def generate_type_values():
functions = json.loads(FIXED_PATH.read_text())
for function in functions:
if function['kind'] != 'FunctionDecl' or function['name'] != 'StringTo':
continue
enum = get_enum(function)
if not enum.startswith('defs::'):
continue
# if enum != 'defs::dacIndex':
# continue
if not function['loc']['file'].endswith('ToString.cpp'):
continue
compound_stmt = get_object_by_kind(function['inner'], 'CompoundStmt')
for ifstmt in compound_stmt['inner']:
if ifstmt['kind'] != 'IfStmt':
continue
enum_val, stringliteral = get_literal(ifstmt)
if enum_val is None:
continue
if enum not in type_values or type_values[enum] is None:
type_values[enum] = []
type_values[enum].append(stringliteral)
items = list(type_values.items())
for key, val in items:
if key.startswith('defs::'):
new_key = key.split('::')[1]
new_key = 'slsDetectorDefs::' + new_key
type_values[new_key] = val
elif key.startswith('slsDetectorDefs::'):
new_key = key.split('::')[1]
new_key = 'defs::' + new_key
type_values[new_key] = val
return json.dumps(type_values, indent=2)
def fix_json():
with DUMP_PATH.open('r') as f:
tmp = '[\n'
for line in f.read().split('\n'):
if line.startswith('}'):
tmp += line + ',\n'
else:
tmp += line + '\n'
tmp = tmp[:-3] + '\n]'
with FIXED_PATH.open('w') as f:
f.write(tmp)
def generate_bash_autocomplete(output_path=Path(__file__).parent / 'bash_autocomplete.sh', input_path=Path(__file__).parent / 'bash_autocomplete.in.sh'):
generate_type_values()
output_file = output_path.open('w')
template_file = input_path.open('r')
def writeline(line):
output_file.write(line + '\n')
class if_block:
def __init__(self, condition):
self.condition = condition
def __enter__(self):
output_file.write('if [[ ' + self.condition + ' ]]; then\n')
def __exit__(self, type, value, traceback):
output_file.write('fi\n')
class function:
def __init__(self, name):
self.name = name
def __enter__(self):
output_file.write(self.name + '() {\n')
def __exit__(self, type, value, traceback):
output_file.write('}\n')
command_path = Path(__file__).parent.parent / 'extended_commands.yaml'
commands = yaml.unsafe_load(command_path.open('r'))
for line in template_file:
if '-- THIS LINE WILL BE REPLACED WITH GENERATED CODE --' not in line:
output_file.write(line)
continue
writeline(f'local SLS_COMMANDS=" {" ".join(commands.keys())} "')
# generate functions
for command_name, command in commands.items():
# added for debugging
if command_name == 'xxxexptime':
continue
with function('__' + command_name):
writeline('FCN_RETURN=""')
actions = ['GET', 'PUT']
for action in actions:
if action in command['actions'] and 'args' in command['actions'][action]:
args = command['actions'][action]['args']
possible_argc = {}
for arg in args:
if arg['argc'] == 0:
pass
for i in range(arg['argc']):
if i + 1 not in possible_argc:
possible_argc[i + 1] = []
possible_argc[i + 1].append(arg['arg_types'][i])
if possible_argc:
with if_block(f'${{IS_GET}} -eq {"1" if action == "GET" else "0"}'):
for argc in possible_argc:
with if_block(f'"${{cword}}" == "{argc + 1}"'):
if "defs::detectorSettings" in possible_argc[argc]:
print(argc, command_name, possible_argc[argc])
choices = get_types(possible_argc[argc])
#check if we got choices back or a bash command
if isinstance(choices, (list,set)):
writeline(f'FCN_RETURN="{" ".join(sorted(choices))}"')
else:
writeline(f'FCN_RETURN="{choices}"')
if 'special::path' in possible_argc[argc]:
writeline('IS_PATH=1')
writeline('return 0')
output_file.close()
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='use parsed c++ code to generate autocomplete snippets')
parser.add_argument('-f', '--fix', action='store_true', help='fix the parsed ast to make it loadable')
# parser.add_argument('-p', '--path', type=str, help='output path to the fixed ast', default='ast.json')
args = parser.parse_known_args()
if args[0].fix:
fix_json()
ret = generate_type_values()
print(ret)

View File

@ -0,0 +1,164 @@
# GENERATED FILE - DO NOT EDIT
# ANY CHANGES TO THIS FILE WILL BE OVERWRITTEN
_sd() {
# Taken from https://github.com/scop/bash-completion/blob/15b74b1050333f425877a7cbd99af2998b95c476/bash_completion#L770C12-L770C12
# Reassemble command line words, excluding specified characters from the
# list of word completion separators (COMP_WORDBREAKS).
# @param $1 chars Characters out of $COMP_WORDBREAKS which should
# NOT be considered word breaks. This is useful for things like scp where
# we want to return host:path and not only path, so we would pass the
# colon (:) as $1 here.
# @param $2 words Name of variable to return words to
# @param $3 cword Name of variable to return cword to
#
_comp__reassemble_words()
{
local exclude="" i j line ref
# Exclude word separator characters?
if [[ $1 ]]; then
# Yes, exclude word separator characters;
# Exclude only those characters, which were really included
exclude="[${1//[^$COMP_WORDBREAKS]/}]"
fi
# Default to cword unchanged
printf -v "$3" %s "$COMP_CWORD"
# Are characters excluded which were former included?
if [[ $exclude ]]; then
# Yes, list of word completion separators has shrunk;
line=$COMP_LINE
# Re-assemble words to complete
for ((i = 0, j = 0; i < ${#COMP_WORDS[@]}; i++, j++)); do
# Is current word not word 0 (the command itself) and is word not
# empty and is word made up of just word separator characters to
# be excluded and is current word not preceded by whitespace in
# original line?
while [[ $i -gt 0 && ${COMP_WORDS[i]} == +($exclude) ]]; do
# Is word separator not preceded by whitespace in original line
# and are we not going to append to word 0 (the command
# itself), then append to current word.
[[ $line != [[:blank:]]* ]] && ((j >= 2)) && ((j--))
# Append word separator to current or new word
ref="$2[$j]"
printf -v "$ref" %s "${!ref-}${COMP_WORDS[i]}"
# Indicate new cword
((i == COMP_CWORD)) && printf -v "$3" %s "$j"
# Remove optional whitespace + word separator from line copy
line=${line#*"${COMP_WORDS[i]}"}
# Indicate next word if available, else end *both* while and
# for loop
if ((i < ${#COMP_WORDS[@]} - 1)); then
((i++))
else
break 2
fi
# Start new word if word separator in original line is
# followed by whitespace.
[[ $line == [[:blank:]]* ]] && ((j++))
done
# Append word to current word
ref="$2[$j]"
printf -v "$ref" %s "${!ref-}${COMP_WORDS[i]}"
# Remove optional whitespace + word from line copy
line=${line#*"${COMP_WORDS[i]}"}
# Indicate new cword
((i == COMP_CWORD)) && printf -v "$3" %s "$j"
done
((i == COMP_CWORD)) && printf -v "$3" %s "$j"
else
# No, list of word completions separators hasn't changed;
for i in "${!COMP_WORDS[@]}"; do
printf -v "$2[i]" %s "${COMP_WORDS[i]}"
done
fi
}
local FCN_RETURN=""
local IS_PATH=0
# -- THIS LINE WILL BE REPLACED WITH GENERATED CODE --
COMPREPLY=()
local OPTIONS_NEW=""
# check if bash or zsh
# _get_comp_words_by_ref is a bash built-in function, we check if it exists
declare -Ff _get_comp_words_by_ref > /dev/null && IS_BASH=1 || IS_BASH=0
# bash interprets the colon character : as a special character and splits the argument in two
# different than what zsh does
# https://stackoverflow.com/a/3224910
# https://stackoverflow.com/a/12495727
local cur
local cword words=()
_comp__reassemble_words ":" words cword
cur=${words[cword]}
# check the action (get or put)
case "${words[0]}" in
"sls_detector_get" | "g" | "detg")
local IS_GET=1
;;
*)
local IS_GET=0
;;
esac
# if no command is written, autocomplete with the commands
if [[ ${cword} -eq 1 ]]; then
# local SLS_COMMANDS="trimbits exptime"
local SLS_COMMANDS_NEW=""
case "$cur" in
[0-9]*:*)
local suggestions=($(compgen -W "${SLS_COMMANDS}" -- "${cur#*:}"))
COMPREPLY=( ${suggestions[*]} )
;;
[0-9]*)
COMPREPLY=()
;;
*)
COMPREPLY=( $( compgen -W "$SLS_COMMANDS -h" -- "$cur" ) );;
esac
return 0
fi
if [[ ${cword} -eq 2 ]] && [[ ${words[1]} == "-h" ]]; then
COMPREPLY=( $( compgen -W "$SLS_COMMANDS" -- "$cur" ) )
return 0
fi
# if a command is written, autocomplete with the options
# call the function for the command
if [[ "$SLS_COMMANDS" == *"${words[1]##*:}"* ]]; then
__"${words[1]##*:}"
fi
# if IS_PATH is activated, autocomplete with the path
if [[ ${IS_PATH} -eq 1 ]]; then
COMPREPLY=($(compgen -f -- "${cur}"))
return 0
fi
# autocomplete with the options
COMPREPLY=($(compgen -W "${FCN_RETURN}" -- "${cur}"))
}
complete -F _sd -o filenames sls_detector_get
complete -F _sd -o filenames g
complete -F _sd -o filenames detg
complete -F _sd -o filenames sls_detector_put
complete -F _sd -o filenames p
complete -F _sd -o filenames detp

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,171 @@
#### simpler version of autocomplete.sh to understand the logic
#### each command has its own function when called it will produce the possible values for autocompletion
_sd() {
# Reassemble command line words, excluding specified characters from the
# list of word completion separators (COMP_WORDBREAKS).
# @param $1 chars Characters out of $COMP_WORDBREAKS which should
# NOT be considered word breaks. This is useful for things like scp where
# we want to return host:path and not only path, so we would pass the
# colon (:) as $1 here.
# @param $2 words Name of variable to return words to
# @param $3 cword Name of variable to return cword to
#
_comp__reassemble_words()
{
local exclude="" i j line ref
# Exclude word separator characters?
if [[ $1 ]]; then
# Yes, exclude word separator characters;
# Exclude only those characters, which were really included
exclude="[${1//[^$COMP_WORDBREAKS]/}]"
fi
# Default to cword unchanged
printf -v "$3" %s "$COMP_CWORD"
# Are characters excluded which were former included?
if [[ $exclude ]]; then
# Yes, list of word completion separators has shrunk;
line=$COMP_LINE
# Re-assemble words to complete
for ((i = 0, j = 0; i < ${#COMP_WORDS[@]}; i++, j++)); do
# Is current word not word 0 (the command itself) and is word not
# empty and is word made up of just word separator characters to
# be excluded and is current word not preceded by whitespace in
# original line?
while [[ $i -gt 0 && ${COMP_WORDS[i]} == +($exclude) ]]; do
# Is word separator not preceded by whitespace in original line
# and are we not going to append to word 0 (the command
# itself), then append to current word.
[[ $line != [[:blank:]]* ]] && ((j >= 2)) && ((j--))
# Append word separator to current or new word
ref="$2[$j]"
printf -v "$ref" %s "${!ref-}${COMP_WORDS[i]}"
# Indicate new cword
((i == COMP_CWORD)) && printf -v "$3" %s "$j"
# Remove optional whitespace + word separator from line copy
line=${line#*"${COMP_WORDS[i]}"}
# Indicate next word if available, else end *both* while and
# for loop
if ((i < ${#COMP_WORDS[@]} - 1)); then
((i++))
else
break 2
fi
# Start new word if word separator in original line is
# followed by whitespace.
[[ $line == [[:blank:]]* ]] && ((j++))
done
# Append word to current word
ref="$2[$j]"
printf -v "$ref" %s "${!ref-}${COMP_WORDS[i]}"
# Remove optional whitespace + word from line copy
line=${line#*"${COMP_WORDS[i]}"}
# Indicate new cword
((i == COMP_CWORD)) && printf -v "$3" %s "$j"
done
((i == COMP_CWORD)) && printf -v "$3" %s "$j"
else
# No, list of word completions separators hasn't changed;
for i in "${!COMP_WORDS[@]}"; do
printf -v "$2[i]" %s "${COMP_WORDS[i]}"
done
fi
}
__exptime(){
if [ "${IS_GET}" == "1" ]; then
if [ "${cword}" == "2" ]; then
FCN_RETURN="s ms us ns"
fi
else
if [ "${cword}" == "2" ]; then
FCN_RETURN=""
fi
if [ "${cword}" == "3" ]; then
FCN_RETURN="s ms us ns"
fi
fi
}
# trimbits will activate IS_PATH and signal that its input is a path
__trimbits(){
if [ "${IS_GET}" == "1" ]; then
if [ "${cword}" == "2" ]; then
FCN_RETURN=""
IS_PATH=1
fi
else
if [ "${cword}" == "2" ]; then
FCN_RETURN=""
IS_PATH=1
fi
fi
}
local cword words=()
_comp__reassemble_words ":" words cword
local FCN_RETURN=""
local IS_PATH=0
COMPREPLY=()
local OPTIONS_NEW=""
# _get_comp_words_by_ref -n : cur
local cur=${words[cword]}
# check the action (get or put)
if [ "${words[0]}" == "sls_detector_get" ]; then
local IS_GET=1
else
local IS_GET=0
fi
# if no command is written, autocomplete with the commands
if [[ ${cword} -eq 1 ]]; then
local SLS_COMMANDS="trimbits exptime"
local SLS_COMMANDS_NEW=""
case "$cur" in
[0-9]*:*)
local suggestions=($(compgen -W "${SLS_COMMANDS}" -- "${cur#*:}"))
COMPREPLY=( ${suggestions[*]} )
;;
[0-9]*)
COMPREPLY=()
;;
*)
COMPREPLY=( $( compgen -W "$SLS_COMMANDS -h" -- "$cur" ) );;
esac
return 0
fi
# if a command is written, autocomplete with the options
# call the function for the command
__"${words[1]##*:}"
# if IS_PATH is activated, autocomplete with the path
if [[ ${IS_PATH} -eq 1 ]]; then
COMPREPLY=($(compgen -f -- "${cur}"))
return 0
fi
# autocomplete with the options
COMPREPLY=($(compgen -W "${FCN_RETURN}" -- "${cur}"))
}
complete -F _sd -o filenames sls_detector_get
complete -F _sd -o filenames g
complete -F _sd -o filenames p
complete -F _sd -o filenames detg
complete -F _sd -o filenames detp
complete -F _sd -o filenames sls_detector_put

View File

@ -0,0 +1,74 @@
# GENERATED FILE - DO NOT EDIT
# ANY CHANGES TO THIS FILE WILL BE OVERWRITTEN
_sd() {
# -- THIS LINE WILL BE REPLACED WITH GENERATED CODE --
local FCN_RETURN=""
local IS_PATH=0
COMPREPLY=()
local OPTIONS_NEW=""
words=("${COMP_WORDS[@]}")
cword=$COMP_CWORD
local cur=${words[cword]}
# check the action (get or put)
case "${words[0]}" in
"sls_detector_get" | "g" | "detg")
local IS_GET=1
;;
*)
local IS_GET=0
;;
esac
# if no command is written, autocomplete with the commands
if [[ ${cword} -eq 1 ]]; then
case "$cur" in
[0-9]*)
for i in $SLS_COMMANDS; do
SLS_COMMANDS_NEW="${SLS_COMMANDS_NEW} ${cur%%:*}:$i"
done
COMPREPLY=( $( compgen -W "${SLS_COMMANDS_NEW}" -- "$cur" ) );;
*)
COMPREPLY=( $( compgen -W "$SLS_COMMANDS -h" -- "$cur" ) );;
esac
return 0
fi
if [[ ${cword} -eq 2 ]] && [[ ${words[1]} == "-h" ]]; then
COMPREPLY=( $( compgen -W "$SLS_COMMANDS" -- "$cur" ) )
return 0
fi
# if a command is written, autocomplete with the options
# call the function for the command
if [[ "$SLS_COMMANDS" == *"${words[1]##*:}"* ]]; then
__"${words[1]##*:}"
fi
# if IS_PATH is activated, autocomplete with the path
if [[ ${IS_PATH} -eq 1 ]]; then
COMPREPLY=($(compgen -f -- "${cur}"))
return 0
fi
# autocomplete with the options
COMPREPLY=($(compgen -W "${FCN_RETURN}" -- "${cur}"))
}
complete -F _sd -o filenames sls_detector_get
complete -F _sd -o filenames g
complete -F _sd -o filenames detg
complete -F _sd -o filenames sls_detector_put
complete -F _sd -o filenames p
complete -F _sd -o filenames detp

File diff suppressed because it is too large Load Diff