Improve doc.

Change-Id: I0b25fd1c645a47084fb7fda1bbb7d41a2d10dcbb

doc/source/protocol/descjson.rst

@@ -0,0 +1,11 @@
+Structure of the descriptive json
+=================================
+
+* json = {"modules": <list_of_modules>, "properties": <list_of_sec-node_properties>, ...}
+* module = {"name": <name_of_module>, "parameters": <list_of_parameters>, "commands": <list_of_commands>, "properties": <list_of_module_properties>}
+* parameter = {"name": ..., "properties": <list_of_properties>}
+* command = {"name": ..., "properties": <list_of_properties>}
+* property = {"name":<name>, "datatype": <datatype>, "value": <value>}
+
+note: property may also be [<name>,<datatype>,<value>]
+

doc/source/protocol/descprop.rst

@@ -0,0 +1,48 @@
+Descriptive properties
+======================
+
+Mandatory descriptive properties
+--------------------------------
+
+parameter-properties
+++++++++++++++++++++
+
+* name (implicit)
+* datatype
+* readonly (bool)
+
+module-properties
++++++++++++++++++
+
+* interface_class [list_of_strings] (MAY be empty)
+
+SEC-Node-properties
++++++++++++++++++++
+
+* no mandatory properties
+
+Optional descriptive properties
+-------------------------------
+
+parameter-properties
+++++++++++++++++++++
+
+* unit (string), SHOULD be given if meaningful (if not given: unitless) (empty string: unit is one)
+* description (string), SHOULD be given
+* visibility
+* group (identifier) (MUST start with an uppercase letter) (if empty string: treat as not specified)
+
+module-properties
++++++++++++++++++
+
+* description (string), SHOULD be given
+* visibility
+* group (identifier) (MUST start with an uppercase letter) (if empty string: treat as not specified)
+* meaning ???
+* importance ???
+
+SEC-Node-properties
++++++++++++++++++++
+
+* description (string), SHOULD be given
+

doc/source/protocol/heartbeat.rst

@@ -0,0 +1,8 @@
+Heartbeat
+=========
+* ping gets an 'intended looptime' argument (as number in seconds or null to disable)
+* server replys as usual
+* if the server received no new message within twice the indended looptime, it may close the connection.
+* if the client receives no pong within 3s it may close the connection
+* later discussions showed, that the ping/pong should stay untouched and the keepalive time should be (de-)activated by a special message instead. Also the 'connection specific settings' from earlier drafts may be resurrected for this....
+

doc/source/protocol/hierarchy.rst

@@ -0,0 +1,9 @@
+Hirarchy
+========
+
+* group property (currently a identifier like string, may be extended to tree like substrucutres by allowing ':')
+* visibility (enum(3:expert, 2:advanced, 1:user)) (default to 1 if not given)
+  if visibility is set to user: everybody should see it
+  if visibility is set to advanced: advanced users should see it
+  if visibility is set to expert: only 'experts' should see it
+    

doc/source/protocol/history.rst

@@ -0,0 +1,66 @@
+History
+=======
+
+Meeting 29.5.2017
+-----------------
+
+* for api: float is 'double'
+* everything countable is int64_t
+* description is string (UTF-8 without embedded \0) (zero terminated for API)
+* names / identifiers are:  [_a-z][_a-z0-9]{0,63}
+* BLOB is [length, string_encoding (base64 or json_string) ] ???
+* enum is transferred by value (api: int64_t)
+* basic data types: string, BLOB(maxsize), int, double, bool, enum(mapping)
+* encode as ["string"] ["blob"] ["int"] ["double"] ["bool"] ["enum", {<number_value>:<name>}]
+* send as json_string [length, json_string] number number 0_or_1 number_value
+* complex data types: array, tuple, struct
+* encode as: ["array", <subtype>] ["tuple", [<list_of_compositing data types>] ["struct", {"name_of_subcomponent":<type of subcomponent>}]
+* send as [array] [array} {mapping}
+* forbid: structs in structs, nesting level > 3, arrays may only contain basic types + tuple
+* essential features should not rely on complex data types
+* fallback: if ECS can not handle a non-basic datatype: handle as string containing the JSON-representation.
+* mandatory for all ECS: enum, int, double, string, bool, tuple(enum,string)
+
+Merge datatype and validator
+++++++++++++++++++++++++++++
+
+* ["enum", {<number_value>:<json_string>}]
+* ["int"] or ["int", <lowest_allowed_value>, <highest_allowed_value>]
+* ["double"] or ["double", <lowest_allowed_value>, <highest_allowed_value>]
+* ["bool"]
+* ["blob", <maximum_size_in_bytes>] or ["blob", <minimum_size_in_bytes>, <maximum_size_in_bytes>]
+* ["string", <maximum_allowed_length>] or ["string", <min_size>, <max_size>]
+* ["array", <basic_data_type>, <max_elements>] or ["array", <dtype>, <min_elements>, <max_elements>]
+* ["tuple", [ <list_of_dtypes ]]
+* ["struct", { <name_of_component_as_json_string>:<dtype>}]
+
+Examples
+++++++++
+
+* status: ["tuple", [ ["enum", {0:"init", 100:"idle", 200:"warn", 300:"busy"}], ["string", 255] ] ]
+* p/pi/pid-triple: ["array", ["double", 0, 100], 1, 3]
+
+
+Meeting 30.5.2017
+-----------------
+
+* data values can be transferred as json_null, meaning: no value yet
+* json_null can not be used inside structured data types
+* property name for datatype is "datatype"
+
+Meeting 11.9.2017
+-----------------
+
+Merge datatype and validator
+++++++++++++++++++++++++++++
+
+  * enum, int, double, bool, tuple, struct as before
+  * ["blob", <maximum_size_in_bytes>] or ["blob", <maximum_size_in_bytes>, <minimum_size_in_bytes>]
+  * ["string", <maximum_allowed_length>] or ["string", <max_size_in_bytes>, <minimum_size_in_bytes>]
+  * ["array", <basic_data_type>, <max_elements>] or ["array", <dtype>, <max_elements>, <min_elements>]
+
+Interface_class
++++++++++++++++
+
+  * Drivable, Writable, Readable, Module (first character uppercase, no middle 'e')
+

doc/source/protocol/index.rst

@@ -0,0 +1,19 @@
+Protocol documentation
+======================
+
+.. toctree::
+   :maxdepth: 2
+
+   messages
+   descprop
+   hierarchy
+   descjson
+   timeformat
+   modsubset
+   heartbeat
+   jsonstruct
+   todo
+   notes
+   history
+
+

doc/source/protocol/jsonstruct.rst

@@ -0,0 +1,90 @@
+JSON structure
+==============
+
+> Mit JSON Freeze meine ich nur Arrays von Objekten also:
+> -Node hat ein Array von Properties und ein Array von Modulen
+> -Module haben ein Array von Properties, ein Array von Parametern und ein Array von Commands
+> -Parameter und Commands haben jeweils ein Array von Properties
+
+node = { 'properties' : [<property>],
+         'modules'    : [<module>] }
+
+module = { 'properties' : [<property>],
+           'parameter'  : [<parameter>],
+           'commands'   : [<command>] }
+
+parameter = { 'properties' : [<property>] }
+
+commands = { 'properties' : [<property>] }
+
+property = { 'property-name' : 'property-value' }
+
+
+ODER
+----
+
+node = [ <array_of_properties>, <array_of_modules> ]
+
+array_of_modules = [ <module>, ... ]
+
+module = <array_of_properties>, <array_of_params>, <array_of_commands>
+#OR#
+module = [<array_of_properties>, <array_of_params>, <array_of_commands>]
+
+array_of_params = [ <param>, ...]
+
+param = <array_of_properties>
+
+array_of_commands = [ <command>, ...]
+
+command = < array_of_properties>
+
+array_of_properties = [ <property>, ...]
+
+property = 'name', 'value'
+#OR#
+property = ['name', 'value']
+#OR#
+property = {'name' : 'value'}
+
+Oder ganz anders?
+
+Siehe auch diskussion im HZB-wiki hierzu....
+
+verwirrend....
+
+vorerst folgende Festlegung:
+{"equipment_id": "cryo_7",
+ "firmware": "The SECoP playground",
+ "modules": ["cryo", {"commands": ["stop", {"resulttype": "None",
+                                            "arguments": "[]",
+                                            "description": "Testing command implementation\n\nwait a second"
+                                           },
+                                   "start", {"resulttype": "None",
+                                             "arguments": "[]",
+                                             "description": "normally does nothing,\n\nbut there may be modules which _start_ the action here\n"
+                                            }
+                                  ],
+                       "group": "very important/stuff",
+                       "implementation": "secop.devices.cryo.Cryostat",
+                       "interfaces": ["Drivable", "Readable", "Device"],
+                       "parameters": ["status", {"readonly": true,
+                                                 "datatype": ["tuple", ["enum", {"unknown":-1,"idle":100, "warn":200, "unstable":250, "busy":300,"error":400}], "string"],
+                                                 "description": "current status of the device"
+                                                },
+                                      "value", {"readonly": true,
+                                                "datatype": ["double",0,null],
+                                                "description": "regulation temperature",
+                                                "unit": "K"
+                                               },
+                                      "target", {"readonly": false,
+                                                 "datatype": ["double",0,null],
+                                                 "description": "target temperature",
+                                                 "unit": "K"
+                                                }
+                                   ]
+                       }
+              ],
+ "version": "2017.01"
+}
+

doc/source/protocol/messages.rst

@@ -0,0 +1,202 @@
+SECoP Messages
+==============
+
+All Messages are formatted in the same way:
+  <keyword>[<space><specifier>[<space><JSON_formatted_data>]]<linefeed>
+
+where [] enclose optional parts. This basically results in 3 different possible
+formattings:
+
+  * type A: "keyword\n"
+  * type B: "keyword specifier\n"
+  * type C: "keyword specifier JSON_data\n"
+
+Note: numerical values and strings appear 'naturally' formatted in JSON, i.e. 5.0 or "a string"
+
+<keyword> is one of a fixed list of defined keywords, <specifier> is either the
+name of the module optionally followed by ':' + the name of a command or parameter,
+or one of a fixed list of predefined keywords, depending on the message keyword.
+
+At the moment it is considered syntactic sugar to omit the parametername in a request.
+In replies the SEC-node (in the playground) will always use the correct parameter.
+On change-requests the parameter is assumed to be 'target', on trigger-requests it is assumed to be 'value'.
+Clients should not rely on this and explicitly state the parametername!
+
+All names and keywords are defined to be identifiers in the sense, that they are not longer than 63 characters and consist only of letters, digits and underscore and do not start with a digit. (i.e. T_9 is ok, whereas t{9} is not!)
+No rule is without exception, there is exactly ONE special case: the identify request consists of the literal string '*IDN?\n' and its answer is formatted like an valid SCPI response for *IDN?.
+
+We rely on the underlying transport to not split messages, i.e. all messages are transported as a whole and no message interrupts another.
+
+Also: each client MUST at any time correctly handle incoming event messages, even if it didn't activate them!
+
+Implementation node:
+Both SEC-node and ECS-client can close the connection at any time!
+
+
+List of Intents/Actions:
+------------------------
+
+  * Identify -> Ident
+  * Describe -> Description
+  * Activate Events -> initial data transfer -> end-of-transfer-marker
+  * Deactivate Async Events -> confirmation
+  * Command <module>:<command> -> confirmation -> result_event
+  * Heartbeat <nonce> -> Heartbeat_reply
+  * Change <module>[:<param>] JSON_VALUE -> confirmation -> readback_event
+  * TRIGGER <module>[:<param>] -> read_event
+
+At any time an Event may be replied. Each request may also trigger an Error.
+
+Line-ending is \lf.
+\cr is ignored and never send.
+Warning: One Message per line: Description line can be looooong!!!
+
+
+Allowed combinations as examples:
+(replace <..> with sensible stuff)
+
+
+Identify
+--------
+
+  * Request: type A: '*IDN?'
+  * Reply:   special: 'SECoP, SECoPTCP, V2016-11-30, rc1'
+  * queries if SECoP protocol is supported and which version it is
+  Format is intentionally choosen to be compatible to SCPI (for this query only).
+  It is NOT intended to transport information about the manufacturer of the hardware, but to identify this as a SECoP device and transfer the protocol version!
+
+
+Describe
+--------
+
+  * Request: type A: 'describe'
+  * Reply:   type C: 'describing <ID> {"modules":{"T1":{"baseclass":"Readable", ....'
+  * request the 'descriptive data'. The format needs to be better defined and
+  may possibly just follow the reference implementation.
+  <ID> identifies the equipment. It should be unique. Our suggestion is to use something along <facility>_<id>, i.e. MLZ_ccr12 or PSI_oven4.
+
+
+Activate Async Events
+---------------------
+
+  * Request: type A: 'activate'
+  * Reply:   several EVENT lines (initial value transfer) followed by: type A: 'active'
+  * Activates sending of Async Events after transferring all live quantities once
+  and an 'end-of-initial-transfer' marker. After this events are enabled.
+
+
+Deactivate Async Events
+-----------------------
+
+  * Request: type A: 'deactivate'
+  * Reply:   type A: 'inactive'
+  * Deactivate sending of async Events. A few events may still be on their way until the 'inactive' message arrives.
+
+
+Execute Command
+---------------
+
+  * Request: type B: 'do <module>:<command>' for commands without arguments
+  * Request: type C: 'do <module>:<command> JSON_argument' for commands with arguments
+  * Reply:   type C: 'done <module>:<command> JSON_result' after the command finished
+  * start executing a command. When it is finished, the reply is send.
+    The JSON_result is the a list of all return values (if any), appended with qualifiers (timestamp)
+
+
+Write
+-----
+
+  * Request: type C: 'change <module>[:<param>] JSON_value'
+  * Reply: type C: 'changed <module>:<param> JSON_read_back_value'
+  * initiate setting a new value for the module or a parameter of it.
+  Once this is done, the read_back value is confirmed by the reply.
+
+
+Trigger
+-------
+
+  * Request: type B: 'read <module>[:<param>]'
+  * Reply:   None directly. However, one Event with the read value will be send.
+  * Read the requested quantity and sends it as an event (even if events are disabled or the value is not different to the last value).
+
+
+Heartbeat
+---------
+
+  * Request: type A: 'ping'
+  * Request: type B: 'ping <nonce>'
+  * Reply:   type A: 'pong'
+  * Reply:   type B: 'pong <nonce>'
+  * Replies the given argument to check the round-trip-time or to confirm that the connection is still working.
+  <nonce> may not contain <space>. It is suggested to limit to a string of up to 63 chars consisting of letters, digits and underscore not beginning with a digit. If <nonce> is not given (Type A), reply without it.
+
+
+EVENT
+-----
+
+Events can be emitted any time from the SEC-node (except if they would interrupt another message).
+
+  * Request: None. Events can be requested by Trigger or by Activating Async Mode.
+  * Reply:   type C: 'event <module>:<param> JSON_VALUE'
+  * Informs the client that a parameter got changed its value.
+  In any case the JSON_value contain the available qualifiers as well:
+    * "t" for the timestamp of the event.
+    * "e" for the error of the value.
+    * "u" for the unit of the value, if deviating from the descriptive data
+    * further qualifiers, if needed, may be specified.
+  The qualifiers are a dictionary at position 2 of a list, where the value occupies position 1.
+  This holds true also for complex datatypes (of value)!
+
+  examples:
+
+  * 'update T1:value [3.479, {"t":"149128925.914882", "e":0.01924}]
+  * 'update T1:p [12, {"t":"149128927.193725"}]'
+  * 'update Vector:value [[0.01, 12.49, 3.92], {"t":"149128925.914882"}]'
+
+
+ERROR
+-----
+
+  * Request: None. can only be a reply if some request fails.
+  * Reply: type C: 'ERROR <errorclass> JSON_additional_stuff'
+  * Following <errorclass> are defined so far:
+    * NoSuchDevice: The action can not be performed as the specified device is non-existent.
+    * NoSuchParameter: The action can not be performed as the specified parameter is non-existent.
+    * NoSuchCommand: The specified command does not exist.
+    * CommandFailed: The command failed to execute.
+    * CommandRunning: The command is already executing.
+    * ReadOnly: The requested write can not be performed on a readonly value..
+    * BadValue: The requested write or Command can not be performed as the value is malformed or of wrong type.
+    * CommunicationFailed: Some communication (with hardware controlled by this SEC-Node) failed.
+    * IsBusy: The reequested write can not be performed while the Module is Busy
+    * IsError: The requested action can not be performed while the module is in error state.
+    * Disabled: The requested action can not be performed at the moment. (Interlocks?)
+    * SyntaxError: A malformed Request was send
+    * InternalError: Something that should never happen just happened.
+  The JSON part should reference the offending request and give an explanatory string.
+
+  examples:
+
+  * 'ERROR Disabled ["change", "V15", "on", "Air pressure too low to actuate the valve.", {"exception":"RuntimeException","file":"devices/blub/valve.py", "line":13127, "frames":[...]}]'
+  * 'ERROR NoSuchDevice ["read","v19", "v19 is not configured on this SEC-node"]'
+  * 'ERROR SyntaxError "meas:Volt?"
+
+
+Examples
+--------
+
+(client connects):
+(client)    '*IDN?'
+(SEC-node)  'Sine2020WP7.1&ISSE, SECoP, V2016-11-30, rc1'
+(client)    'describe'
+(SEC-node)  'describing SECoP_Testing {"modules":{"T1":{"baseclass":"Readable", ...
+(client)    'activate'
+(SEC-node)  'update T1 [3.45,{"t":"149128925.914882","e":0.01924}]'
+...
+(SEC-node)  'active'
+(SEC-node)  'update T1 [3.46,{"t":"149128935.914882","e":0.01912}]'
+(client)    'ping fancy_nonce_37'
+(SEC-node)  'pong fancy_nonce_37'
+(SEC-node)  'update T1 [3.49,{"t":"149128945.921397","e":0.01897}]'
+...
+

doc/source/protocol/modsubset.rst

@@ -0,0 +1,8 @@
+Activate subset of modules
+==========================
+
+* activate/deactivate may get an optional 2nd argument and work only on this.
+* add equipment_id [_0-9a-zA-A]{4,} as SEC-node property (mandatory) (prefixed with ficility/manufacturer)
+* change response to 'describe' to 'describing ALL <json_description>'
+* '(de-)activate samething' -> '(de-)activated something'
+

doc/source/protocol/notes.rst

@@ -0,0 +1,42 @@
+Notes
+=====
+
+No installation required or recommended
+---------------------------------------
+
+everything runs directly from the checkout.
+
+you need:
+ - python2.7.*
+ - pip
+ - linux OS (Mac may work as well)
+
+install requirements with pip:
+$ sudo pip install -r requirements.txt
+
+to execute a program, prefix its name with bin/, e.g.:
+$ bin/make_doc.py
+$ bin/server.py start test
+
+a testsuite is planned but nothing is there yet.
+
+Structure
+---------
+ 
+* bin contains the executables (make_doc.py, server.py)
+* doc is the root node of the docu (see index.md)
+* etc contains the configurations for the server(s) and devices
+* html contains the docu after make_doc.py was run
+* log contains some (hopefully) log output from the servers
+* pid contains pidfiles if a server is running
+* src contains the python source
+
+  * src/client: client specific stuff (proxy)
+  * src/devices: devices to be used by the server (and exported via SECoP)
+  * src/lib: helper stuff (startup, pidfiles, etc)
+  * src/protocol: protocol specific stuff
+  * src/errors.py: internal errors
+  * src/server.py: device-managing part of the server (transport is in src/protocol/transport)
+  * src/validators.py: validators used by the devices. may be moved to src/protocol
+
+

doc/source/protocol/timeformat.rst

@@ -0,0 +1,7 @@
+Timeformat
+==========
+
+* format goes to 'timestamp since epoch as double with a resolution of at least 1ms'
+* SEC-node-property: timestamp is accurate or relative
+* Or: extend pong response contains the localtime (formatted as a timestamp)
+

doc/source/protocol/todo.rst

@@ -0,0 +1,101 @@
+=========
+Todo List
+=========
+
+Open questions
+==============
+
+* Datatype for a mapping (like the calibration for the amagnet)
+* How to set a Mapping or an enum (with all the name-value mappings) in the configfile?
+* use toml vs. ini parser?
+* error handling should be more consistent
+* error-info stuff should be consistent
+* dynamic device creation / modification (i.e. how to set the mapping of an enum during runtime?)
+* propagation of units from main value to params
+* switch params+cmds to 'accessibles'/attributes and keep in orderdDict?
+  -> needs Datatype for a function
+* datatype for funcion: is argin a list anyway, or just one (structured) argument?
+  (so far we use a list and argout is a single datatype (which may be stuctured))
+* polling: vendor specific or do we propose a common way to handle it? 
+  (playground uses a per-device pollinterval. params may be polled less often)
+* interface classes !!! (maybe with feature mixins like WindowTimeout?)
+* hardware access: unify? how?
+* demo-hardware?
+* If more than one connection exists: shall all events be relayed to all listeners?
+* how about WRITE/COMMAND replies? Shall they go to all connected clients?
+* structure of descriptive data needs to be specified
+* same for JSON_stuff for Error Messages
+* 'changed' may be 'readback'
+* 'change' may be 'write'
+* 'read' may be 'poll'
+* the whole message may be json object (bigger, uglier to read)
+* which events are broadcast or unicast?
+* do we need a way to correlate a reply with a request?
+
+
+Todos
+=====
+
+Structure
+---------
+
+ * stronger structure insides src
+
+   * src/server for everything server related
+   * src/client for everything client related (ProxyDevice!)
+   * src/protocol for protocol specific things
+
+     * need subtree for different implementations to play with
+
+   * src/lib for helpers and other stuff
+
+ * possibly a parallel src tree for cpp version
+
+
+Client
+------
+
+ * maybe start with a python shell and some import magic
+ * later a GUI may be a good idea
+ * client: one connection for each device?
+ * another connection for async data?
+
+
+Server
+------
+
+ * rewrite MessageHandler to be agnostic of server
+ * move encoding to interface
+ * allow multiple interfaces per server
+ * fix error handling an make it consistent
+
+Device framework
+----------------
+
+ * supply properties for PARAMS to auto-generate async data units
+ * self-polling support
+ * generic devicethreads
+ * proxydevice
+ * make get_device uri-aware
+
+
+Testsuite
+---------
+
+ * need a testsuite (pytest)
+ * embedded tests inside the actual files grow difficult to maintain
+
+
+Documentation
+-------------
+
+ * transfer build docu into wiki via automated jobfile
+
+
+ 
+
+Transfer of blobs via json
+--------------------------
+
+ * use base64
+