typo

Document stream-base.dbd usage when sCalcout is not needed

docs/formats.html

@@ -485,6 +485,12 @@ than 9.
 <p>
 This is not a normal "converter", because no user data is converted.
 Instead, a checksum is calculated from the input or output.
+<span class="new">
+Any pre-processing of input, e.g. by the <a href="#regsub">regsub</a> converter
+is ignored for the calculation of the checksum.
+</span>
+</p>
+<p>
 The <em>width</em> field is the byte number from which to start
 calculating the checksum.
 Default is 0, i.e. the first byte of the input or output of the current
@@ -711,10 +717,14 @@ However if an empty string is matched, searching advances by 1 character in orde
 avoid matching the same empty string again.</span>
 </p>
 <p>
-In input this converter pre-processes data received from the device before
+In input, this converter pre-processes data received from the device before
 following converters read it.
 Converters preceding this one will read unmodified input.
 Thus place this converter before those whose input should be pre-processed.
+<span class="new">
+However, <a href="#chksum">checksum</a> converters will always use the unmodified
+input as sent by the device because the modified input would not match the checksum.
+</span>
 </p>
 <p>
 In output it post-processes data already formatted by preceding converters

docs/setup.html

@@ -85,7 +85,7 @@ Make sure that the <em>asyn</em> library can be found by adding the path to the
 ASYN=/home/epics/asyn4-30
 </pre>
 
-<h4>Support for <em>sCalcout</em> record</h4>
+<h4 id="scalcout">Support for <em>sCalcout</em> record</h4>
 <p>
 The <a 
 href="https://htmlpreview.github.io/?https://raw.githubusercontent.com/epics-modules/calc/R3-6-1/documentation/sCalcoutRecord.html"
@@ -109,7 +109,10 @@ modules. Release R2-8 or newer is recommended.
 </p>
 <p>
 Support for the <em>sCalcout</em> is optional. <em>StreamDevice</em> works
-as well without <em>sCalcout</em> or <em>SynApps</em>.
+as well without <em>sCalcout</em> or <em>SynApps</em>. If your application does
+not need this record support, you may load <kbd>stream-base.dbd</kbd> instead of
+<kbd>stream.dbd</kbd>, making it optional to include <em>calc</em> as an
+application dependency.
 </p>
 
 <h4>Support for regular expression matching</h4>
@@ -185,13 +188,14 @@ Regular expressions are optional. If you don't want them, you don't need this.
 Go to the <em>StreamDevice</em> directory
 and run <code>make</code> (or <code>gmake</code>).
 This will create and install the <em>stream</em> library and the
-<kbd>stream.dbd</kbd> file and an example IOC application.
+<em>StreamDevice</em> database definition files and an example IOC application.
 </p>
 
 <p>
 To use <em>StreamDevice</em>, your own application must be built with the
 <em>stream</em> and <em>asyn</em> (and optionally <em>pcre</em>) libraries
-and must load <kbd>asyn.dbd</kbd> and <kbd>stream.dbd</kbd>.
+and must load <kbd>asyn.dbd</kbd> and <kbd>stream.dbd</kbd> (or alternatively
+<kbd>stream-base.dbd</kbd>; see <a href="#scalcout">Support for sCalcout record</a>).
 </p>
 <p>
 Include the following lines in your application <kbd>Makefile</kbd>:
@@ -385,35 +389,45 @@ See the <a href="protocol.html">next chapter</a> for protocol files in depth.
 Generation of debug and error messages is controlled with two shell variables,
 <code>streamDebug</code> and <code>streamError</code>.
 Setting those variables to 1 (actually to any number but 0) enables the
-messages.
+messages. A few noisy and rarely useful debug messages are only enabled when
+setting <code>streamDebug</code> to 2.
 Per default debug messages are switched off and error messages are switched on.
 Errors occuring while loading protocol files are always shown.
 </p>
 <p>
-Warning: Enabling debug messages can create a lot of output!
-At the moment, there is no way to set filters on debug or error messages.
+Warning: Enabling debug messages this way can create a lot of output!
+Therefore, some limited debugging can be enabled per record, independent of
+the <code>streamDebug</code> variable using the <code>.TPRO</code> field of
+the record. Currently, setting <code>.TPRO</code> to 1 or 2 enables some
+basic information about the processing of a record and its i/o.
 </p>
 <p>
 Debug output can be redirected to a file with the command
 <code>streamSetLogfile("<var>filename</var>")</code>.
-When called without a filename, debug output is directed back
-to the console.
+If the file already exists, it will be overwritten, not appended to.
+While debug messages are only written to the defined log file, error messages
+are still printed to <var>stderr</var> too.
+Calling <code>streamSetLogfile</code> without a filename directs debug output
+back to <var>stderr</var> and closes the log file.
 </p>
 <p>
-By default the debug/error output is set to be colored if the terminal allows
-it but this can be set to always colored or never colored by setting 
-<code>streamDebugColored</code> to 1 or 0 respectively.
+By default, error messages to the console are printed in red color if
+<var>stderr</var> is a tty at startup time, using ANSI color codes. Some
+terminals may not support this properly.
+The variable <code>streamDebugColored</code> can be set to 0 or 1 to
+disable or enable colored error messages explicitly.
+Error messages written to a log file do not use colors.
 </p>
 <p>
 Error and debug messages are prefixed with a time stamp unless the variable
 <code>streamMsgTimeStamped</code> is set to 0.
 </p>
 <p>
-When a device is disconnected StreamDevice can produce many repeated timeout 
-messages. To reduce this logging you can set <code>streamErrorDeadTime</code>
-to an integer number of seconds. When this is set repeated timeout messages 
-will not be printed in the specified dead time after the last message. The 
-default dead time is 0, resulting in every message being printed.
+when a device is unresponsive, StreamDevice may produce many repeated timeout 
+messages. To reduce this, you can set <code>streamErrorDeadTime</code>
+to an integer number of seconds. In this case, repeated timeout messages 
+will not be printed during the specified dead time after the last printed
+message. The default dead time is 0, resulting in every message being printed.
 </p>
 
 <h3>Example (vxWorks):</h3>

src/ChecksumConverter.cc

@@ -616,6 +616,16 @@ static uint32_t hexlrc(const uint8_t* data, size_t len, uint32_t sum)
     return sum;
 }
 
+// Checksum used by Spellman High Voltage Supplies MPS
+static uint32_t hv_mps(const uint8_t* data, size_t len, uint32_t sum)
+{
+    while (len--)
+    {
+        sum += *data++;
+    }
+    return (~sum & 0x7F) | 0x40;
+}
+
 struct checksum
 {
     const char* name;
@@ -665,19 +675,20 @@ static checksum checksumMap[] =
     {"bitsum8", bitsum,           0x00,       0x00,       1}, // 0x21
     {"bitsum16",bitsum,           0x0000,     0x0000,     2}, // 0x0021
     {"bitsum32",bitsum,           0x00000000, 0x00000000, 4}, // 0x00000021
+    {"hv_mps",  hv_mps,           0xFF,       0x00,       1}  // 0x63
 };
 
 static uint32_t mask[5] = {0, 0xFF, 0xFFFF, 0xFFFFFF, 0xFFFFFFFF};
 
 class ChecksumConverter : public StreamFormatConverter
 {
-    int parse (const StreamFormat&, StreamBuffer&, const char*&, bool);
+    int parse (const StreamFormat&, StreamBuffer&, const char*&, bool scanFormat);
     bool printPseudo(const StreamFormat&, StreamBuffer&);
     ssize_t scanPseudo(const StreamFormat&, StreamBuffer&, size_t& cursor);
 };
 
 int ChecksumConverter::
-parse(const StreamFormat&, StreamBuffer& info, const char*& source, bool)
+parse(const StreamFormat&, StreamBuffer& info, const char*& source, bool scanFormat)
 {
     const char* p = strchr(source, '>');
     if (!p)
@@ -731,7 +742,7 @@ parse(const StreamFormat&, StreamBuffer& info, const char*& source, bool)
             info.append(&xorout, sizeof(xorout));
             info.append(fnum);
             source = p+1;
-            return pseudo_format;
+            return scanFormat ? needs_original_format : pseudo_format;
         }
     }
 
@@ -836,7 +847,7 @@ scanPseudo(const StreamFormat& format, StreamBuffer& input, size_t& cursor)
         else length = 0;
     }
 
-    debug("ChecksumConverter %s: input to check: \"%s\n",
+    debug("ChecksumConverter %s: input to check: \"%s\"\n",
         checksumMap[fnum].name, input.expand(start,length)());
 
     uint8_t nDigits =
@@ -848,8 +859,8 @@ scanPseudo(const StreamFormat& format, StreamBuffer& input, size_t& cursor)
 
     if ((ssize_t)( input.length() - cursor ) < expectedLength)
     {
-        debug("ChecksumConverter %s: Input '%s' too short for checksum\n",
-            checksumMap[fnum].name, input.expand(cursor)());
+        debug("ChecksumConverter %s: Input '%s' too short (%zu-%zu<%zu) for checksum\n",
+            checksumMap[fnum].name, input.expand(cursor)(), input.length(), cursor, expectedLength);
         return -1;
     }
 

src/StreamCore.cc

@@ -603,6 +603,7 @@ evalOut()
     // flush all unread input
     unparsedInput = false;
     inputBuffer.clear();
+    inputLine.clear();
     if (!formatOutput())
     {
         finishProtocol(FormatError);
@@ -1011,6 +1012,7 @@ readCallback(StreamIoStatus status,
             finishProtocol(Fault);
             return 0;
     }
+    inputHook(input, size);
     inputBuffer.append(input, size);
     debug("StreamCore::readCallback(%s) inputBuffer=\"%s\", size %" Z "u\n",
         name(), inputBuffer.expand()(), inputBuffer.length());
@@ -1130,7 +1132,7 @@ readCallback(StreamIoStatus status,
     inputBuffer.remove(end + termlen);
     if (inputBuffer)
     {
-        debug("StreamCore::readCallback(%s) unpared input left: \"%s\"\n",
+        debug("StreamCore::readCallback(%s) unparsed input left: \"%s\"\n",
             name(), inputBuffer.expand()());
         unparsedInput = true;
     }
@@ -1182,6 +1184,7 @@ matchInput()
     char command;
     const char* fieldName = NULL;
     StreamBuffer formatstring;
+    ssize_t delta = 0;
 
     consumedInput = 0;
 
@@ -1216,7 +1219,7 @@ normal_format:
                 debug("StreamCore::matchInput(%s): format = \"%%%s\"\n",
                     name(), formatstring());
 
-                if (fmt.flags & skip_flag || fmt.type == pseudo_format)
+                if (fmt.flags & skip_flag || fmt.type == pseudo_format || fmt.type == needs_original_format)
                 {
                     long ldummy;
                     double ddummy;
@@ -1238,9 +1241,20 @@ normal_format:
                                 scanString(fmt, inputLine(consumedInput), NULL, size);
                             break;
                         case pseudo_format:
-                            // pass complete input
+                            // pass complete input line for scan and/or re-write
+                            size = inputLine.length();
                             consumed = StreamFormatConverter::find(fmt.conv)->
                                 scanPseudo(fmt, inputLine, consumedInput);
+                            delta += inputLine.length() - size; // track length changes
+                            debug("after rewrite delta=%zi\n", delta);
+                            break;
+                        case needs_original_format:
+                            // pass original input with adjusted current position
+                            debug("before checksum delta=%zi\n", delta);
+                            consumedInput -= delta; // correct for length changes
+                            consumed = StreamFormatConverter::find(fmt.conv)->
+                                scanPseudo(fmt, inputBuffer, consumedInput);
+                            consumedInput += delta;
                             break;
                         default:
                             error("INTERNAL ERROR (%s): illegal format.type 0x%02x\n",

src/StreamCore.h

@@ -219,6 +219,7 @@ protected:
 
 // virtual methods
     virtual void protocolStartHook() {}
+    virtual void inputHook(const void* input, size_t size) {};
     virtual void protocolFinishHook(ProtocolResult) {}
     virtual void startTimer(unsigned long timeout) = 0;
     virtual bool formatValue(const StreamFormat&, const void* fieldaddress) = 0;

src/StreamEpics.cc

@@ -137,6 +137,7 @@ class Stream : protected StreamCore
 #endif
 
 // StreamCore methods
+    void inputHook(const void* input, size_t size);
     void protocolFinishHook(ProtocolResult);
     void startTimer(unsigned long timeout);
     bool getFieldAddress(const char* fieldname,
@@ -934,6 +935,10 @@ process()
     debug("Stream::process(%s) start\n", name());
     status = NO_ALARM;
     convert = OK;
+    if (record->tpro)
+    {
+        StreamDebugClass(record->name).print("start protocol '%s'\n", protocolname());
+    }
     if (!startProtocol(record->proc == 2 ? StreamCore::StartInit : StreamCore::StartNormal))
     {
         debug("Stream::process(%s): could not start %sprotocol, status=%s (%d)\n",
@@ -1028,11 +1033,26 @@ expire(const epicsTime&)
 
 // StreamCore virtual methods ////////////////////////////////////////////
 
+void Stream::
+inputHook(const void* input, size_t size)
+{
+    if (record->tpro > 1)
+    {
+        StreamDebugClass(record->name).print("received \"%s\"\n",
+            StreamBuffer(input, size).expand()());
+    }
+}
+
 void Stream::
 protocolFinishHook(ProtocolResult result)
 {
     debug("Stream::protocolFinishHook(%s, %s)\n",
         name(), toStr(result));
+    if (record->tpro)
+    {
+        StreamDebugClass(record->name).print("%s. out=\"%s\", in=\"%s\"\n",
+            toStr(result), outputLine.expand()(), inputLine.expand()());
+    }
     switch (result)
     {
         case Success:

src/StreamError.cc

@@ -171,8 +171,6 @@ print(const char* fmt, ...)
 {
     va_list args;
     va_start(args, fmt);
-    const char* f = strrchr(file, '/');
-    if (f) f++; else f = file;
     FILE* fp = StreamDebugFile ? StreamDebugFile : stderr;
     if (streamMsgTimeStamped)
     {
@@ -184,7 +182,13 @@ print(const char* fmt, ...)
     {
         fprintf(fp, "%s ", StreamGetThreadNameFunction());
     }
-    fprintf(fp, "%s:%d: ", f, line);
+    if (file) {
+        const char* f = strrchr(file, '/');
+        if (f) f++; else f = file;
+        fprintf(fp, "%s:", f);
+        if (line) fprintf(fp, "%d:", line);
+        fprintf(fp, " ");
+    }
     vfprintf(fp, fmt, args);
     fflush(fp);
     va_end(args);

src/StreamError.h

@@ -56,19 +56,15 @@ class StreamDebugClass
     const char* file;
     int line;
 public:
-    StreamDebugClass(const char* file, int line) :
+    StreamDebugClass(const char* file = NULL, int line = 0) :
         file(file), line(line) {}
     int print(const char* fmt, ...)
         __attribute__((__format__(__printf__,2,3)));
 };
 
-inline StreamDebugClass
-StreamDebugObject(const char* file, int line)
-{ return StreamDebugClass(file, line); }
-
 #define error StreamError
-#define debug (!streamDebug)?0:StreamDebugObject(__FILE__,__LINE__).print
-#define debug2 (streamDebug<2)?0:StreamDebugObject(__FILE__,__LINE__).print
+#define debug (!streamDebug)?0:StreamDebugClass(__FILE__,__LINE__).print
+#define debug2 (streamDebug<2)?0:StreamDebugClass(__FILE__,__LINE__).print
 
 /*
  * ANSI escape sequences for terminal output

src/StreamFormat.h

@@ -42,7 +42,8 @@ typedef enum {
     enum_format,
     double_format,
     string_format,
-    pseudo_format
+    pseudo_format,
+    needs_original_format
 } StreamFormatType;
 
 extern const char* StreamFormatTypeStr[];

streamApp/Makefile

@@ -42,10 +42,12 @@ PROD_SRCS_vxWorks = -nil-
 PROD_LIBS = stream
 
 ifdef ASYN
-# edit asynRegistrars.dbd if necessary
-streamApp_DBD += asynRegistrars.dbd
-# add asynRecord.dbd if you like
-streamApp_DBD += asynRecord.dbd
+streamApp_DBD += asyn.dbd
+streamApp_DBD += drvAsynIPPort.dbd
+streamApp_DBD += drvAsynSerialPort.dbd
+# vxi11 support is optional in recent asyn versions
+#streamApp_DBD += drvVxi11.dbd
+
 PROD_LIBS += asyn
 # cygwin needs separate RPC library for asyn
 PROD_SYS_LIBS_cygwin32 += $(CYGWIN_RPC_LIB)

streamApp/asynRegistrars.dbd

@@ -1,9 +0,0 @@
-registrar(asynRegister)
-registrar(asynInterposeFlushRegister)
-registrar(asynInterposeEosRegister)
-
-# asynDriver up to version 4-16 does not support serial port for Windows!
-registrar(drvAsynSerialPortRegisterCommands)
-registrar(drvAsynIPPortRegisterCommands)
-registrar(drvAsynIPServerPortRegisterCommands)
-registrar(vxi11RegisterCommands)

streamApp/simple.proto

@@ -19,7 +19,7 @@
 # along with StreamDevice. If not, see https://www.gnu.org/licenses/.
 #########################################################################/
 
-terminator = CR LF;
+terminator = LF;
 
 cmd {
     out "%s";

streamApp/tests/testChecksum

@@ -256,6 +256,9 @@ set protocol {
         out "bitsum8  %s %#-9.1<bitsum8>";   in "bitsum8  %=s %#-9.1<bitsum8>";
         out "bitsum16 %s %#-9.1<bitsum16>";  in "bitsum16 %=s %#-9.1<bitsum16>";
         out "bitsum32 %s %#-9.1<bitsum32>";  in "bitsum32 %=s %#-9.1<bitsum32>";
+
+# Check combination of regsub and checksum. Always check what the device sees.
+        out "crc8     %s%#/[0-9]{2}/&:/ %9.1<crc8>";        in "crc8     %#/[0-9]{2}/&:/%s %9.1<crc8>"; out "%s";
         out "DONE";
     }
 }
@@ -740,6 +743,12 @@ assure "bitsum16 123456789 \x32\x31\x30\x30\n"
 send   "bitsum16 123456789 \x32\x31\x30\x30\n"
 assure "bitsum32 123456789 \x32\x31\x30\x30\x30\x30\x30\x30\n"
 send   "bitsum32 123456789 \x32\x31\x30\x30\x30\x30\x30\x30\n"
+
+# check regsub and checksums
+assure "crc8     12:34:56:78:9 \x07\n" ;# modified output string and checksum
+send   "crc8     123456789 \xF4\n"     ;# original input string and checksum
+assure "12:34:56:78:9\n"               ;# modified input string
+
 assure "DONE\n"
                 
 finish