epics/epics-base
0
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
Files
6595ff9518fb6b3d91c55fdf1590607e7c6c0671
epics-base/src/libCom/flex/flexdoc.html
Michael Davidsaver f7fc564556 Fold antelope/flex and asHost into libCom 
Build lexer and parser from libCom/Makefile.
Since libCom now includes asLib.c and asLib_lex.c we must build
antelope and flex without linking them to Com.  This works because
they only need epicsTempFile anyway.  However make doesn't like a
subdirectory with the same name as a target object, so the antelope
source directory is now called yacc.  The two main.c files were also
renamed to avoid other build problems.

Merge asHost into Com and remove mentions in CONFIG_BASE

Lots of noise since SRCS must be renamed to Com_SRCS
2011-02-25 15:39:44 -06:00

1891 lines
76 KiB
HTML
Raw Blame History

/*************************************************************************\
* Copyright (c) 2002 The University of Chicago, as Operator of Argonne
* National Laboratory.
* Copyright (c) 2002 The Regents of the University of California, as
* Operator of Los Alamos National Laboratory.
* EPICS BASE Versions 3.13.7
* and higher are distributed subject to a Software License Agreement found
* in file LICENSE that is included with this distribution.
\*************************************************************************/
<HTML>
<BODY>
<PRE>
<!-- Manpage converted by man2html 3.0.1 -->
</PRE>
<H2>NAME</H2><PRE>
flex - fast lexical analyzer generator
</PRE>
<H2>SYNOPSIS</H2><PRE>
flex [-bcdfinpstvFILT8 -C[efmF] -Sskeleton] [<I>filename</I> ...]
</PRE>
<H2>DESCRIPTION</H2><PRE>
<I>flex</I> is a tool for generating <I>scanners</I>: programs which
recognized lexical patterns in text. <I>flex</I> reads the given
input files, or its standard input if no file names are
given, for a description of a scanner to generate. The
description is in the form of pairs of regular expressions
and C code, called <I>rules</I>. <I>flex</I> generates as output a C
source file, lex.yy.c, which defines a routine yylex(). This
file is compiled and linked with the -lfl library to produce
an executable. When the executable is run, it analyzes its
input for occurrences of the regular expressions. Whenever
it finds one, it executes the corresponding C code.
</PRE>
<H2>SOME SIMPLE EXAMPLES</H2><PRE>
First some simple examples to get the flavor of how one uses
<I>flex</I>. The following <I>flex</I> input specifies a scanner which
whenever it encounters the string "username" will replace it
with the user's login name:
%%
username printf( "%s", getlogin() );
By default, any text not matched by a <I>flex</I> scanner is copied
to the output, so the net effect of this scanner is to copy
its input file to its output with each occurrence of "user-
name" expanded. In this input, there is just one rule.
"username" is the <I>pattern</I> and the "printf" is the <I>action</I>.
The "%%" marks the beginning of the rules.
Here's another simple example:
int num_lines = 0, num_chars = 0;
%%
\n ++num_lines; ++num_chars;
. ++num_chars;
%%
main()
{
yylex();
printf( "# of lines = %d, # of chars = %d\n",
num_lines, num_chars );
}
This scanner counts the number of characters and the number
of lines in its input (it produces no output other than the
final report on the counts). The first line declares two
globals, "num_lines" and "num_chars", which are accessible
both inside yylex() and in the main() routine declared after
the second "%%". There are two rules, one which matches a
newline ("\n") and increments both the line count and the
character count, and one which matches any character other
than a newline (indicated by the "." regular expression).
A somewhat more complicated example:
/* scanner for a toy Pascal-like language */
%{
/* need this for the call to atof() below */
#include &lt;math.h&gt;
%}
DIGIT [0-9]
ID [a-z][a-z0-9]*
%%
{DIGIT}+ {
printf( "An integer: %s (%d)\n", yytext,
atoi( yytext ) );
}
{DIGIT}+"."{DIGIT}* {
printf( "A float: %s (%g)\n", yytext,
atof( yytext ) );
}
if|then|begin|end|procedure|function {
printf( "A keyword: %s\n", yytext );
}
{ID} printf( "An identifier: %s\n", yytext );
"+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext );
"{"[^}\n]*"}" /* eat up one-line comments */
[ \t\n]+ /* eat up whitespace */
. printf( "Unrecognized character: %s\n", yytext );
%%
main( argc, argv )
int argc;
char **argv;
{
++argv, --argc; /* skip over program name */
if ( argc &gt; 0 )
yyin = fopen( argv[0], "r" );
else
yyin = stdin;
yylex();
}
This is the beginnings of a simple scanner for a language
like Pascal. It identifies different types of <I>tokens</I> and
reports on what it has seen.
The details of this example will be explained in the follow-
ing sections.
</PRE>
<H2>FORMAT OF THE INPUT FILE</H2><PRE>
The <I>flex</I> input file consists of three sections, separated by
a line with just %% in it:
definitions
%%
rules
%%
user code
The <I>definitions</I> section contains declarations of simple <I>name</I>
definitions to simplify the scanner specification, and
declarations of <I>start</I> <I>conditions</I>, which are explained in a
later section.
Name definitions have the form:
name definition
The "name" is a word beginning with a letter or an under-
score ('_') followed by zero or more letters, digits, '_',
or '-' (dash). The definition is taken to begin at the
first non-white-space character following the name and con-
tinuing to the end of the line. The definition can subse-
quently be referred to using "{name}", which will expand to
"(definition)". For example,
DIGIT [0-9]
ID [a-z][a-z0-9]*
defines "DIGIT" to be a regular expression which matches a
single digit, and "ID" to be a regular expression which
matches a letter followed by zero-or-more letters-or-digits.
A subsequent reference to
{DIGIT}+"."{DIGIT}*
is identical to
([0-9])+"."([0-9])*
and matches one-or-more digits followed by a '.' followed by
zero-or-more digits.
The <I>rules</I> section of the <I>flex</I> input contains a series of
rules of the form:
pattern action
where the pattern must be unindented and the action must
begin on the same line.
See below for a further description of patterns and actions.
Finally, the user code section is simply copied to lex.yy.c
verbatim. It is used for companion routines which call or
are called by the scanner. The presence of this section is
optional; if it is missing, the second %% in the input file
may be skipped, too.
In the definitions and rules sections, any <I>indented</I> text or
text enclosed in %{ and %} is copied verbatim to the output
(with the %{}'s removed). The %{}'s must appear unindented
on lines by themselves.
In the rules section, any indented or %{} text appearing
before the first rule may be used to declare variables which
are local to the scanning routine and (after the declara-
tions) code which is to be executed whenever the scanning
routine is entered. Other indented or %{} text in the rule
section is still copied to the output, but its meaning is
not well-defined and it may well cause compile-time errors
(this feature is present for <I>POSIX</I> compliance; see below for
other such features).
In the definitions section, an unindented comment (i.e., a
line beginning with "/*") is also copied verbatim to the
output up to the next "*/". Also, any line in the defini-
tions section beginning with '#' is ignored, though this
style of comment is deprecated and may go away in the
future.
</PRE>
<H2>PATTERNS</H2><PRE>
The patterns in the input are written using an extended set
of regular expressions. These are:
x match the character 'x'
. any character except newline
[xyz] a "character class"; in this case, the pattern
matches either an 'x', a 'y', or a 'z'
[abj-oZ] a "character class" with a range in it; matches
an 'a', a 'b', any letter from 'j' through 'o',
or a 'Z'
[^A-Z] a "negated character class", i.e., any character
but those in the class. In this case, any
character EXCEPT an uppercase letter.
[^A-Z\n] any character EXCEPT an uppercase letter or
a newline
r* zero or more r's, where r is any regular expression
r+ one or more r's
r? zero or one r's (that is, "an optional r")
r{2,5} anywhere from two to five r's
r{2,} two or more r's
r{4} exactly 4 r's
{name} the expansion of the "name" definition
(see above)
"[xyz]\"foo"
the literal string: [xyz]"foo
\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
then the ANSI-C interpretation of \x.
Otherwise, a literal 'X' (used to escape
operators such as '*')
\123 the character with octal value 123
\x2a the character with hexadecimal value 2a
(r) match an r; parentheses are used to override
precedence (see below)
rs the regular expression r followed by the
regular expression s; called "concatenation"
r|s either an r or an s
r/s an r but only if it is followed by an s. The
s is not part of the matched text. This type
of pattern is called as "trailing context".
^r an r, but only at the beginning of a line
r$ an r, but only at the end of a line. Equivalent
to "r/\n".
&lt;s&gt;r an r, but only in start condition s (see
below for discussion of start conditions)
&lt;s1,s2,s3&gt;r
same, but in any of start conditions s1,
s2, or s3
&lt;&lt;EOF&gt;&gt; an end-of-file
&lt;s1,s2&gt;&lt;&lt;EOF&gt;&gt;
an end-of-file when in start condition s1 or s2
The regular expressions listed above are grouped according
to precedence, from highest precedence at the top to lowest
at the bottom. Those grouped together have equal pre-
cedence. For example,
foo|bar*
is the same as
(foo)|(ba(r*))
since the '*' operator has higher precedence than concatena-
tion, and concatenation higher than alternation ('|'). This
pattern therefore matches <I>either</I> the string "foo" <I>or</I> the
string "ba" followed by zero-or-more r's. To match "foo" or
zero-or-more "bar"'s, use:
foo|(bar)*
and to match zero-or-more "foo"'s-or-"bar"'s:
(foo|bar)*
Some notes on patterns:
- A negated character class such as the example "[^A-Z]"
above <I>will</I> <I>match</I> <I>a</I> <I>newline</I> unless "\n" (or an
equivalent escape sequence) is one of the characters
explicitly present in the negated character class
(e.g., "[^A-Z\n]"). This is unlike how many other reg-
ular expression tools treat negated character classes,
but unfortunately the inconsistency is historically
entrenched. Matching newlines means that a pattern
like [^"]* can match an entire input (overflowing the
scanner's input buffer) unless there's another quote in
the input.
- A rule can have at most one instance of trailing con-
text (the '/' operator or the '$' operator). The start
condition, '^', and "&lt;&lt;EOF&gt;&gt;" patterns can only occur
at the beginning of a pattern, and, as well as with '/'
and '$', cannot be grouped inside parentheses. A '^'
which does not occur at the beginning of a rule or a
'$' which does not occur at the end of a rule loses its
special properties and is treated as a normal charac-
ter.
The following are illegal:
foo/bar$
&lt;sc1&gt;foo&lt;sc2&gt;bar
Note that the first of these, can be written
"foo/bar\n".
The following will result in '$' or '^' being treated
as a normal character:
foo|(bar$)
foo|^bar
If what's wanted is a "foo" or a bar-followed-by-a-
newline, the following could be used (the special '|'
action is explained below):
foo |
bar$ /* action goes here */
A similar trick will work for matching a foo or a bar-
at-the-beginning-of-a-line.
</PRE>
<H2>HOW THE INPUT IS MATCHED</H2><PRE>
When the generated scanner is run, it analyzes its input
looking for strings which match any of its patterns. If it
finds more than one match, it takes the one matching the
most text (for trailing context rules, this includes the
length of the trailing part, even though it will then be
returned to the input). If it finds two or more matches of
the same length, the rule listed first in the <I>flex</I> input
file is chosen.
Once the match is determined, the text corresponding to the
match (called the <I>token</I>) is made available in the global
character pointer yytext, and its length in the global
integer yyleng. The <I>action</I> corresponding to the matched pat-
tern is then executed (a more detailed description of
actions follows), and then the remaining input is scanned
for another match.
If no match is found, then the <I>default</I> <I>rule</I> is executed: the
next character in the input is considered matched and copied
to the standard output. Thus, the simplest legal <I>flex</I> input
is:
%%
which generates a scanner that simply copies its input (one
character at a time) to its output.
</PRE>
<H2>ACTIONS</H2><PRE>
Each pattern in a rule has a corresponding action, which can
be any arbitrary C statement. The pattern ends at the first
non-escaped whitespace character; the remainder of the line
is its action. If the action is empty, then when the pat-
tern is matched the input token is simply discarded. For
example, here is the specification for a program which
deletes all occurrences of "zap me" from its input:
%%
"zap me"
(It will copy all other characters in the input to the out-
put since they will be matched by the default rule.)
Here is a program which compresses multiple blanks and tabs
down to a single blank, and throws away whitespace found at
the end of a line:
%%
[ \t]+ putchar( ' ' );
[ \t]+$ /* ignore this token */
If the action contains a '{', then the action spans till the
balancing '}' is found, and the action may cross multiple
lines. <I>flex</I> knows about C strings and comments and won't be
fooled by braces found within them, but also allows actions
to begin with %{ and will consider the action to be all the
text up to the next %} (regardless of ordinary braces inside
the action).
An action consisting solely of a vertical bar ('|') means
"same as the action for the next rule." See below for an
illustration.
Actions can include arbitrary C code, including return
statements to return a value to whatever routine called
yylex(). Each time yylex() is called it continues processing
tokens from where it last left off until it either reaches
the end of the file or executes a return. Once it reaches
an end-of-file, however, then any subsequent call to yylex()
will simply immediately return, unless yyrestart() is first
called (see below).
Actions are not allowed to modify yytext or yyleng.
There are a number of special directives which can be
included within an action:
- ECHO copies yytext to the scanner's output.
- BEGIN followed by the name of a start condition places
the scanner in the corresponding start condition (see
below).
- REJECT directs the scanner to proceed on to the "second
best" rule which matched the input (or a prefix of the
input). The rule is chosen as described above in "How
the Input is Matched", and yytext and yyleng set up
appropriately. It may either be one which matched as
much text as the originally chosen rule but came later
in the <I>flex</I> input file, or one which matched less text.
For example, the following will both count the words in
the input and call the routine special() whenever
"frob" is seen:
int word_count = 0;
%%
frob special(); REJECT;
[^ \t\n]+ ++word_count;
Without the REJECT, any "frob"'s in the input would not
be counted as words, since the scanner normally exe-
cutes only one action per token. Multiple REJECT's are
allowed, each one finding the next best choice to the
currently active rule. For example, when the following
scanner scans the token "abcd", it will write "abcdab-
caba" to the output:
%%
a |
ab |
abc |
abcd ECHO; REJECT;
.|\n /* eat up any unmatched character */
(The first three rules share the fourth's action since
they use the special '|' action.) REJECT is a particu-
larly expensive feature in terms scanner performance;
if it is used in <I>any</I> of the scanner's actions it will
slow down <I>all</I> of the scanner's matching. Furthermore,
REJECT cannot be used with the -<I>f</I> or -<I>F</I> options (see
below).
Note also that unlike the other special actions, REJECT
is a <I>branch</I>; code immediately following it in the
action will <I>not</I> be executed.
- yymore() tells the scanner that the next time it
matches a rule, the corresponding token should be
<I>appended</I> onto the current value of yytext rather than
replacing it. For example, given the input "mega-
kludge" the following will write "mega-mega-kludge" to
the output:
%%
mega- ECHO; yymore();
kludge ECHO;
First "mega-" is matched and echoed to the output.
Then "kludge" is matched, but the previous "mega-" is
still hanging around at the beginning of yytext so the
ECHO for the "kludge" rule will actually write "mega-
kludge". The presence of yymore() in the scanner's
action entails a minor performance penalty in the
scanner's matching speed.
- yyless(n) returns all but the first <I>n</I> characters of the
current token back to the input stream, where they will
be rescanned when the scanner looks for the next match.
yytext and yyleng are adjusted appropriately (e.g.,
yyleng will now be equal to <I>n</I> ). For example, on the
input "foobar" the following will write out "foobar-
bar":
%%
foobar ECHO; <B>yyless(3)</B>;
[a-z]+ ECHO;
An argument of 0 to yyless will cause the entire
current input string to be scanned again. Unless
you've changed how the scanner will subsequently pro-
cess its input (using BEGIN, for example), this will
result in an endless loop.
- unput(c) puts the character <I>c</I> back onto the input
stream. It will be the next character scanned. The
following action will take the current token and cause
it to be rescanned enclosed in parentheses.
{
int i;
unput( ')' );
for ( i = yyleng - 1; i &gt;= 0; --i )
unput( yytext[i] );
unput( '(' );
}
Note that since each unput() puts the given character
back at the <I>beginning</I> of the input stream, pushing back
strings must be done back-to-front.
- input() reads the next character from the input stream.
For example, the following is one way to eat up C
comments:
%%
"/*" {
register int c;
for ( ; ; )
{
while ( (c = input()) != '*' &amp;&amp;
c != EOF )
; /* eat up text of comment */
if ( c == '*' )
{
while ( (c = input()) == '*' )
;
if ( c == '/' )
break; /* found the end */
}
if ( c == EOF )
{
error( "EOF in comment" );
break;
}
}
}
(Note that if the scanner is compiled using C++, then
input() is instead referred to as yyinput(), in order
to avoid a name clash with the C++ stream by the name
of <I>input</I>.)
- yyterminate() can be used in lieu of a return statement
in an action. It terminates the scanner and returns a
0 to the scanner's caller, indicating "all done". Sub-
sequent calls to the scanner will immediately return
unless preceded by a call to yyrestart() (see below).
By default, yyterminate() is also called when an end-
of-file is encountered. It is a macro and may be rede-
fined.
</PRE>
<H2>THE GENERATED SCANNER</H2><PRE>
The output of <I>flex</I> is the file lex.yy.c, which contains the
scanning routine yylex(), a number of tables used by it for
matching tokens, and a number of auxiliary routines and mac-
ros. By default, yylex() is declared as follows:
int yylex()
{
... various definitions and the actions in here ...
}
(If your environment supports function prototypes, then it
will be "int yylex( void )".) This definition may be
changed by redefining the "YY_DECL" macro. For example, you
could use:
#undef YY_DECL
#define YY_DECL float lexscan( a, b ) float a, b;
to give the scanning routine the name <I>lexscan</I>, returning a
float, and taking two floats as arguments. Note that if you
give arguments to the scanning routine using a K&amp;R-
style/non-prototyped function declaration, you must ter-
minate the definition with a semi-colon (;).
Whenever yylex() is called, it scans tokens from the global
input file <I>yyin</I> (which defaults to stdin). It continues
until it either reaches an end-of-file (at which point it
returns the value 0) or one of its actions executes a <I>return</I>
statement. In the former case, when called again the
scanner will immediately return unless yyrestart() is called
to point <I>yyin</I> at the new input file. ( yyrestart() takes
one argument, a FILE * pointer.) In the latter case (i.e.,
when an action executes a return), the scanner may then be
called again and it will resume scanning where it left off.
By default (and for purposes of efficiency), the scanner
uses block-reads rather than simple <I>getc</I>() calls to read
characters from <I>yyin</I>. The nature of how it gets its input
can be controlled by redefining the YY_INPUT macro.
YY_INPUT's calling sequence is
"YY_INPUT(buf,result,max_size)". Its action is to place up
to <I>max</I>_<I>size</I> characters in the character array <I>buf</I> and return
in the integer variable <I>result</I> either the number of charac-
ters read or the constant YY_NULL (0 on Unix systems) to
indicate EOF. The default YY_INPUT reads from the global
file-pointer "yyin".
A sample redefinition of YY_INPUT (in the definitions sec-
tion of the input file):
%{
#undef YY_INPUT
#define YY_INPUT(buf,result,max_size) \
{ \
int c = getchar(); \
result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
}
%}
This definition will change the input processing to occur
one character at a time.
You also can add in things like keeping track of the input
line number this way; but don't expect your scanner to go
very fast.
When the scanner receives an end-of-file indication from
YY_INPUT, it then checks the yywrap() function. If yywrap()
returns false (zero), then it is assumed that the function
has gone ahead and set up <I>yyin</I> to point to another input
file, and scanning continues. If it returns true (non-
zero), then the scanner terminates, returning 0 to its
caller.
The default yywrap() always returns 1. Presently, to rede-
fine it you must first "#undef yywrap", as it is currently
implemented as a macro. As indicated by the hedging in the
previous sentence, it may be changed to a true function in
the near future.
The scanner writes its ECHO output to the <I>yyout</I> global
(default, stdout), which may be redefined by the user simply
by assigning it to some other FILE pointer.
</PRE>
<H2>START CONDITIONS</H2><PRE>
<I>flex</I> provides a mechanism for conditionally activating
rules. Any rule whose pattern is prefixed with "&lt;sc&gt;" will
only be active when the scanner is in the start condition
named "sc". For example,
&lt;STRING&gt;[^"]* { /* eat up the string body ... */
...
}
will be active only when the scanner is in the "STRING"
start condition, and
&lt;INITIAL,STRING,QUOTE&gt;\. { /* handle an escape ... */
...
}
will be active only when the current start condition is
either "INITIAL", "STRING", or "QUOTE".
Start conditions are declared in the definitions (first)
section of the input using unindented lines beginning with
either %s or %x followed by a list of names. The former
declares <I>inclusive</I> start conditions, the latter <I>exclusive</I>
start conditions. A start condition is activated using the
BEGIN action. Until the next BEGIN action is executed,
rules with the given start condition will be active and
rules with other start conditions will be inactive. If the
start condition is <I>inclusive</I>, then rules with no start con-
ditions at all will also be active. If it is <I>exclusive</I>,
then <I>only</I> rules qualified with the start condition will be
active. A set of rules contingent on the same exclusive
start condition describe a scanner which is independent of
any of the other rules in the <I>flex</I> input. Because of this,
exclusive start conditions make it easy to specify "mini-
scanners" which scan portions of the input that are syntac-
tically different from the rest (e.g., comments).
If the distinction between inclusive and exclusive start
conditions is still a little vague, here's a simple example
illustrating the connection between the two. The set of
rules:
%s example
%%
&lt;example&gt;foo /* do something */
is equivalent to
%x example
%%
&lt;INITIAL,example&gt;foo /* do something */
The default rule (to ECHO any unmatched character) remains
active in start conditions.
<B>BEGIN(0)</B> returns to the original state where only the rules
with no start conditions are active. This state can also be
referred to as the start-condition "INITIAL", so
BEGIN(INITIAL) is equivalent to <B>BEGIN(0)</B>. (The parentheses
around the start condition name are not required but are
considered good style.)
BEGIN actions can also be given as indented code at the
beginning of the rules section. For example, the following
will cause the scanner to enter the "SPECIAL" start condi-
tion whenever <I>yylex</I>() is called and the global variable
<I>enter</I>_<I>special</I> is true:
int enter_special;
%x SPECIAL
%%
if ( enter_special )
BEGIN(SPECIAL);
&lt;SPECIAL&gt;blahblahblah
...more rules follow...
To illustrate the uses of start conditions, here is a
scanner which provides two different interpretations of a
string like "123.456". By default it will treat it as as
three tokens, the integer "123", a dot ('.'), and the
integer "456". But if the string is preceded earlier in the
line by the string "expect-floats" it will treat it as a
single token, the floating-point number 123.456:
%{
#include &lt;math.h&gt;
%}
%s expect
%%
expect-floats BEGIN(expect);
&lt;expect&gt;[0-9]+"."[0-9]+ {
printf( "found a float, = %f\n",
atof( yytext ) );
}
&lt;expect&gt;\n {
/* that's the end of the line, so
* we need another "expect-number"
* before we'll recognize any more
* numbers
*/
BEGIN(INITIAL);
}
[0-9]+ {
printf( "found an integer, = %d\n",
atoi( yytext ) );
}
"." printf( "found a dot\n" );
Here is a scanner which recognizes (and discards) C comments
while maintaining a count of the current input line.
%x comment
%%
int line_num = 1;
"/*" BEGIN(comment);
&lt;comment&gt;[^*\n]* /* eat anything that's not a '*' */
&lt;comment&gt;"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
&lt;comment&gt;\n ++line_num;
&lt;comment&gt;"*"+"/" BEGIN(INITIAL);
Note that start-conditions names are really integer values
and can be stored as such. Thus, the above could be
extended in the following fashion:
%x comment foo
%%
int line_num = 1;
int comment_caller;
"/*" {
comment_caller = INITIAL;
BEGIN(comment);
}
...
&lt;foo&gt;"/*" {
comment_caller = foo;
BEGIN(comment);
}
&lt;comment&gt;[^*\n]* /* eat anything that's not a '*' */
&lt;comment&gt;"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
&lt;comment&gt;\n ++line_num;
&lt;comment&gt;"*"+"/" BEGIN(comment_caller);
One can then implement a "stack" of start conditions using
an array of integers. (It is likely that such stacks will
become a full-fledged <I>flex</I> feature in the future.) Note,
though, that start conditions do not have their own name-
space; %s's and %x's declare names in the same fashion as
#define's.
</PRE>
<H2>MULTIPLE INPUT BUFFERS</H2><PRE>
Some scanners (such as those which support "include" files)
require reading from several input streams. As <I>flex</I>
scanners do a large amount of buffering, one cannot control
where the next input will be read from by simply writing a
YY_INPUT which is sensitive to the scanning context.
YY_INPUT is only called when the scanner reaches the end of
its buffer, which may be a long time after scanning a state-
ment such as an "include" which requires switching the input
source.
To negotiate these sorts of problems, <I>flex</I> provides a
mechanism for creating and switching between multiple input
buffers. An input buffer is created by using:
YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )
which takes a <I>FILE</I> pointer and a size and creates a buffer
associated with the given file and large enough to hold <I>size</I>
characters (when in doubt, use YY_BUF_SIZE for the size).
It returns a YY_BUFFER_STATE handle, which may then be
passed to other routines:
void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )
switches the scanner's input buffer so subsequent tokens
will come from <I>new</I>_<I>buffer</I>. Note that yy_switch_to_buffer()
may be used by yywrap() to sets things up for continued
scanning, instead of opening a new file and pointing <I>yyin</I> at
it.
void yy_delete_buffer( YY_BUFFER_STATE buffer )
is used to reclaim the storage associated with a buffer.
yy_new_buffer() is an alias for yy_create_buffer(), provided
for compatibility with the C++ use of <I>new</I> and <I>delete</I> for
creating and destroying dynamic objects.
Finally, the YY_CURRENT_BUFFER macro returns a
YY_BUFFER_STATE handle to the current buffer.
Here is an example of using these features for writing a
scanner which expands include files (the &lt;&lt;EOF&gt;&gt; feature is
discussed below):
/* the "incl" state is used for picking up the name
* of an include file
*/
%x incl
%{
#define MAX_INCLUDE_DEPTH 10
YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
int include_stack_ptr = 0;
%}
%%
include BEGIN(incl);
[a-z]+ ECHO;
[^a-z\n]*\n? ECHO;
&lt;incl&gt;[ \t]* /* eat the whitespace */
&lt;incl&gt;[^ \t\n]+ { /* got the include file name */
if ( include_stack_ptr &gt;= MAX_INCLUDE_DEPTH )
{
fprintf( stderr, "Includes nested too deeply" );
exit( 1 );
}
include_stack[include_stack_ptr++] =
YY_CURRENT_BUFFER;
yyin = fopen( yytext, "r" );
if ( ! yyin )
error( ... );
yy_switch_to_buffer(
yy_create_buffer( yyin, YY_BUF_SIZE ) );
BEGIN(INITIAL);
}
&lt;&lt;EOF&gt;&gt; {
if ( --include_stack_ptr &lt; 0 )
{
yyterminate();
}
else
yy_switch_to_buffer(
include_stack[include_stack_ptr] );
}
</PRE>
<H2>END-OF-FILE RULES</H2><PRE>
The special rule "&lt;&lt;EOF&gt;&gt;" indicates actions which are to be
taken when an end-of-file is encountered and yywrap()
returns non-zero (i.e., indicates no further files to pro-
cess). The action must finish by doing one of four things:
- the special YY_NEW_FILE action, if <I>yyin</I> has been
pointed at a new file to process;
- a <I>return</I> statement;
- the special yyterminate() action;
- or, switching to a new buffer using
yy_switch_to_buffer() as shown in the example above.
&lt;&lt;EOF&gt;&gt; rules may not be used with other patterns; they may
only be qualified with a list of start conditions. If an
unqualified &lt;&lt;EOF&gt;&gt; rule is given, it applies to <I>all</I> start
conditions which do not already have &lt;&lt;EOF&gt;&gt; actions. To
specify an &lt;&lt;EOF&gt;&gt; rule for only the initial start condi-
tion, use
&lt;INITIAL&gt;&lt;&lt;EOF&gt;&gt;
These rules are useful for catching things like unclosed
comments. An example:
%x quote
%%
...other rules for dealing with quotes...
&lt;quote&gt;&lt;&lt;EOF&gt;&gt; {
error( "unterminated quote" );
yyterminate();
}
&lt;&lt;EOF&gt;&gt; {
if ( *++filelist )
{
yyin = fopen( *filelist, "r" );
YY_NEW_FILE;
}
else
yyterminate();
}
</PRE>
<H2>MISCELLANEOUS MACROS</H2><PRE>
The macro YY_USER_ACTION can be redefined to provide an
action which is always executed prior to the matched rule's
action. For example, it could be #define'd to call a rou-
tine to convert yytext to lower-case.
The macro YY_USER_INIT may be redefined to provide an action
which is always executed before the first scan (and before
the scanner's internal initializations are done). For exam-
ple, it could be used to call a routine to read in a data
table or open a logging file.
In the generated scanner, the actions are all gathered in
one large switch statement and separated using YY_BREAK,
which may be redefined. By default, it is simply a "break",
to separate each rule's action from the following rule's.
Redefining YY_BREAK allows, for example, C++ users to
#define YY_BREAK to do nothing (while being very careful
that every rule ends with a "break" or a "return"!) to avoid
suffering from unreachable statement warnings where because
a rule's action ends with "return", the YY_BREAK is inacces-
sible.
</PRE>
<H2>INTERFACING WITH YACC</H2><PRE>
One of the main uses of <I>flex</I> is as a companion to the <I>yacc</I>
parser-generator. <I>yacc</I> parsers expect to call a routine
named yylex() to find the next input token. The routine is
supposed to return the type of the next token as well as
putting any associated value in the global yylval. To use
<I>flex</I> with <I>yacc</I>, one specifies the -d option to <I>yacc</I> to
instruct it to generate the file y.tab.h containing defini-
tions of all the %tokens appearing in the <I>yacc</I> input. This
file is then included in the <I>flex</I> scanner. For example, if
one of the tokens is "TOK_NUMBER", part of the scanner might
look like:
%{
#include "y.tab.h"
%}
%%
[0-9]+ yylval = atoi( yytext ); return TOK_NUMBER;
</PRE>
<H2>TRANSLATION TABLE</H2><PRE>
In the name of POSIX compliance, <I>flex</I> supports a <I>translation</I>
<I>table</I> for mapping input characters into groups. The table
is specified in the first section, and its format looks
like:
%t
1 abcd
2 ABCDEFGHIJKLMNOPQRSTUVWXYZ
52 0123456789
6 \t\ \n
%t
This example specifies that the characters 'a', 'b', 'c',
and 'd' are to all be lumped into group #1, upper-case
letters in group #2, digits in group #52, tabs, blanks, and
newlines into group #6, and <I>no</I> <I>other</I> <I>characters</I> <I>will</I> <I>appear</I>
<I>in</I> <I>the</I> <I>patterns</I>. The group numbers are actually disregarded
by <I>flex</I>; %t serves, though, to lump characters together.
Given the above table, for example, the pattern "a(AA)*5" is
equivalent to "d(ZQ)*0". They both say, "match any charac-
ter in group #1, followed by zero-or-more pairs of charac-
ters from group #2, followed by a character from group #52."
Thus %t provides a crude way for introducing equivalence
classes into the scanner specification.
Note that the -i option (see below) coupled with the
equivalence classes which <I>flex</I> automatically generates take
care of virtually all the instances when one might consider
using %t. But what the hell, it's there if you want it.
</PRE>
<H2>OPTIONS</H2><PRE>
<I>flex</I> has the following options:
-b Generate backtracking information to <I>lex</I>.<I>backtrack</I>.
This is a list of scanner states which require back-
tracking and the input characters on which they do so.
By adding rules one can remove backtracking states. If
all backtracking states are eliminated and -f or -F is
used, the generated scanner will run faster (see the -p
flag). Only users who wish to squeeze every last cycle
out of their scanners need worry about this option.
(See the section on PERFORMANCE CONSIDERATIONS below.)
-c is a do-nothing, deprecated option included for POSIX
compliance.
NOTE: in previous releases of <I>flex</I> -c specified table-
compression options. This functionality is now given
by the -C flag. To ease the the impact of this change,
when <I>flex</I> encounters -c, it currently issues a warning
message and assumes that -C was desired instead. In
the future this "promotion" of -c to -C will go away in
the name of full POSIX compliance (unless the POSIX
meaning is removed first).
-d makes the generated scanner run in <I>debug</I> mode. When-
ever a pattern is recognized and the global
yy_flex_debug is non-zero (which is the default), the
scanner will write to <I>stderr</I> a line of the form:
--accepting rule at line 53 ("the matched text")
The line number refers to the location of the rule in
the file defining the scanner (i.e., the file that was
fed to flex). Messages are also generated when the
scanner backtracks, accepts the default rule, reaches
the end of its input buffer (or encounters a NUL; at
this point, the two look the same as far as the
scanner's concerned), or reaches an end-of-file.
-f specifies (take your pick) <I>full</I> <I>table</I> or <I>fast</I> <I>scanner</I>.
No table compression is done. The result is large but
fast. This option is equivalent to -Cf (see below).
-i instructs <I>flex</I> to generate a <I>case</I>-<I>insensitive</I> scanner.
The case of letters given in the <I>flex</I> input patterns
will be ignored, and tokens in the input will be
matched regardless of case. The matched text given in
<I>yytext</I> will have the preserved case (i.e., it will not
be folded).
-n is another do-nothing, deprecated option included only
for POSIX compliance.
-p generates a performance report to stderr. The report
consists of comments regarding features of the <I>flex</I>
input file which will cause a loss of performance in
the resulting scanner. Note that the use of <I>REJECT</I> and
variable trailing context (see the BUGS section in
<B>flex(1)</B>) entails a substantial performance penalty; use
of <I>yymore</I>(), the ^ operator, and the -I flag entail
minor performance penalties.
-s causes the <I>default</I> <I>rule</I> (that unmatched scanner input
is echoed to <I>stdout</I>) to be suppressed. If the scanner
encounters input that does not match any of its rules,
it aborts with an error. This option is useful for
finding holes in a scanner's rule set.
-t instructs <I>flex</I> to write the scanner it generates to
standard output instead of lex.yy.c.
-v specifies that <I>flex</I> should write to <I>stderr</I> a summary of
statistics regarding the scanner it generates. Most of
the statistics are meaningless to the casual <I>flex</I> user,
but the first line identifies the version of <I>flex</I>,
which is useful for figuring out where you stand with
respect to patches and new releases, and the next two
lines give the date when the scanner was created and a
summary of the flags which were in effect.
-F specifies that the <I>fast</I> scanner table representation
should be used. This representation is about as fast
as the full table representation (-<I>f</I>), and for some
sets of patterns will be considerably smaller (and for
others, larger). In general, if the pattern set con-
tains both "keywords" and a catch-all, "identifier"
rule, such as in the set:
"case" return TOK_CASE;
"switch" return TOK_SWITCH;
...
"default" return TOK_DEFAULT;
[a-z]+ return TOK_ID;
then you're better off using the full table representa-
tion. If only the "identifier" rule is present and you
then use a hash table or some such to detect the key-
words, you're better off using -<I>F</I>.
This option is equivalent to -CF (see below).
-I instructs <I>flex</I> to generate an <I>interactive</I> scanner.
Normally, scanners generated by <I>flex</I> always look ahead
one character before deciding that a rule has been
matched. At the cost of some scanning overhead, <I>flex</I>
will generate a scanner which only looks ahead when
needed. Such scanners are called <I>interactive</I> because
if you want to write a scanner for an interactive sys-
tem such as a command shell, you will probably want the
user's input to be terminated with a newline, and
without -I the user will have to type a character in
addition to the newline in order to have the newline
recognized. This leads to dreadful interactive perfor-
mance.
If all this seems to confusing, here's the general
rule: if a human will be typing in input to your
scanner, use -I, otherwise don't; if you don't care
about squeezing the utmost performance from your
scanner and you don't want to make any assumptions
about the input to your scanner, use -I.
Note, -I cannot be used in conjunction with <I>full</I> or
<I>fast</I> <I>tables</I>, i.e., the -f, -F, -Cf, or -CF flags.
-L instructs <I>flex</I> not to generate #line directives.
Without this option, <I>flex</I> peppers the generated scanner
with #line directives so error messages in the actions
will be correctly located with respect to the original
<I>flex</I> input file, and not to the fairly meaningless line
numbers of lex.yy.c. (Unfortunately <I>flex</I> does not
presently generate the necessary directives to "retar-
get" the line numbers for those parts of lex.yy.c which
it generated. So if there is an error in the generated
code, a meaningless line number is reported.)
-T makes <I>flex</I> run in <I>trace</I> mode. It will generate a lot
of messages to <I>stdout</I> concerning the form of the input
and the resultant non-deterministic and deterministic
finite automata. This option is mostly for use in
maintaining <I>flex</I>.
-8 instructs <I>flex</I> to generate an 8-bit scanner, i.e., one
which can recognize 8-bit characters. On some sites,
<I>flex</I> is installed with this option as the default. On
others, the default is 7-bit characters. To see which
is the case, check the verbose (-v) output for
"equivalence classes created". If the denominator of
the number shown is 128, then by default <I>flex</I> is gen-
erating 7-bit characters. If it is 256, then the
default is 8-bit characters and the -8 flag is not
required (but may be a good idea to keep the scanner
specification portable). Feeding a 7-bit scanner 8-bit
characters will result in infinite loops, bus errors,
or other such fireworks, so when in doubt, use the
flag. Note that if equivalence classes are used, 8-bit
scanners take only slightly more table space than 7-bit
scanners (128 bytes, to be exact); if equivalence
classes are not used, however, then the tables may grow
up to twice their 7-bit size.
-C[efmF]
controls the degree of table compression.
-Ce directs <I>flex</I> to construct <I>equivalence</I> <I>classes</I>,
i.e., sets of characters which have identical lexical
properties (for example, if the only appearance of
digits in the <I>flex</I> input is in the character class
"[0-9]" then the digits '0', '1', ..., '9' will all be
put in the same equivalence class). Equivalence
classes usually give dramatic reductions in the final
table/object file sizes (typically a factor of 2-5) and
are pretty cheap performance-wise (one array look-up
per character scanned).
-Cf specifies that the <I>full</I> scanner tables should be
generated - <I>flex</I> should not compress the tables by tak-
ing advantages of similar transition functions for dif-
ferent states.
-CF specifies that the alternate fast scanner represen-
tation (described above under the -F flag) should be
used.
-Cm directs <I>flex</I> to construct <I>meta</I>-<I>equivalence</I> <I>classes</I>,
which are sets of equivalence classes (or characters,
if equivalence classes are not being used) that are
commonly used together. Meta-equivalence classes are
often a big win when using compressed tables, but they
have a moderate performance impact (one or two "if"
tests and one array look-up per character scanned).
A lone -C specifies that the scanner tables should be
compressed but neither equivalence classes nor meta-
equivalence classes should be used.
The options -Cf or -CF and -Cm do not make sense
together - there is no opportunity for meta-equivalence
classes if the table is not being compressed. Other-
wise the options may be freely mixed.
The default setting is -Cem, which specifies that <I>flex</I>
should generate equivalence classes and meta-
equivalence classes. This setting provides the highest
degree of table compression. You can trade off
faster-executing scanners at the cost of larger tables
with the following generally being true:
slowest &amp; smallest
-Cem
-Cm
-Ce
-C
-C{f,F}e
-C{f,F}
fastest &amp; largest
Note that scanners with the smallest tables are usually
generated and compiled the quickest, so during develop-
ment you will usually want to use the default, maximal
compression.
-Cfe is often a good compromise between speed and size
for production scanners.
-C options are not cumulative; whenever the flag is
encountered, the previous -C settings are forgotten.
-Sskeleton_file
overrides the default skeleton file from which <I>flex</I>
constructs its scanners. You'll never need this option
unless you are doing <I>flex</I> maintenance or development.
</PRE>
<H2>PERFORMANCE CONSIDERATIONS</H2><PRE>
The main design goal of <I>flex</I> is that it generate high-
performance scanners. It has been optimized for dealing
well with large sets of rules. Aside from the effects of
table compression on scanner speed outlined above, there are
a number of options/actions which degrade performance.
These are, from most expensive to least:
REJECT
pattern sets that require backtracking
arbitrary trailing context
'^' beginning-of-line operator
yymore()
with the first three all being quite expensive and the last
two being quite cheap.
REJECT should be avoided at all costs when performance is
important. It is a particularly expensive option.
Getting rid of backtracking is messy and often may be an
enormous amount of work for a complicated scanner. In prin-
cipal, one begins by using the -b flag to generate a
<I>lex</I>.<I>backtrack</I> file. For example, on the input
%%
foo return TOK_KEYWORD;
foobar return TOK_KEYWORD;
the file looks like:
State #6 is non-accepting -
associated rule line numbers:
2 3
out-transitions: [ o ]
jam-transitions: EOF [ \001-n p-\177 ]
State #8 is non-accepting -
associated rule line numbers:
3
out-transitions: [ a ]
jam-transitions: EOF [ \001-` b-\177 ]
State #9 is non-accepting -
associated rule line numbers:
3
out-transitions: [ r ]
jam-transitions: EOF [ \001-q s-\177 ]
Compressed tables always backtrack.
The first few lines tell us that there's a scanner state in
which it can make a transition on an 'o' but not on any
other character, and that in that state the currently
scanned text does not match any rule. The state occurs when
trying to match the rules found at lines 2 and 3 in the
input file. If the scanner is in that state and then reads
something other than an 'o', it will have to backtrack to
find a rule which is matched. With a bit of headscratching
one can see that this must be the state it's in when it has
seen "fo". When this has happened, if anything other than
another 'o' is seen, the scanner will have to back up to
simply match the 'f' (by the default rule).
The comment regarding State #8 indicates there's a problem
when "foob" has been scanned. Indeed, on any character
other than a 'b', the scanner will have to back up to accept
"foo". Similarly, the comment for State #9 concerns when
"fooba" has been scanned.
The final comment reminds us that there's no point going to
all the trouble of removing backtracking from the rules
unless we're using -f or -F, since there's no performance
gain doing so with compressed scanners.
The way to remove the backtracking is to add "error" rules:
%%
foo return TOK_KEYWORD;
foobar return TOK_KEYWORD;
fooba |
foob |
fo {
/* false alarm, not really a keyword */
return TOK_ID;
}
Eliminating backtracking among a list of keywords can also
be done using a "catch-all" rule:
%%
foo return TOK_KEYWORD;
foobar return TOK_KEYWORD;
[a-z]+ return TOK_ID;
This is usually the best solution when appropriate.
Backtracking messages tend to cascade. With a complicated
set of rules it's not uncommon to get hundreds of messages.
If one can decipher them, though, it often only takes a
dozen or so rules to eliminate the backtracking (though it's
easy to make a mistake and have an error rule accidentally
match a valid token. A possible future <I>flex</I> feature will be
to automatically add rules to eliminate backtracking).
<I>Variable</I> trailing context (where both the leading and trail-
ing parts do not have a fixed length) entails almost the
same performance loss as <I>REJECT</I> (i.e., substantial). So
when possible a rule like:
%%
mouse|rat/(cat|dog) run();
is better written:
%%
mouse/cat|dog run();
rat/cat|dog run();
or as
%%
mouse|rat/cat run();
mouse|rat/dog run();
Note that here the special '|' action does <I>not</I> provide any
savings, and can even make things worse (see BUGS in
<B>flex(1)</B>).
Another area where the user can increase a scanner's perfor-
mance (and one that's easier to implement) arises from the
fact that the longer the tokens matched, the faster the
scanner will run. This is because with long tokens the pro-
cessing of most input characters takes place in the (short)
inner scanning loop, and does not often have to go through
the additional work of setting up the scanning environment
(e.g., yytext) for the action. Recall the scanner for C
comments:
%x comment
%%
int line_num = 1;
"/*" BEGIN(comment);
&lt;comment&gt;[^*\n]*
&lt;comment&gt;"*"+[^*/\n]*
&lt;comment&gt;\n ++line_num;
&lt;comment&gt;"*"+"/" BEGIN(INITIAL);
This could be sped up by writing it as:
%x comment
%%
int line_num = 1;
"/*" BEGIN(comment);
&lt;comment&gt;[^*\n]*
&lt;comment&gt;[^*\n]*\n ++line_num;
&lt;comment&gt;"*"+[^*/\n]*
&lt;comment&gt;"*"+[^*/\n]*\n ++line_num;
&lt;comment&gt;"*"+"/" BEGIN(INITIAL);
Now instead of each newline requiring the processing of
another action, recognizing the newlines is "distributed"
over the other rules to keep the matched text as long as
possible. Note that <I>adding</I> rules does <I>not</I> slow down the
scanner! The speed of the scanner is independent of the
number of rules or (modulo the considerations given at the
beginning of this section) how complicated the rules are
with regard to operators such as '*' and '|'.
A final example in speeding up a scanner: suppose you want
to scan through a file containing identifiers and keywords,
one per line and with no other extraneous characters, and
recognize all the keywords. A natural first approach is:
%%
asm |
auto |
break |
... etc ...
volatile |
while /* it's a keyword */
.|\n /* it's not a keyword */
To eliminate the back-tracking, introduce a catch-all rule:
%%
asm |
auto |
break |
... etc ...
volatile |
while /* it's a keyword */
[a-z]+ |
.|\n /* it's not a keyword */
Now, if it's guaranteed that there's exactly one word per
line, then we can reduce the total number of matches by a
half by merging in the recognition of newlines with that of
the other tokens:
%%
asm\n |
auto\n |
break\n |
... etc ...
volatile\n |
while\n /* it's a keyword */
[a-z]+\n |
.|\n /* it's not a keyword */
One has to be careful here, as we have now reintroduced
backtracking into the scanner. In particular, while <I>we</I> know
that there will never be any characters in the input stream
other than letters or newlines, <I>flex</I> can't figure this out,
and it will plan for possibly needing backtracking when it
has scanned a token like "auto" and then the next character
is something other than a newline or a letter. Previously
it would then just match the "auto" rule and be done, but
now it has no "auto" rule, only a "auto\n" rule. To elim-
inate the possibility of backtracking, we could either
duplicate all rules but without final newlines, or, since we
never expect to encounter such an input and therefore don't
how it's classified, we can introduce one more catch-all
rule, this one which doesn't include a newline:
%%
asm\n |
auto\n |
break\n |
... etc ...
volatile\n |
while\n /* it's a keyword */
[a-z]+\n |
[a-z]+ |
.|\n /* it's not a keyword */
Compiled with -Cf, this is about as fast as one can get a
<I>flex</I> scanner to go for this particular problem.
A final note: <I>flex</I> is slow when matching NUL's, particu-
larly when a token contains multiple NUL's. It's best to
write rules which match <I>short</I> amounts of text if it's anti-
cipated that the text will often include NUL's.
</PRE>
<H2>INCOMPATIBILITIES WITH LEX AND POSIX</H2><PRE>
<I>flex</I> is a rewrite of the Unix <I>lex</I> tool (the two implementa-
tions do not share any code, though), with some extensions
and incompatibilities, both of which are of concern to those
who wish to write scanners acceptable to either implementa-
tion. At present, the POSIX <I>lex</I> draft is very close to the
original <I>lex</I> implementation, so some of these incompatibili-
ties are also in conflict with the POSIX draft. But the
intent is that except as noted below, <I>flex</I> as it presently
stands will ultimately be POSIX conformant (i.e., that those
areas of conflict with the POSIX draft will be resolved in
<I>flex</I>'<I>s</I> favor). Please bear in mind that all the comments
which follow are with regard to the POSIX <I>draft</I> standard of
Summer 1989, and not the final document (or subsequent
drafts); they are included so <I>flex</I> users can be aware of the
standardization issues and those areas where <I>flex</I> may in the
near future undergo changes incompatible with its current
definition.
<I>flex</I> is fully compatible with <I>lex</I> with the following excep-
tions:
- The undocumented <I>lex</I> scanner internal variable yylineno
is not supported. It is difficult to support this
option efficiently, since it requires examining every
character scanned and reexamining the characters when
the scanner backs up. Things get more complicated when
the end of buffer or file is reached or a NUL is
scanned (since the scan must then be restarted with the
proper line number count), or the user uses the
yyless(), unput(), or REJECT actions, or the multiple
input buffer functions.
The fix is to add rules which, upon seeing a newline,
increment yylineno. This is usually an easy process,
though it can be a drag if some of the patterns can
match multiple newlines along with other characters.
yylineno is not part of the POSIX draft.
- The input() routine is not redefinable, though it may
be called to read characters following whatever has
been matched by a rule. If input() encounters an end-
of-file the normal yywrap() processing is done. A
``real'' end-of-file is returned by input() as <I>EOF</I>.
Input is instead controlled by redefining the YY_INPUT
macro.
The <I>flex</I> restriction that input() cannot be redefined
is in accordance with the POSIX draft, but YY_INPUT has
not yet been accepted into the draft (and probably
won't; it looks like the draft will simply not specify
any way of controlling the scanner's input other than
by making an initial assignment to <I>yyin</I>).
- <I>flex</I> scanners do not use stdio for input. Because of
this, when writing an interactive scanner one must
explicitly call fflush() on the stream associated with
the terminal after writing out a prompt. With <I>lex</I> such
writes are automatically flushed since <I>lex</I> scanners use
getchar() for their input. Also, when writing interac-
tive scanners with <I>flex</I>, the -I flag must be used.
- <I>flex</I> scanners are not as reentrant as <I>lex</I> scanners. In
particular, if you have an interactive scanner and an
interrupt handler which long-jumps out of the scanner,
and the scanner is subsequently called again, you may
get the following message:
fatal flex scanner internal error--end of buffer missed
To reenter the scanner, first use
yyrestart( yyin );
- output() is not supported. Output from the ECHO macro
is done to the file-pointer <I>yyout</I> (default <I>stdout</I>).
The POSIX draft mentions that an output() routine
exists but currently gives no details as to what it
does.
- <I>lex</I> does not support exclusive start conditions (%x),
though they are in the current POSIX draft.
- When definitions are expanded, <I>flex</I> encloses them in
parentheses. With lex, the following:
NAME [A-Z][A-Z0-9]*
%%
foo{NAME}? printf( "Found it\n" );
%%
will not match the string "foo" because when the macro
is expanded the rule is equivalent to "foo[A-Z][A-Z0-
9]*?" and the precedence is such that the '?' is asso-
ciated with "[A-Z0-9]*". With <I>flex</I>, the rule will be
expanded to "foo([A-Z][A-Z0-9]*)?" and so the string
"foo" will match. Note that because of this, the ^, $,
&lt;s&gt;, /, and &lt;&lt;EOF&gt;&gt; operators cannot be used in a <I>flex</I>
definition.
The POSIX draft interpretation is the same as <I>flex</I>'<I>s</I>.
- To specify a character class which matches anything but
a left bracket (']'), in <I>lex</I> one can use "[^]]" but
with <I>flex</I> one must use "[^\]]". The latter works with
<I>lex</I>, too.
- The <I>lex</I> %r (generate a Ratfor scanner) option is not
supported. It is not part of the POSIX draft.
- If you are providing your own yywrap() routine, you
must include a "#undef yywrap" in the definitions sec-
tion (section 1). Note that the "#undef" will have to
be enclosed in %{}'s.
The POSIX draft specifies that yywrap() is a function
and this is very unlikely to change; so <I>flex</I> <I>users</I> <I>are</I>
<I>warned</I> that yywrap() is likely to be changed to a func-
tion in the near future.
- After a call to unput(), <I>yytext</I> and <I>yyleng</I> are unde-
fined until the next token is matched. This is not the
case with <I>lex</I> or the present POSIX draft.
- The precedence of the {} (numeric range) operator is
different. <I>lex</I> interprets "abc{1,3}" as "match one,
two, or three occurrences of 'abc'", whereas <I>flex</I>
interprets it as "match 'ab' followed by one, two, or
three occurrences of 'c'". The latter is in agreement
with the current POSIX draft.
- The precedence of the ^ operator is different. <I>lex</I>
interprets "^foo|bar" as "match either 'foo' at the
beginning of a line, or 'bar' anywhere", whereas <I>flex</I>
interprets it as "match either 'foo' or 'bar' if they
come at the beginning of a line". The latter is in
agreement with the current POSIX draft.
- To refer to yytext outside of the scanner source file,
the correct definition with <I>flex</I> is "extern char
*yytext" rather than "extern char yytext[]". This is
contrary to the current POSIX draft but a point on
which <I>flex</I> will not be changing, as the array represen-
tation entails a serious performance penalty. It is
hoped that the POSIX draft will be emended to support
the <I>flex</I> variety of declaration (as this is a fairly
painless change to require of <I>lex</I> users).
- <I>yyin</I> is <I>initialized</I> by <I>lex</I> to be <I>stdin</I>; <I>flex</I>, on the
other hand, initializes <I>yyin</I> to NULL and then <I>assigns</I>
it to <I>stdin</I> the first time the scanner is called, pro-
viding <I>yyin</I> has not already been assigned to a non-NULL
value. The difference is subtle, but the net effect is
that with <I>flex</I> scanners, <I>yyin</I> does not have a valid
value until the scanner has been called.
- The special table-size declarations such as %a sup-
ported by <I>lex</I> are not required by <I>flex</I> scanners; <I>flex</I>
ignores them.
- The name FLEX_SCANNER is #define'd so scanners may be
written for use with either <I>flex</I> or <I>lex</I>.
The following <I>flex</I> features are not included in <I>lex</I> or the
POSIX draft standard:
yyterminate()
&lt;&lt;EOF&gt;&gt;
YY_DECL
#line directives
%{}'s around actions
yyrestart()
comments beginning with '#' (deprecated)
multiple actions on a line
This last feature refers to the fact that with <I>flex</I> you can
put multiple actions on the same line, separated with semi-
colons, while with <I>lex</I>, the following
foo handle_foo(); ++num_foos_seen;
is (rather surprisingly) truncated to
foo handle_foo();
<I>flex</I> does not truncate the action. Actions that are not
enclosed in braces are simply terminated at the end of the
line.
</PRE>
<H2>DIAGNOSTICS</H2><PRE>
<I>reject</I>_<I>used</I>_<I>but</I>_<I>not</I>_<I>detected</I> <I>undefined</I> or
<I>yymore</I>_<I>used</I>_<I>but</I>_<I>not</I>_<I>detected</I> <I>undefined</I> - These errors can
occur at compile time. They indicate that the scanner uses
REJECT or yymore() but that <I>flex</I> failed to notice the fact,
meaning that <I>flex</I> scanned the first two sections looking for
occurrences of these actions and failed to find any, but
somehow you snuck some in (via a #include file, for exam-
ple). Make an explicit reference to the action in your <I>flex</I>
input file. (Note that previously <I>flex</I> supported a
%used/%unused mechanism for dealing with this problem; this
feature is still supported but now deprecated, and will go
away soon unless the author hears from people who can argue
compellingly that they need it.)
<I>flex</I> <I>scanner</I> <I>jammed</I> - a scanner compiled with -s has encoun-
tered an input string which wasn't matched by any of its
rules.
<I>flex</I> <I>input</I> <I>buffer</I> <I>overflowed</I> - a scanner rule matched a
string long enough to overflow the scanner's internal input
buffer (16K bytes by default - controlled by YY_BUF_SIZE in
"flex.skel". Note that to redefine this macro, you must
first #undefine it).
<I>scanner</I> <I>requires</I> -<I>8</I> <I>flag</I> - Your scanner specification
includes recognizing 8-bit characters and you did not
specify the -8 flag (and your site has not installed flex
with -8 as the default).
<I>fatal</I> <I>flex</I> <I>scanner</I> <I>internal</I> <I>error</I>--<I>end</I> <I>of</I> <I>buffer</I> <I>missed</I> -
This can occur in an scanner which is reentered after a
long-jump has jumped out (or over) the scanner's activation
frame. Before reentering the scanner, use:
yyrestart( yyin );
<I>too</I> <I>many</I> %<I>t</I> <I>classes</I>! - You managed to put every single char-
acter into its own %t class. <I>flex</I> requires that at least
one of the classes share characters.
</PRE>
<H2>DEFICIENCIES / BUGS</H2><PRE>
See <B>flex(1)</B>.
</PRE>
<H2>SEE ALSO</H2><PRE>
<B>flex(1)</B>, <B>lex(1)</B>, <B>yacc(1)</B>, <B>sed(1)</B>, <B>awk(1)</B>.
M. E. Lesk and E. Schmidt, <I>LEX</I> - <I>Lexical</I> <I>Analyzer</I> <I>Generator</I>
</PRE>
<H2>AUTHOR</H2><PRE>
Vern Paxson, with the help of many ideas and much inspira-
tion from Van Jacobson. Original version by Jef Poskanzer.
The fast table representation is a partial implementation of
a design done by Van Jacobson. The implementation was done
by Kevin Gong and Vern Paxson.
Thanks to the many <I>flex</I> beta-testers, feedbackers, and con-
tributors, especially Casey Leedom, benson@odi.com, Keith
Bostic, Frederic Brehm, Nick Christopher, Jason Coughlin,
Scott David Daniels, Leo Eskin, Chris Faylor, Eric Goldman,
Eric Hughes, Jeffrey R. Jones, Kevin B. Kenny, Ronald Lam-
precht, Greg Lee, Craig Leres, Mohamed el Lozy, Jim Meyer-
ing, Marc Nozell, Esmond Pitt, Jef Poskanzer, Jim Roskind,
Dave Tallman, Frank Whaley, Ken Yap, and those whose names
have slipped my marginal mail-archiving skills but whose
contributions are appreciated all the same.
Thanks to Keith Bostic, John Gilmore, Craig Leres, Bob Mul-
cahy, Rich Salz, and Richard Stallman for help with various
distribution headaches.
Thanks to Esmond Pitt and Earle Horton for 8-bit character
support; to Benson Margulies and Fred Burke for C++ support;
to Ove Ewerlid for the basics of support for NUL's; and to
Eric Hughes for the basics of support for multiple buffers.
Work is being done on extending <I>flex</I> to generate scanners in
which the state machine is directly represented in C code
rather than tables. These scanners may well be substan-
tially faster than those generated using -f or -F. If you
are working in this area and are interested in comparing
notes and seeing whether redundant work can be avoided, con-
tact Ove Ewerlid (ewerlid@mizar.DoCS.UU.SE).
This work was primarily done when I was at the Real Time
Systems Group at the Lawrence Berkeley Laboratory in Berke-
ley, CA. Many thanks to all there for the support I
received.
Send comments to:
Vern Paxson
Computer Science Department
4126 Upson Hall
Cornell University
Ithaca, NY 14853-7501
vern@cs.cornell.edu
decvax!cornell!vern
</PRE>
<HR>
<ADDRESS>
Man(1) output converted with
<a href="http://www.oac.uci.edu/indiv/ehood/man2html.html">man2html</a>
</ADDRESS>
</BODY>
</HTML>
Reference in New Issue View Git Blame Copy Permalink