John Winans
1793 lines
74 KiB
Plaintext
1793 lines
74 KiB
Plaintext
26 May 1990 FLEX(1)
|
|
|
|
NAME
|
|
flex - fast lexical analyzer generator
|
|
|
|
SYNOPSIS
|
|
flex [-bcdfinpstvFILT8 -C[efmF] -Sskeleton] [filename ...]
|
|
|
|
DESCRIPTION
|
|
flex is a tool for generating scanners: programs which recognized lexi-
|
|
cal patterns in text. flex reads the given input files, or its standard
|
|
input if no file names are given, for a description of a scanner to gen-
|
|
erate. The description is in the form of pairs of regular expressions
|
|
and C code, called rules. flex generates as output a C source file,
|
|
lex.yy.c, which defines a routine yylex(). This file is compiled and
|
|
linked with the -lfl library to produce an executable. When the execut-
|
|
able is run, it analyzes its input for occurrences of the regular
|
|
expressions. Whenever it finds one, it executes the corresponding C
|
|
code.
|
|
|
|
SOME SIMPLE EXAMPLES
|
|
|
|
First some simple examples to get the flavor of how one uses flex. The
|
|
following flex input specifies a scanner which whenever it encounters
|
|
the string "username" will replace it with the user's login name:
|
|
|
|
%%
|
|
username printf( "%s", getlogin() );
|
|
|
|
By default, any text not matched by a flex scanner is copied to the out-
|
|
put, so the net effect of this scanner is to copy its input file to its
|
|
output with each occurrence of "username" expanded. In this input,
|
|
there is just one rule. "username" is the pattern and the "printf" is
|
|
the action. The "%%" marks the beginning of the rules.
|
|
|
|
Here's another simple example:
|
|
|
|
int num_lines = 0, num_chars = 0;
|
|
|
|
%%
|
|
\n ++num_lines; ++num_chars;
|
|
. ++num_chars;
|
|
|
|
%%
|
|
main()
|
|
{
|
|
yylex();
|
|
printf( "# of lines = %d, # of chars = %d\n",
|
|
num_lines, num_chars );
|
|
}
|
|
|
|
This scanner counts the number of characters and the number of lines in
|
|
its input (it produces no output other than the final report on the
|
|
counts). The first line declares two globals, "num_lines" and
|
|
"num_chars", which are accessible both inside yylex() and in the main()
|
|
|
|
Version 2.3 1
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
routine declared after the second "%%". There are two rules, one which
|
|
matches a newline ("\n") and increments both the line count and the
|
|
character count, and one which matches any character other than a new-
|
|
line (indicated by the "." regular expression).
|
|
|
|
A somewhat more complicated example:
|
|
|
|
/* scanner for a toy Pascal-like language */
|
|
|
|
%{
|
|
/* need this for the call to atof() below */
|
|
#include <math.h>
|
|
%}
|
|
|
|
DIGIT [0-9]
|
|
ID [a-z][a-z0-9]*
|
|
|
|
%%
|
|
|
|
{DIGIT}+ {
|
|
printf( "An integer: %s (%d)\n", yytext,
|
|
atoi( yytext ) );
|
|
}
|
|
|
|
{DIGIT}+"."{DIGIT}* {
|
|
printf( "A float: %s (%g)\n", yytext,
|
|
atof( yytext ) );
|
|
}
|
|
|
|
if|then|begin|end|procedure|function {
|
|
printf( "A keyword: %s\n", yytext );
|
|
}
|
|
|
|
{ID} printf( "An identifier: %s\n", yytext );
|
|
|
|
"+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext );
|
|
|
|
"{"[^}\n]*"}" /* eat up one-line comments */
|
|
|
|
[ \t\n]+ /* eat up whitespace */
|
|
|
|
. printf( "Unrecognized character: %s\n", yytext );
|
|
|
|
%%
|
|
|
|
main( argc, argv )
|
|
int argc;
|
|
char **argv;
|
|
{
|
|
++argv, --argc; /* skip over program name */
|
|
if ( argc > 0 )
|
|
yyin = fopen( argv[0], "r" );
|
|
else
|
|
yyin = stdin;
|
|
|
|
2 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
yylex();
|
|
}
|
|
|
|
This is the beginnings of a simple scanner for a language like Pascal.
|
|
It identifies different types of tokens and reports on what it has seen.
|
|
|
|
The details of this example will be explained in the following sections.
|
|
|
|
FORMAT OF THE INPUT FILE
|
|
The flex input file consists of three sections, separated by a line with
|
|
just %% in it:
|
|
|
|
definitions
|
|
%%
|
|
rules
|
|
%%
|
|
user code
|
|
|
|
The definitions section contains declarations of simple name definitions
|
|
to simplify the scanner specification, and declarations of start condi-
|
|
tions, which are explained in a later section.
|
|
|
|
Name definitions have the form:
|
|
|
|
name definition
|
|
|
|
The "name" is a word beginning with a letter or an underscore ('_') fol-
|
|
lowed by zero or more letters, digits, '_', or '-' (dash). The defini-
|
|
tion is taken to begin at the first non-white-space character following
|
|
the name and continuing to the end of the line. The definition can sub-
|
|
sequently be referred to using "{name}", which will expand to "(defini-
|
|
tion)". For example,
|
|
|
|
DIGIT [0-9]
|
|
ID [a-z][a-z0-9]*
|
|
|
|
defines "DIGIT" to be a regular expression which matches a single digit,
|
|
and "ID" to be a regular expression which matches a letter followed by
|
|
zero-or-more letters-or-digits. A subsequent reference to
|
|
|
|
{DIGIT}+"."{DIGIT}*
|
|
|
|
is identical to
|
|
|
|
([0-9])+"."([0-9])*
|
|
|
|
and matches one-or-more digits followed by a '.' followed by zero-or-
|
|
more digits.
|
|
|
|
The rules section of the flex input contains a series of rules of the
|
|
form:
|
|
|
|
pattern action
|
|
|
|
Version 2.3 3
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
where the pattern must be unindented and the action must begin on the
|
|
same line.
|
|
|
|
See below for a further description of patterns and actions.
|
|
|
|
Finally, the user code section is simply copied to lex.yy.c verbatim.
|
|
It is used for companion routines which call or are called by the
|
|
scanner. The presence of this section is optional; if it is missing,
|
|
the second %% in the input file may be skipped, too.
|
|
|
|
In the definitions and rules sections, any indented text or text
|
|
enclosed in %{ and %} is copied verbatim to the output (with the %{}'s
|
|
removed). The %{}'s must appear unindented on lines by themselves.
|
|
|
|
In the rules section, any indented or %{} text appearing before the
|
|
first rule may be used to declare variables which are local to the scan-
|
|
ning routine and (after the declarations) code which is to be executed
|
|
whenever the scanning routine is entered. Other indented or %{} text in
|
|
the rule section is still copied to the output, but its meaning is not
|
|
well-defined and it may well cause compile-time errors (this feature is
|
|
present for POSIX compliance; see below for other such features).
|
|
|
|
In the definitions section, an unindented comment (i.e., a line begin-
|
|
ning with "/*") is also copied verbatim to the output up to the next
|
|
"*/". Also, any line in the definitions section beginning with '#' is
|
|
ignored, though this style of comment is deprecated and may go away in
|
|
the future.
|
|
|
|
PATTERNS
|
|
The patterns in the input are written using an extended set of regular
|
|
expressions. These are:
|
|
|
|
x match the character 'x'
|
|
. any character except newline
|
|
[xyz] a "character class"; in this case, the pattern
|
|
matches either an 'x', a 'y', or a 'z'
|
|
[abj-oZ] a "character class" with a range in it; matches
|
|
an 'a', a 'b', any letter from 'j' through 'o',
|
|
or a 'Z'
|
|
[^A-Z] a "negated character class", i.e., any character
|
|
but those in the class. In this case, any
|
|
character EXCEPT an uppercase letter.
|
|
[^A-Z\n] any character EXCEPT an uppercase letter or
|
|
a newline
|
|
r* zero or more r's, where r is any regular expression
|
|
r+ one or more r's
|
|
r? zero or one r's (that is, "an optional r")
|
|
r{2,5} anywhere from two to five r's
|
|
r{2,} two or more r's
|
|
r{4} exactly 4 r's
|
|
{name} the expansion of the "name" definition
|
|
(see above)
|
|
"[xyz]\"foo"
|
|
|
|
4 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
the literal string: [xyz]"foo
|
|
\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
|
|
then the ANSI-C interpretation of \x.
|
|
Otherwise, a literal 'X' (used to escape
|
|
operators such as '*')
|
|
\123 the character with octal value 123
|
|
\x2a the character with hexadecimal value 2a
|
|
(r) match an r; parentheses are used to override
|
|
precedence (see below)
|
|
|
|
rs the regular expression r followed by the
|
|
regular expression s; called "concatenation"
|
|
|
|
r|s either an r or an s
|
|
|
|
r/s an r but only if it is followed by an s. The
|
|
s is not part of the matched text. This type
|
|
of pattern is called as "trailing context".
|
|
^r an r, but only at the beginning of a line
|
|
r$ an r, but only at the end of a line. Equivalent
|
|
to "r/\n".
|
|
|
|
<s>r an r, but only in start condition s (see
|
|
below for discussion of start conditions)
|
|
<s1,s2,s3>r
|
|
same, but in any of start conditions s1,
|
|
s2, or s3
|
|
|
|
<<EOF>> an end-of-file
|
|
<s1,s2><<EOF>>
|
|
an end-of-file when in start condition s1 or s2
|
|
|
|
The regular expressions listed above are grouped according to pre-
|
|
cedence, from highest precedence at the top to lowest at the bottom.
|
|
Those grouped together have equal precedence. For example,
|
|
|
|
foo|bar*
|
|
|
|
is the same as
|
|
|
|
(foo)|(ba(r*))
|
|
|
|
since the '*' operator has higher precedence than concatenation, and
|
|
concatenation higher than alternation ('|'). This pattern therefore
|
|
matches either the string "foo" or the string "ba" followed by zero-or-
|
|
more r's. To match "foo" or zero-or-more "bar"'s, use:
|
|
|
|
foo|(bar)*
|
|
|
|
Version 2.3 5
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
and to match zero-or-more "foo"'s-or-"bar"'s:
|
|
|
|
(foo|bar)*
|
|
|
|
Some notes on patterns:
|
|
|
|
- A negated character class such as the example "[^A-Z]" above will
|
|
match a newline unless "\n" (or an equivalent escape sequence) is
|
|
one of the characters explicitly present in the negated character
|
|
class (e.g., "[^A-Z\n]"). This is unlike how many other regular
|
|
expression tools treat negated character classes, but unfortunately
|
|
the inconsistency is historically entrenched. Matching newlines
|
|
means that a pattern like [^"]* can match an entire input (over-
|
|
flowing the scanner's input buffer) unless there's another quote in
|
|
the input.
|
|
|
|
- A rule can have at most one instance of trailing context (the '/'
|
|
operator or the '$' operator). The start condition, '^', and
|
|
"<<EOF>>" patterns can only occur at the beginning of a pattern,
|
|
and, as well as with '/' and '$', cannot be grouped inside
|
|
parentheses. A '^' which does not occur at the beginning of a rule
|
|
or a '$' which does not occur at the end of a rule loses its spe-
|
|
cial properties and is treated as a normal character.
|
|
|
|
The following are illegal:
|
|
|
|
foo/bar$
|
|
<sc1>foo<sc2>bar
|
|
|
|
Note that the first of these, can be written "foo/bar\n".
|
|
|
|
The following will result in '$' or '^' being treated as a normal
|
|
character:
|
|
|
|
foo|(bar$)
|
|
foo|^bar
|
|
|
|
If what's wanted is a "foo" or a bar-followed-by-a-newline, the
|
|
following could be used (the special '|' action is explained
|
|
below):
|
|
|
|
foo |
|
|
bar$ /* action goes here */
|
|
|
|
A similar trick will work for matching a foo or a bar-at-the-
|
|
beginning-of-a-line.
|
|
|
|
HOW THE INPUT IS MATCHED
|
|
When the generated scanner is run, it analyzes its input looking for
|
|
strings which match any of its patterns. If it finds more than one
|
|
match, it takes the one matching the most text (for trailing context
|
|
rules, this includes the length of the trailing part, even though it
|
|
will then be returned to the input). If it finds two or more matches of
|
|
|
|
6 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
the same length, the rule listed first in the flex input file is chosen.
|
|
|
|
Once the match is determined, the text corresponding to the match
|
|
(called the token) is made available in the global character pointer
|
|
yytext, and its length in the global integer yyleng. The action
|
|
corresponding to the matched pattern is then executed (a more detailed
|
|
description of actions follows), and then the remaining input is scanned
|
|
for another match.
|
|
|
|
If no match is found, then the default rule is executed: the next char-
|
|
acter in the input is considered matched and copied to the standard out-
|
|
put. Thus, the simplest legal flex input is:
|
|
|
|
%%
|
|
|
|
which generates a scanner that simply copies its input (one character at
|
|
a time) to its output.
|
|
|
|
ACTIONS
|
|
Each pattern in a rule has a corresponding action, which can be any
|
|
arbitrary C statement. The pattern ends at the first non-escaped whi-
|
|
tespace character; the remainder of the line is its action. If the
|
|
action is empty, then when the pattern is matched the input token is
|
|
simply discarded. For example, here is the specification for a program
|
|
which deletes all occurrences of "zap me" from its input:
|
|
|
|
%%
|
|
"zap me"
|
|
|
|
(It will copy all other characters in the input to the output since they
|
|
will be matched by the default rule.)
|
|
|
|
Here is a program which compresses multiple blanks and tabs down to a
|
|
single blank, and throws away whitespace found at the end of a line:
|
|
|
|
%%
|
|
[ \t]+ putchar( ' ' );
|
|
[ \t]+$ /* ignore this token */
|
|
|
|
If the action contains a '{', then the action spans till the balancing
|
|
'}' is found, and the action may cross multiple lines. flex knows about
|
|
C strings and comments and won't be fooled by braces found within them,
|
|
but also allows actions to begin with %{ and will consider the action to
|
|
be all the text up to the next %} (regardless of ordinary braces inside
|
|
the action).
|
|
|
|
An action consisting solely of a vertical bar ('|') means "same as the
|
|
action for the next rule." See below for an illustration.
|
|
|
|
Actions can include arbitrary C code, including return statements to
|
|
return a value to whatever routine called yylex(). Each time yylex() is
|
|
called it continues processing tokens from where it last left off until
|
|
it either reaches the end of the file or executes a return. Once it
|
|
|
|
Version 2.3 7
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
reaches an end-of-file, however, then any subsequent call to yylex()
|
|
will simply immediately return, unless yyrestart() is first called (see
|
|
below).
|
|
|
|
Actions are not allowed to modify yytext or yyleng.
|
|
|
|
There are a number of special directives which can be included within an
|
|
action:
|
|
|
|
- ECHO copies yytext to the scanner's output.
|
|
|
|
- BEGIN followed by the name of a start condition places the scanner
|
|
in the corresponding start condition (see below).
|
|
|
|
- REJECT directs the scanner to proceed on to the "second best" rule
|
|
which matched the input (or a prefix of the input). The rule is
|
|
chosen as described above in "How the Input is Matched", and yytext
|
|
and yyleng set up appropriately. It may either be one which
|
|
matched as much text as the originally chosen rule but came later
|
|
in the flex input file, or one which matched less text. For exam-
|
|
ple, the following will both count the words in the input and call
|
|
the routine special() whenever "frob" is seen:
|
|
|
|
int word_count = 0;
|
|
%%
|
|
|
|
frob special(); REJECT;
|
|
[^ \t\n]+ ++word_count;
|
|
|
|
Without the REJECT, any "frob"'s in the input would not be counted
|
|
as words, since the scanner normally executes only one action per
|
|
token. Multiple REJECT's are allowed, each one finding the next
|
|
best choice to the currently active rule. For example, when the
|
|
following scanner scans the token "abcd", it will write "abcdab-
|
|
caba" to the output:
|
|
|
|
%%
|
|
a |
|
|
ab |
|
|
abc |
|
|
abcd ECHO; REJECT;
|
|
.|\n /* eat up any unmatched character */
|
|
|
|
(The first three rules share the fourth's action since they use the
|
|
special '|' action.) REJECT is a particularly expensive feature in
|
|
terms scanner performance; if it is used in any of the scanner's
|
|
actions it will slow down all of the scanner's matching. Further-
|
|
more, REJECT cannot be used with the -f or -F options (see below).
|
|
|
|
Note also that unlike the other special actions, REJECT is a
|
|
branch; code immediately following it in the action will not be
|
|
executed.
|
|
|
|
- yymore() tells the scanner that the next time it matches a rule,
|
|
|
|
8 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
the corresponding token should be appended onto the current value
|
|
of yytext rather than replacing it. For example, given the input
|
|
"mega-kludge" the following will write "mega-mega-kludge" to the
|
|
output:
|
|
|
|
%%
|
|
mega- ECHO; yymore();
|
|
kludge ECHO;
|
|
|
|
First "mega-" is matched and echoed to the output. Then "kludge"
|
|
is matched, but the previous "mega-" is still hanging around at the
|
|
beginning of yytext so the ECHO for the "kludge" rule will actually
|
|
write "mega-kludge". The presence of yymore() in the scanner's
|
|
action entails a minor performance penalty in the scanner's match-
|
|
ing speed.
|
|
|
|
- yyless(n) returns all but the first n characters of the current
|
|
token back to the input stream, where they will be rescanned when
|
|
the scanner looks for the next match. yytext and yyleng are
|
|
adjusted appropriately (e.g., yyleng will now be equal to n ). For
|
|
example, on the input "foobar" the following will write out
|
|
"foobarbar":
|
|
|
|
%%
|
|
foobar ECHO; yyless(3);
|
|
[a-z]+ ECHO;
|
|
|
|
An argument of 0 to yyless will cause the entire current input
|
|
string to be scanned again. Unless you've changed how the scanner
|
|
will subsequently process its input (using BEGIN, for example),
|
|
this will result in an endless loop.
|
|
|
|
- unput(c) puts the character c back onto the input stream. It will
|
|
be the next character scanned. The following action will take the
|
|
current token and cause it to be rescanned enclosed in parentheses.
|
|
|
|
{
|
|
int i;
|
|
unput( ')' );
|
|
for ( i = yyleng - 1; i >= 0; --i )
|
|
unput( yytext[i] );
|
|
unput( '(' );
|
|
}
|
|
|
|
Note that since each unput() puts the given character back at the
|
|
beginning of the input stream, pushing back strings must be done
|
|
back-to-front.
|
|
|
|
- input() reads the next character from the input stream. For exam-
|
|
ple, the following is one way to eat up C comments:
|
|
|
|
%%
|
|
"/*" {
|
|
register int c;
|
|
|
|
Version 2.3 9
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
for ( ; ; )
|
|
{
|
|
while ( (c = input()) != '*' &&
|
|
c != EOF )
|
|
; /* eat up text of comment */
|
|
|
|
if ( c == '*' )
|
|
{
|
|
while ( (c = input()) == '*' )
|
|
;
|
|
if ( c == '/' )
|
|
break; /* found the end */
|
|
}
|
|
|
|
if ( c == EOF )
|
|
{
|
|
error( "EOF in comment" );
|
|
break;
|
|
}
|
|
}
|
|
}
|
|
|
|
(Note that if the scanner is compiled using C++, then input() is
|
|
instead referred to as yyinput(), in order to avoid a name clash
|
|
with the C++ stream by the name of input.)
|
|
|
|
- yyterminate() can be used in lieu of a return statement in an
|
|
action. It terminates the scanner and returns a 0 to the scanner's
|
|
caller, indicating "all done". Subsequent calls to the scanner
|
|
will immediately return unless preceded by a call to yyrestart()
|
|
(see below). By default, yyterminate() is also called when an
|
|
end-of-file is encountered. It is a macro and may be redefined.
|
|
|
|
THE GENERATED SCANNER
|
|
The output of flex is the file lex.yy.c, which contains the scanning
|
|
routine yylex(), a number of tables used by it for matching tokens, and
|
|
a number of auxiliary routines and macros. By default, yylex() is
|
|
declared as follows:
|
|
|
|
int yylex()
|
|
{
|
|
... various definitions and the actions in here ...
|
|
}
|
|
|
|
(If your environment supports function prototypes, then it will be "int
|
|
yylex( void )".) This definition may be changed by redefining the
|
|
"YY_DECL" macro. For example, you could use:
|
|
|
|
#undef YY_DECL
|
|
#define YY_DECL float lexscan( a, b ) float a, b;
|
|
|
|
to give the scanning routine the name lexscan, returning a float, and
|
|
taking two floats as arguments. Note that if you give arguments to the
|
|
|
|
10 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
scanning routine using a K&R-style/non-prototyped function declaration,
|
|
you must terminate the definition with a semi-colon (;).
|
|
|
|
Whenever yylex() is called, it scans tokens from the global input file
|
|
yyin (which defaults to stdin). It continues until it either reaches an
|
|
end-of-file (at which point it returns the value 0) or one of its
|
|
actions executes a return statement. In the former case, when called
|
|
again the scanner will immediately return unless yyrestart() is called
|
|
to point yyin at the new input file. ( yyrestart() takes one argument,
|
|
a FILE * pointer.) In the latter case (i.e., when an action executes a
|
|
return), the scanner may then be called again and it will resume scan-
|
|
ning where it left off.
|
|
|
|
By default (and for purposes of efficiency), the scanner uses block-
|
|
reads rather than simple getc() calls to read characters from yyin. The
|
|
nature of how it gets its input can be controlled by redefining the
|
|
YY_INPUT macro. YY_INPUT's calling sequence is
|
|
"YY_INPUT(buf,result,max_size)". Its action is to place up to max_size
|
|
characters in the character array buf and return in the integer variable
|
|
result either the number of characters read or the constant YY_NULL (0
|
|
on Unix systems) to indicate EOF. The default YY_INPUT reads from the
|
|
global file-pointer "yyin".
|
|
|
|
A sample redefinition of YY_INPUT (in the definitions section of the
|
|
input file):
|
|
|
|
%{
|
|
#undef YY_INPUT
|
|
#define YY_INPUT(buf,result,max_size) \
|
|
{ \
|
|
int c = getchar(); \
|
|
result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
|
|
}
|
|
%}
|
|
|
|
This definition will change the input processing to occur one character
|
|
at a time.
|
|
|
|
You also can add in things like keeping track of the input line number
|
|
this way; but don't expect your scanner to go very fast.
|
|
|
|
When the scanner receives an end-of-file indication from YY_INPUT, it
|
|
then checks the yywrap() function. If yywrap() returns false (zero),
|
|
then it is assumed that the function has gone ahead and set up yyin to
|
|
point to another input file, and scanning continues. If it returns true
|
|
(non-zero), then the scanner terminates, returning 0 to its caller.
|
|
|
|
The default yywrap() always returns 1. Presently, to redefine it you
|
|
must first "#undef yywrap", as it is currently implemented as a macro.
|
|
As indicated by the hedging in the previous sentence, it may be changed
|
|
to a true function in the near future.
|
|
|
|
The scanner writes its ECHO output to the yyout global (default,
|
|
stdout), which may be redefined by the user simply by assigning it to
|
|
|
|
Version 2.3 11
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
some other FILE pointer.
|
|
|
|
START CONDITIONS
|
|
flex provides a mechanism for conditionally activating rules. Any rule
|
|
whose pattern is prefixed with "<sc>" will only be active when the
|
|
scanner is in the start condition named "sc". For example,
|
|
|
|
<STRING>[^"]* { /* eat up the string body ... */
|
|
...
|
|
}
|
|
|
|
will be active only when the scanner is in the "STRING" start condition,
|
|
and
|
|
|
|
<INITIAL,STRING,QUOTE>\. { /* handle an escape ... */
|
|
...
|
|
}
|
|
|
|
will be active only when the current start condition is either "INI-
|
|
TIAL", "STRING", or "QUOTE".
|
|
|
|
Start conditions are declared in the definitions (first) section of the
|
|
input using unindented lines beginning with either %s or %x followed by
|
|
a list of names. The former declares inclusive start conditions, the
|
|
latter exclusive start conditions. A start condition is activated using
|
|
the BEGIN action. Until the next BEGIN action is executed, rules with
|
|
the given start condition will be active and rules with other start con-
|
|
ditions will be inactive. If the start condition is inclusive, then
|
|
rules with no start conditions at all will also be active. If it is
|
|
exclusive, then only rules qualified with the start condition will be
|
|
active. A set of rules contingent on the same exclusive start condition
|
|
describe a scanner which is independent of any of the other rules in the
|
|
flex input. Because of this, exclusive start conditions make it easy to
|
|
specify "mini-scanners" which scan portions of the input that are syn-
|
|
tactically different from the rest (e.g., comments).
|
|
|
|
If the distinction between inclusive and exclusive start conditions is
|
|
still a little vague, here's a simple example illustrating the connec-
|
|
tion between the two. The set of rules:
|
|
|
|
%s example
|
|
%%
|
|
<example>foo /* do something */
|
|
|
|
is equivalent to
|
|
|
|
%x example
|
|
%%
|
|
<INITIAL,example>foo /* do something */
|
|
|
|
The default rule (to ECHO any unmatched character) remains active in
|
|
start conditions.
|
|
|
|
12 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
BEGIN(0) returns to the original state where only the rules with no
|
|
start conditions are active. This state can also be referred to as the
|
|
start-condition "INITIAL", so BEGIN(INITIAL) is equivalent to BEGIN(0).
|
|
(The parentheses around the start condition name are not required but
|
|
are considered good style.)
|
|
|
|
BEGIN actions can also be given as indented code at the beginning of the
|
|
rules section. For example, the following will cause the scanner to
|
|
enter the "SPECIAL" start condition whenever yylex() is called and the
|
|
global variable enter_special is true:
|
|
|
|
int enter_special;
|
|
|
|
%x SPECIAL
|
|
%%
|
|
if ( enter_special )
|
|
BEGIN(SPECIAL);
|
|
|
|
<SPECIAL>blahblahblah
|
|
...more rules follow...
|
|
|
|
To illustrate the uses of start conditions, here is a scanner which pro-
|
|
vides two different interpretations of a string like "123.456". By
|
|
default it will treat it as as three tokens, the integer "123", a dot
|
|
('.'), and the integer "456". But if the string is preceded earlier in
|
|
the line by the string "expect-floats" it will treat it as a single
|
|
token, the floating-point number 123.456:
|
|
|
|
%{
|
|
#include <math.h>
|
|
%}
|
|
%s expect
|
|
|
|
%%
|
|
expect-floats BEGIN(expect);
|
|
|
|
<expect>[0-9]+"."[0-9]+ {
|
|
printf( "found a float, = %f\n",
|
|
atof( yytext ) );
|
|
}
|
|
<expect>\n {
|
|
/* that's the end of the line, so
|
|
* we need another "expect-number"
|
|
* before we'll recognize any more
|
|
* numbers
|
|
*/
|
|
BEGIN(INITIAL);
|
|
}
|
|
|
|
[0-9]+ {
|
|
printf( "found an integer, = %d\n",
|
|
atoi( yytext ) );
|
|
}
|
|
|
|
Version 2.3 13
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
"." printf( "found a dot\n" );
|
|
|
|
Here is a scanner which recognizes (and discards) C comments while main-
|
|
taining a count of the current input line.
|
|
|
|
%x comment
|
|
%%
|
|
int line_num = 1;
|
|
|
|
"/*" BEGIN(comment);
|
|
|
|
<comment>[^*\n]* /* eat anything that's not a '*' */
|
|
<comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
|
|
<comment>\n ++line_num;
|
|
<comment>"*"+"/" BEGIN(INITIAL);
|
|
|
|
Note that start-conditions names are really integer values and can be
|
|
stored as such. Thus, the above could be extended in the following
|
|
fashion:
|
|
|
|
%x comment foo
|
|
%%
|
|
int line_num = 1;
|
|
int comment_caller;
|
|
|
|
"/*" {
|
|
comment_caller = INITIAL;
|
|
BEGIN(comment);
|
|
}
|
|
|
|
...
|
|
|
|
<foo>"/*" {
|
|
comment_caller = foo;
|
|
BEGIN(comment);
|
|
}
|
|
|
|
<comment>[^*\n]* /* eat anything that's not a '*' */
|
|
<comment>"*"+[^*/\n]* /* eat up '*'s not followed by '/'s */
|
|
<comment>\n ++line_num;
|
|
<comment>"*"+"/" BEGIN(comment_caller);
|
|
|
|
One can then implement a "stack" of start conditions using an array of
|
|
integers. (It is likely that such stacks will become a full-fledged
|
|
flex feature in the future.) Note, though, that start conditions do not
|
|
have their own name-space; %s's and %x's declare names in the same
|
|
fashion as #define's.
|
|
|
|
MULTIPLE INPUT BUFFERS
|
|
Some scanners (such as those which support "include" files) require
|
|
reading from several input streams. As flex scanners do a large amount
|
|
of buffering, one cannot control where the next input will be read from
|
|
by simply writing a YY_INPUT which is sensitive to the scanning context.
|
|
|
|
14 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
YY_INPUT is only called when the scanner reaches the end of its buffer,
|
|
which may be a long time after scanning a statement such as an "include"
|
|
which requires switching the input source.
|
|
|
|
To negotiate these sorts of problems, flex provides a mechanism for
|
|
creating and switching between multiple input buffers. An input buffer
|
|
is created by using:
|
|
|
|
YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )
|
|
|
|
which takes a FILE pointer and a size and creates a buffer associated
|
|
with the given file and large enough to hold size characters (when in
|
|
doubt, use YY_BUF_SIZE for the size). It returns a YY_BUFFER_STATE han-
|
|
dle, which may then be passed to other routines:
|
|
|
|
void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )
|
|
|
|
switches the scanner's input buffer so subsequent tokens will come from
|
|
new_buffer. Note that yy_switch_to_buffer() may be used by yywrap() to
|
|
sets things up for continued scanning, instead of opening a new file and
|
|
pointing yyin at it.
|
|
|
|
void yy_delete_buffer( YY_BUFFER_STATE buffer )
|
|
|
|
is used to reclaim the storage associated with a buffer.
|
|
|
|
yy_new_buffer() is an alias for yy_create_buffer(), provided for compa-
|
|
tibility with the C++ use of new and delete for creating and destroying
|
|
dynamic objects.
|
|
|
|
Finally, the YY_CURRENT_BUFFER macro returns a YY_BUFFER_STATE handle to
|
|
the current buffer.
|
|
|
|
Here is an example of using these features for writing a scanner which
|
|
expands include files (the <<EOF>> feature is discussed below):
|
|
|
|
/* the "incl" state is used for picking up the name
|
|
* of an include file
|
|
*/
|
|
%x incl
|
|
|
|
%{
|
|
#define MAX_INCLUDE_DEPTH 10
|
|
YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
|
|
int include_stack_ptr = 0;
|
|
%}
|
|
|
|
%%
|
|
include BEGIN(incl);
|
|
|
|
[a-z]+ ECHO;
|
|
[^a-z\n]*\n? ECHO;
|
|
|
|
<incl>[ \t]* /* eat the whitespace */
|
|
|
|
Version 2.3 15
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
<incl>[^ \t\n]+ { /* got the include file name */
|
|
if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
|
|
{
|
|
fprintf( stderr, "Includes nested too deeply" );
|
|
exit( 1 );
|
|
}
|
|
|
|
include_stack[include_stack_ptr++] =
|
|
YY_CURRENT_BUFFER;
|
|
|
|
yyin = fopen( yytext, "r" );
|
|
|
|
if ( ! yyin )
|
|
error( ... );
|
|
|
|
yy_switch_to_buffer(
|
|
yy_create_buffer( yyin, YY_BUF_SIZE ) );
|
|
|
|
BEGIN(INITIAL);
|
|
}
|
|
|
|
<<EOF>> {
|
|
if ( --include_stack_ptr < 0 )
|
|
{
|
|
yyterminate();
|
|
}
|
|
|
|
else
|
|
yy_switch_to_buffer(
|
|
include_stack[include_stack_ptr] );
|
|
}
|
|
|
|
END-OF-FILE RULES
|
|
The special rule "<<EOF>>" indicates actions which are to be taken when
|
|
an end-of-file is encountered and yywrap() returns non-zero (i.e., indi-
|
|
cates no further files to process). The action must finish by doing one
|
|
of four things:
|
|
|
|
- the special YY_NEW_FILE action, if yyin has been pointed at a new
|
|
file to process;
|
|
|
|
- a return statement;
|
|
|
|
- the special yyterminate() action;
|
|
|
|
- or, switching to a new buffer using yy_switch_to_buffer() as shown
|
|
in the example above.
|
|
|
|
<<EOF>> rules may not be used with other patterns; they may only be
|
|
qualified with a list of start conditions. If an unqualified <<EOF>>
|
|
rule is given, it applies to all start conditions which do not already
|
|
have <<EOF>> actions. To specify an <<EOF>> rule for only the initial
|
|
start condition, use
|
|
|
|
16 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
<INITIAL><<EOF>>
|
|
|
|
These rules are useful for catching things like unclosed comments. An
|
|
example:
|
|
|
|
%x quote
|
|
%%
|
|
|
|
...other rules for dealing with quotes...
|
|
|
|
<quote><<EOF>> {
|
|
error( "unterminated quote" );
|
|
yyterminate();
|
|
}
|
|
<<EOF>> {
|
|
if ( *++filelist )
|
|
{
|
|
yyin = fopen( *filelist, "r" );
|
|
YY_NEW_FILE;
|
|
}
|
|
else
|
|
yyterminate();
|
|
}
|
|
|
|
MISCELLANEOUS MACROS
|
|
The macro YY_USER_ACTION can be redefined to provide an action which is
|
|
always executed prior to the matched rule's action. For example, it
|
|
could be #define'd to call a routine to convert yytext to lower-case.
|
|
|
|
The macro YY_USER_INIT may be redefined to provide an action which is
|
|
always executed before the first scan (and before the scanner's internal
|
|
initializations are done). For example, it could be used to call a rou-
|
|
tine to read in a data table or open a logging file.
|
|
|
|
In the generated scanner, the actions are all gathered in one large
|
|
switch statement and separated using YY_BREAK, which may be redefined.
|
|
By default, it is simply a "break", to separate each rule's action from
|
|
the following rule's. Redefining YY_BREAK allows, for example, C++
|
|
users to #define YY_BREAK to do nothing (while being very careful that
|
|
every rule ends with a "break" or a "return"!) to avoid suffering from
|
|
unreachable statement warnings where because a rule's action ends with
|
|
"return", the YY_BREAK is inaccessible.
|
|
|
|
INTERFACING WITH YACC
|
|
One of the main uses of flex is as a companion to the yacc parser-
|
|
generator. yacc parsers expect to call a routine named yylex() to find
|
|
the next input token. The routine is supposed to return the type of the
|
|
next token as well as putting any associated value in the global yylval.
|
|
To use flex with yacc, one specifies the -d option to yacc to instruct
|
|
it to generate the file y.tab.h containing definitions of all the
|
|
%tokens appearing in the yacc input. This file is then included in the
|
|
|
|
Version 2.3 17
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
flex scanner. For example, if one of the tokens is "TOK_NUMBER", part
|
|
of the scanner might look like:
|
|
|
|
%{
|
|
#include "y.tab.h"
|
|
%}
|
|
|
|
%%
|
|
|
|
[0-9]+ yylval = atoi( yytext ); return TOK_NUMBER;
|
|
|
|
TRANSLATION TABLE
|
|
In the name of POSIX compliance, flex supports a translation table for
|
|
mapping input characters into groups. The table is specified in the
|
|
first section, and its format looks like:
|
|
|
|
%t
|
|
1 abcd
|
|
2 ABCDEFGHIJKLMNOPQRSTUVWXYZ
|
|
52 0123456789
|
|
6 \t\ \n
|
|
%t
|
|
|
|
This example specifies that the characters 'a', 'b', 'c', and 'd' are to
|
|
all be lumped into group #1, upper-case letters in group #2, digits in
|
|
group #52, tabs, blanks, and newlines into group #6, and no other char-
|
|
acters will appear in the patterns. The group numbers are actually
|
|
disregarded by flex; %t serves, though, to lump characters together.
|
|
Given the above table, for example, the pattern "a(AA)*5" is equivalent
|
|
to "d(ZQ)*0". They both say, "match any character in group #1, followed
|
|
by zero-or-more pairs of characters from group #2, followed by a charac-
|
|
ter from group #52." Thus %t provides a crude way for introducing
|
|
equivalence classes into the scanner specification.
|
|
|
|
Note that the -i option (see below) coupled with the equivalence classes
|
|
which flex automatically generates take care of virtually all the
|
|
instances when one might consider using %t. But what the hell, it's
|
|
there if you want it.
|
|
|
|
OPTIONS
|
|
flex has the following options:
|
|
|
|
-b Generate backtracking information to lex.backtrack. This is a list
|
|
of scanner states which require backtracking and the input charac-
|
|
ters on which they do so. By adding rules one can remove back-
|
|
tracking states. If all backtracking states are eliminated and -f
|
|
or -F is used, the generated scanner will run faster (see the -p
|
|
flag). Only users who wish to squeeze every last cycle out of
|
|
their scanners need worry about this option. (See the section on
|
|
PERFORMANCE CONSIDERATIONS below.)
|
|
|
|
-c is a do-nothing, deprecated option included for POSIX compliance.
|
|
|
|
18 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
NOTE: in previous releases of flex -c specified table-compression
|
|
options. This functionality is now given by the -C flag. To ease
|
|
the the impact of this change, when flex encounters -c, it
|
|
currently issues a warning message and assumes that -C was desired
|
|
instead. In the future this "promotion" of -c to -C will go away
|
|
in the name of full POSIX compliance (unless the POSIX meaning is
|
|
removed first).
|
|
|
|
-d makes the generated scanner run in debug mode. Whenever a pattern
|
|
is recognized and the global yy_flex_debug is non-zero (which is
|
|
the default), the scanner will write to stderr a line of the form:
|
|
|
|
--accepting rule at line 53 ("the matched text")
|
|
|
|
The line number refers to the location of the rule in the file
|
|
defining the scanner (i.e., the file that was fed to flex). Mes-
|
|
sages are also generated when the scanner backtracks, accepts the
|
|
default rule, reaches the end of its input buffer (or encounters a
|
|
NUL; at this point, the two look the same as far as the scanner's
|
|
concerned), or reaches an end-of-file.
|
|
|
|
-f specifies (take your pick) full table or fast scanner. No table
|
|
compression is done. The result is large but fast. This option is
|
|
equivalent to -Cf (see below).
|
|
|
|
-i instructs flex to generate a case-insensitive scanner. The case of
|
|
letters given in the flex input patterns will be ignored, and
|
|
tokens in the input will be matched regardless of case. The
|
|
matched text given in yytext will have the preserved case (i.e., it
|
|
will not be folded).
|
|
|
|
-n is another do-nothing, deprecated option included only for POSIX
|
|
compliance.
|
|
|
|
-p generates a performance report to stderr. The report consists of
|
|
comments regarding features of the flex input file which will cause
|
|
a loss of performance in the resulting scanner. Note that the use
|
|
of REJECT and variable trailing context (see the BUGS section in
|
|
flex(1)) entails a substantial performance penalty; use of
|
|
yymore(), the ^ operator, and the -I flag entail minor performance
|
|
penalties.
|
|
|
|
-s causes the default rule (that unmatched scanner input is echoed to
|
|
stdout) to be suppressed. If the scanner encounters input that
|
|
does not match any of its rules, it aborts with an error. This
|
|
option is useful for finding holes in a scanner's rule set.
|
|
|
|
-t instructs flex to write the scanner it generates to standard output
|
|
instead of lex.yy.c.
|
|
|
|
-v specifies that flex should write to stderr a summary of statistics
|
|
regarding the scanner it generates. Most of the statistics are
|
|
meaningless to the casual flex user, but the first line identifies
|
|
the version of flex, which is useful for figuring out where you
|
|
|
|
Version 2.3 19
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
stand with respect to patches and new releases, and the next two
|
|
lines give the date when the scanner was created and a summary of
|
|
the flags which were in effect.
|
|
|
|
-F specifies that the fast scanner table representation should be
|
|
used. This representation is about as fast as the full table
|
|
representation (-f), and for some sets of patterns will be consid-
|
|
erably smaller (and for others, larger). In general, if the pat-
|
|
tern set contains both "keywords" and a catch-all, "identifier"
|
|
rule, such as in the set:
|
|
|
|
"case" return TOK_CASE;
|
|
"switch" return TOK_SWITCH;
|
|
...
|
|
"default" return TOK_DEFAULT;
|
|
[a-z]+ return TOK_ID;
|
|
|
|
then you're better off using the full table representation. If
|
|
only the "identifier" rule is present and you then use a hash table
|
|
or some such to detect the keywords, you're better off using -F.
|
|
|
|
This option is equivalent to -CF (see below).
|
|
|
|
-I instructs flex to generate an interactive scanner. Normally,
|
|
scanners generated by flex always look ahead one character before
|
|
deciding that a rule has been matched. At the cost of some scan-
|
|
ning overhead, flex will generate a scanner which only looks ahead
|
|
when needed. Such scanners are called interactive because if you
|
|
want to write a scanner for an interactive system such as a command
|
|
shell, you will probably want the user's input to be terminated
|
|
with a newline, and without -I the user will have to type a charac-
|
|
ter in addition to the newline in order to have the newline recog-
|
|
nized. This leads to dreadful interactive performance.
|
|
|
|
If all this seems to confusing, here's the general rule: if a human
|
|
will be typing in input to your scanner, use -I, otherwise don't;
|
|
if you don't care about squeezing the utmost performance from your
|
|
scanner and you don't want to make any assumptions about the input
|
|
to your scanner, use -I.
|
|
|
|
|
|
Note, -I cannot be used in conjunction with full or fast tables,
|
|
i.e., the -f, -F, -Cf, or -CF flags.
|
|
|
|
-L instructs flex not to generate #line directives. Without this
|
|
option, flex peppers the generated scanner with #line directives so
|
|
error messages in the actions will be correctly located with
|
|
respect to the original flex input file, and not to the fairly
|
|
meaningless line numbers of lex.yy.c. (Unfortunately flex does not
|
|
presently generate the necessary directives to "retarget" the line
|
|
numbers for those parts of lex.yy.c which it generated. So if
|
|
there is an error in the generated code, a meaningless line number
|
|
is reported.)
|
|
|
|
-T makes flex run in trace mode. It will generate a lot of messages
|
|
|
|
20 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
to stdout concerning the form of the input and the resultant non-
|
|
deterministic and deterministic finite automata. This option is
|
|
mostly for use in maintaining flex.
|
|
|
|
-8 instructs flex to generate an 8-bit scanner, i.e., one which can
|
|
recognize 8-bit characters. On some sites, flex is installed with
|
|
this option as the default. On others, the default is 7-bit char-
|
|
acters. To see which is the case, check the verbose (-v) output
|
|
for "equivalence classes created". If the denominator of the
|
|
number shown is 128, then by default flex is generating 7-bit char-
|
|
acters. If it is 256, then the default is 8-bit characters and the
|
|
-8 flag is not required (but may be a good idea to keep the scanner
|
|
specification portable). Feeding a 7-bit scanner 8-bit characters
|
|
will result in infinite loops, bus errors, or other such fireworks,
|
|
so when in doubt, use the flag. Note that if equivalence classes
|
|
are used, 8-bit scanners take only slightly more table space than
|
|
7-bit scanners (128 bytes, to be exact); if equivalence classes are
|
|
not used, however, then the tables may grow up to twice their 7-bit
|
|
size.
|
|
|
|
-C[efmF]
|
|
controls the degree of table compression.
|
|
|
|
-Ce directs flex to construct equivalence classes, i.e., sets of
|
|
characters which have identical lexical properties (for example, if
|
|
the only appearance of digits in the flex input is in the character
|
|
class "[0-9]" then the digits '0', '1', ..., '9' will all be put in
|
|
the same equivalence class). Equivalence classes usually give
|
|
dramatic reductions in the final table/object file sizes (typically
|
|
a factor of 2-5) and are pretty cheap performance-wise (one array
|
|
look-up per character scanned).
|
|
|
|
-Cf specifies that the full scanner tables should be generated -
|
|
flex should not compress the tables by taking advantages of similar
|
|
transition functions for different states.
|
|
|
|
-CF specifies that the alternate fast scanner representation
|
|
(described above under the -F flag) should be used.
|
|
|
|
-Cm directs flex to construct meta-equivalence classes, which are
|
|
sets of equivalence classes (or characters, if equivalence classes
|
|
are not being used) that are commonly used together. Meta-
|
|
equivalence classes are often a big win when using compressed
|
|
tables, but they have a moderate performance impact (one or two
|
|
"if" tests and one array look-up per character scanned).
|
|
|
|
A lone -C specifies that the scanner tables should be compressed
|
|
but neither equivalence classes nor meta-equivalence classes should
|
|
be used.
|
|
|
|
The options -Cf or -CF and -Cm do not make sense together - there
|
|
is no opportunity for meta-equivalence classes if the table is not
|
|
being compressed. Otherwise the options may be freely mixed.
|
|
|
|
Version 2.3 21
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
The default setting is -Cem, which specifies that flex should gen-
|
|
erate equivalence classes and meta-equivalence classes. This set-
|
|
ting provides the highest degree of table compression. You can
|
|
trade off faster-executing scanners at the cost of larger tables
|
|
with the following generally being true:
|
|
|
|
slowest & smallest
|
|
-Cem
|
|
-Cm
|
|
-Ce
|
|
-C
|
|
-C{f,F}e
|
|
-C{f,F}
|
|
fastest & largest
|
|
|
|
Note that scanners with the smallest tables are usually generated
|
|
and compiled the quickest, so during development you will usually
|
|
want to use the default, maximal compression.
|
|
|
|
-Cfe is often a good compromise between speed and size for produc-
|
|
tion scanners.
|
|
|
|
-C options are not cumulative; whenever the flag is encountered,
|
|
the previous -C settings are forgotten.
|
|
|
|
-Sskeleton_file
|
|
overrides the default skeleton file from which flex constructs its
|
|
scanners. You'll never need this option unless you are doing flex
|
|
maintenance or development.
|
|
|
|
PERFORMANCE CONSIDERATIONS
|
|
The main design goal of flex is that it generate high-performance
|
|
scanners. It has been optimized for dealing well with large sets of
|
|
rules. Aside from the effects of table compression on scanner speed
|
|
outlined above, there are a number of options/actions which degrade per-
|
|
formance. These are, from most expensive to least:
|
|
|
|
REJECT
|
|
|
|
pattern sets that require backtracking
|
|
arbitrary trailing context
|
|
|
|
'^' beginning-of-line operator
|
|
yymore()
|
|
|
|
with the first three all being quite expensive and the last two being
|
|
quite cheap.
|
|
|
|
REJECT should be avoided at all costs when performance is important. It
|
|
is a particularly expensive option.
|
|
|
|
Getting rid of backtracking is messy and often may be an enormous amount
|
|
of work for a complicated scanner. In principal, one begins by using
|
|
the -b flag to generate a lex.backtrack file. For example, on the input
|
|
|
|
22 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
%%
|
|
foo return TOK_KEYWORD;
|
|
foobar return TOK_KEYWORD;
|
|
|
|
the file looks like:
|
|
|
|
State #6 is non-accepting -
|
|
associated rule line numbers:
|
|
2 3
|
|
out-transitions: [ o ]
|
|
jam-transitions: EOF [ \001-n p-\177 ]
|
|
|
|
State #8 is non-accepting -
|
|
associated rule line numbers:
|
|
3
|
|
out-transitions: [ a ]
|
|
jam-transitions: EOF [ \001-` b-\177 ]
|
|
|
|
State #9 is non-accepting -
|
|
associated rule line numbers:
|
|
3
|
|
out-transitions: [ r ]
|
|
jam-transitions: EOF [ \001-q s-\177 ]
|
|
|
|
Compressed tables always backtrack.
|
|
|
|
The first few lines tell us that there's a scanner state in which it can
|
|
make a transition on an 'o' but not on any other character, and that in
|
|
that state the currently scanned text does not match any rule. The
|
|
state occurs when trying to match the rules found at lines 2 and 3 in
|
|
the input file. If the scanner is in that state and then reads some-
|
|
thing other than an 'o', it will have to backtrack to find a rule which
|
|
is matched. With a bit of headscratching one can see that this must be
|
|
the state it's in when it has seen "fo". When this has happened, if
|
|
anything other than another 'o' is seen, the scanner will have to back
|
|
up to simply match the 'f' (by the default rule).
|
|
|
|
The comment regarding State #8 indicates there's a problem when "foob"
|
|
has been scanned. Indeed, on any character other than a 'b', the
|
|
scanner will have to back up to accept "foo". Similarly, the comment
|
|
for State #9 concerns when "fooba" has been scanned.
|
|
|
|
The final comment reminds us that there's no point going to all the
|
|
trouble of removing backtracking from the rules unless we're using -f or
|
|
-F, since there's no performance gain doing so with compressed scanners.
|
|
|
|
The way to remove the backtracking is to add "error" rules:
|
|
|
|
%%
|
|
foo return TOK_KEYWORD;
|
|
foobar return TOK_KEYWORD;
|
|
|
|
fooba |
|
|
|
|
Version 2.3 23
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
foob |
|
|
fo {
|
|
/* false alarm, not really a keyword */
|
|
return TOK_ID;
|
|
}
|
|
|
|
Eliminating backtracking among a list of keywords can also be done using
|
|
a "catch-all" rule:
|
|
|
|
%%
|
|
foo return TOK_KEYWORD;
|
|
foobar return TOK_KEYWORD;
|
|
|
|
[a-z]+ return TOK_ID;
|
|
|
|
This is usually the best solution when appropriate.
|
|
|
|
Backtracking messages tend to cascade. With a complicated set of rules
|
|
it's not uncommon to get hundreds of messages. If one can decipher
|
|
them, though, it often only takes a dozen or so rules to eliminate the
|
|
backtracking (though it's easy to make a mistake and have an error rule
|
|
accidentally match a valid token. A possible future flex feature will
|
|
be to automatically add rules to eliminate backtracking).
|
|
|
|
Variable trailing context (where both the leading and trailing parts do
|
|
not have a fixed length) entails almost the same performance loss as
|
|
REJECT (i.e., substantial). So when possible a rule like:
|
|
|
|
%%
|
|
mouse|rat/(cat|dog) run();
|
|
|
|
is better written:
|
|
|
|
%%
|
|
mouse/cat|dog run();
|
|
rat/cat|dog run();
|
|
|
|
or as
|
|
|
|
%%
|
|
mouse|rat/cat run();
|
|
mouse|rat/dog run();
|
|
|
|
Note that here the special '|' action does not provide any savings, and
|
|
can even make things worse (see BUGS in flex(1)).
|
|
|
|
Another area where the user can increase a scanner's performance (and
|
|
one that's easier to implement) arises from the fact that the longer the
|
|
tokens matched, the faster the scanner will run. This is because with
|
|
long tokens the processing of most input characters takes place in the
|
|
(short) inner scanning loop, and does not often have to go through the
|
|
additional work of setting up the scanning environment (e.g., yytext)
|
|
for the action. Recall the scanner for C comments:
|
|
|
|
24 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
%x comment
|
|
%%
|
|
int line_num = 1;
|
|
|
|
"/*" BEGIN(comment);
|
|
|
|
<comment>[^*\n]*
|
|
<comment>"*"+[^*/\n]*
|
|
<comment>\n ++line_num;
|
|
<comment>"*"+"/" BEGIN(INITIAL);
|
|
|
|
This could be sped up by writing it as:
|
|
|
|
%x comment
|
|
%%
|
|
int line_num = 1;
|
|
|
|
"/*" BEGIN(comment);
|
|
|
|
<comment>[^*\n]*
|
|
<comment>[^*\n]*\n ++line_num;
|
|
<comment>"*"+[^*/\n]*
|
|
<comment>"*"+[^*/\n]*\n ++line_num;
|
|
<comment>"*"+"/" BEGIN(INITIAL);
|
|
|
|
Now instead of each newline requiring the processing of another action,
|
|
recognizing the newlines is "distributed" over the other rules to keep
|
|
the matched text as long as possible. Note that adding rules does not
|
|
slow down the scanner! The speed of the scanner is independent of the
|
|
number of rules or (modulo the considerations given at the beginning of
|
|
this section) how complicated the rules are with regard to operators
|
|
such as '*' and '|'.
|
|
|
|
A final example in speeding up a scanner: suppose you want to scan
|
|
through a file containing identifiers and keywords, one per line and
|
|
with no other extraneous characters, and recognize all the keywords. A
|
|
natural first approach is:
|
|
|
|
%%
|
|
asm |
|
|
auto |
|
|
break |
|
|
... etc ...
|
|
volatile |
|
|
while /* it's a keyword */
|
|
|
|
.|\n /* it's not a keyword */
|
|
|
|
To eliminate the back-tracking, introduce a catch-all rule:
|
|
|
|
%%
|
|
asm |
|
|
auto |
|
|
|
|
Version 2.3 25
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
break |
|
|
... etc ...
|
|
volatile |
|
|
while /* it's a keyword */
|
|
|
|
[a-z]+ |
|
|
.|\n /* it's not a keyword */
|
|
|
|
Now, if it's guaranteed that there's exactly one word per line, then we
|
|
can reduce the total number of matches by a half by merging in the
|
|
recognition of newlines with that of the other tokens:
|
|
|
|
%%
|
|
asm\n |
|
|
auto\n |
|
|
break\n |
|
|
... etc ...
|
|
volatile\n |
|
|
while\n /* it's a keyword */
|
|
|
|
[a-z]+\n |
|
|
.|\n /* it's not a keyword */
|
|
|
|
One has to be careful here, as we have now reintroduced backtracking
|
|
into the scanner. In particular, while we know that there will never be
|
|
any characters in the input stream other than letters or newlines, flex
|
|
can't figure this out, and it will plan for possibly needing backtrack-
|
|
ing when it has scanned a token like "auto" and then the next character
|
|
is something other than a newline or a letter. Previously it would then
|
|
just match the "auto" rule and be done, but now it has no "auto" rule,
|
|
only a "auto\n" rule. To eliminate the possibility of backtracking, we
|
|
could either duplicate all rules but without final newlines, or, since
|
|
we never expect to encounter such an input and therefore don't how it's
|
|
classified, we can introduce one more catch-all rule, this one which
|
|
doesn't include a newline:
|
|
|
|
%%
|
|
asm\n |
|
|
auto\n |
|
|
break\n |
|
|
... etc ...
|
|
volatile\n |
|
|
while\n /* it's a keyword */
|
|
|
|
[a-z]+\n |
|
|
[a-z]+ |
|
|
.|\n /* it's not a keyword */
|
|
|
|
Compiled with -Cf, this is about as fast as one can get a flex scanner
|
|
to go for this particular problem.
|
|
|
|
A final note: flex is slow when matching NUL's, particularly when a
|
|
token contains multiple NUL's. It's best to write rules which match
|
|
short amounts of text if it's anticipated that the text will often
|
|
|
|
26 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
include NUL's.
|
|
|
|
INCOMPATIBILITIES WITH LEX AND POSIX
|
|
flex is a rewrite of the Unix lex tool (the two implementations do not
|
|
share any code, though), with some extensions and incompatibilities,
|
|
both of which are of concern to those who wish to write scanners accept-
|
|
able to either implementation. At present, the POSIX lex draft is very
|
|
close to the original lex implementation, so some of these incompatibil-
|
|
ities are also in conflict with the POSIX draft. But the intent is that
|
|
except as noted below, flex as it presently stands will ultimately be
|
|
POSIX conformant (i.e., that those areas of conflict with the POSIX
|
|
draft will be resolved in flex's favor). Please bear in mind that all
|
|
the comments which follow are with regard to the POSIX draft standard of
|
|
Summer 1989, and not the final document (or subsequent drafts); they are
|
|
included so flex users can be aware of the standardization issues and
|
|
those areas where flex may in the near future undergo changes incompati-
|
|
ble with its current definition.
|
|
|
|
flex is fully compatible with lex with the following exceptions:
|
|
|
|
- The undocumented lex scanner internal variable yylineno is not sup-
|
|
ported. It is difficult to support this option efficiently, since
|
|
it requires examining every character scanned and reexamining the
|
|
characters when the scanner backs up. Things get more complicated
|
|
when the end of buffer or file is reached or a NUL is scanned
|
|
(since the scan must then be restarted with the proper line number
|
|
count), or the user uses the yyless(), unput(), or REJECT actions,
|
|
or the multiple input buffer functions.
|
|
|
|
The fix is to add rules which, upon seeing a newline, increment
|
|
yylineno. This is usually an easy process, though it can be a drag
|
|
if some of the patterns can match multiple newlines along with
|
|
other characters.
|
|
|
|
yylineno is not part of the POSIX draft.
|
|
|
|
- The input() routine is not redefinable, though it may be called to
|
|
read characters following whatever has been matched by a rule. If
|
|
input() encounters an end-of-file the normal yywrap() processing is
|
|
done. A ``real'' end-of-file is returned by input() as EOF.
|
|
|
|
Input is instead controlled by redefining the YY_INPUT macro.
|
|
|
|
The flex restriction that input() cannot be redefined is in accor-
|
|
dance with the POSIX draft, but YY_INPUT has not yet been accepted
|
|
into the draft (and probably won't; it looks like the draft will
|
|
simply not specify any way of controlling the scanner's input other
|
|
than by making an initial assignment to yyin).
|
|
|
|
- flex scanners do not use stdio for input. Because of this, when
|
|
writing an interactive scanner one must explicitly call fflush() on
|
|
the stream associated with the terminal after writing out a prompt.
|
|
With lex such writes are automatically flushed since lex scanners
|
|
use getchar() for their input. Also, when writing interactive
|
|
|
|
Version 2.3 27
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
scanners with flex, the -I flag must be used.
|
|
|
|
- flex scanners are not as reentrant as lex scanners. In particular,
|
|
if you have an interactive scanner and an interrupt handler which
|
|
long-jumps out of the scanner, and the scanner is subsequently
|
|
called again, you may get the following message:
|
|
|
|
fatal flex scanner internal error--end of buffer missed
|
|
|
|
To reenter the scanner, first use
|
|
|
|
yyrestart( yyin );
|
|
|
|
- output() is not supported. Output from the ECHO macro is done to
|
|
the file-pointer yyout (default stdout).
|
|
|
|
The POSIX draft mentions that an output() routine exists but
|
|
currently gives no details as to what it does.
|
|
|
|
- lex does not support exclusive start conditions (%x), though they
|
|
are in the current POSIX draft.
|
|
|
|
- When definitions are expanded, flex encloses them in parentheses.
|
|
With lex, the following:
|
|
|
|
NAME [A-Z][A-Z0-9]*
|
|
%%
|
|
foo{NAME}? printf( "Found it\n" );
|
|
%%
|
|
|
|
will not match the string "foo" because when the macro is expanded
|
|
the rule is equivalent to "foo[A-Z][A-Z0-9]*?" and the precedence
|
|
is such that the '?' is associated with "[A-Z0-9]*". With flex,
|
|
the rule will be expanded to "foo([A-Z][A-Z0-9]*)?" and so the
|
|
string "foo" will match. Note that because of this, the ^, $, <s>,
|
|
/, and <<EOF>> operators cannot be used in a flex definition.
|
|
|
|
The POSIX draft interpretation is the same as flex's.
|
|
|
|
- To specify a character class which matches anything but a left
|
|
bracket (']'), in lex one can use "[^]]" but with flex one must use
|
|
"[^\]]". The latter works with lex, too.
|
|
|
|
- The lex %r (generate a Ratfor scanner) option is not supported. It
|
|
is not part of the POSIX draft.
|
|
|
|
- If you are providing your own yywrap() routine, you must include a
|
|
"#undef yywrap" in the definitions section (section 1). Note that
|
|
the "#undef" will have to be enclosed in %{}'s.
|
|
|
|
The POSIX draft specifies that yywrap() is a function and this is
|
|
very unlikely to change; so flex users are warned that yywrap() is
|
|
likely to be changed to a function in the near future.
|
|
|
|
28 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
- After a call to unput(), yytext and yyleng are undefined until the
|
|
next token is matched. This is not the case with lex or the
|
|
present POSIX draft.
|
|
|
|
- The precedence of the {} (numeric range) operator is different.
|
|
lex interprets "abc{1,3}" as "match one, two, or three occurrences
|
|
of 'abc'", whereas flex interprets it as "match 'ab' followed by
|
|
one, two, or three occurrences of 'c'". The latter is in agreement
|
|
with the current POSIX draft.
|
|
|
|
- The precedence of the ^ operator is different. lex interprets
|
|
"^foo|bar" as "match either 'foo' at the beginning of a line, or
|
|
'bar' anywhere", whereas flex interprets it as "match either 'foo'
|
|
or 'bar' if they come at the beginning of a line". The latter is
|
|
in agreement with the current POSIX draft.
|
|
|
|
- To refer to yytext outside of the scanner source file, the correct
|
|
definition with flex is "extern char *yytext" rather than "extern
|
|
char yytext[]". This is contrary to the current POSIX draft but a
|
|
point on which flex will not be changing, as the array representa-
|
|
tion entails a serious performance penalty. It is hoped that the
|
|
POSIX draft will be emended to support the flex variety of declara-
|
|
tion (as this is a fairly painless change to require of lex users).
|
|
|
|
- yyin is initialized by lex to be stdin; flex, on the other hand,
|
|
initializes yyin to NULL and then assigns it to stdin the first
|
|
time the scanner is called, providing yyin has not already been
|
|
assigned to a non-NULL value. The difference is subtle, but the
|
|
net effect is that with flex scanners, yyin does not have a valid
|
|
value until the scanner has been called.
|
|
|
|
- The special table-size declarations such as %a supported by lex are
|
|
not required by flex scanners; flex ignores them.
|
|
|
|
- The name FLEX_SCANNER is #define'd so scanners may be written for
|
|
use with either flex or lex.
|
|
|
|
The following flex features are not included in lex or the POSIX draft
|
|
standard:
|
|
|
|
yyterminate()
|
|
<<EOF>>
|
|
YY_DECL
|
|
#line directives
|
|
%{}'s around actions
|
|
yyrestart()
|
|
comments beginning with '#' (deprecated)
|
|
multiple actions on a line
|
|
|
|
This last feature refers to the fact that with flex you can put multiple
|
|
actions on the same line, separated with semi-colons, while with lex,
|
|
the following
|
|
|
|
foo handle_foo(); ++num_foos_seen;
|
|
|
|
Version 2.3 29
|
|
|
|
FLEX(1) 26 May 1990
|
|
|
|
is (rather surprisingly) truncated to
|
|
|
|
foo handle_foo();
|
|
|
|
flex does not truncate the action. Actions that are not enclosed in
|
|
braces are simply terminated at the end of the line.
|
|
|
|
DIAGNOSTICS
|
|
reject_used_but_not_detected undefined or yymore_used_but_not_detected
|
|
undefined - These errors can occur at compile time. They indicate that
|
|
the scanner uses REJECT or yymore() but that flex failed to notice the
|
|
fact, meaning that flex scanned the first two sections looking for
|
|
occurrences of these actions and failed to find any, but somehow you
|
|
snuck some in (via a #include file, for example). Make an explicit
|
|
reference to the action in your flex input file. (Note that previously
|
|
flex supported a %used/%unused mechanism for dealing with this problem;
|
|
this feature is still supported but now deprecated, and will go away
|
|
soon unless the author hears from people who can argue compellingly that
|
|
they need it.)
|
|
|
|
flex scanner jammed - a scanner compiled with -s has encountered an
|
|
input string which wasn't matched by any of its rules.
|
|
|
|
flex input buffer overflowed - a scanner rule matched a string long
|
|
enough to overflow the scanner's internal input buffer (16K bytes by
|
|
default - controlled by YY_BUF_SIZE in "flex.skel". Note that to rede-
|
|
fine this macro, you must first #undefine it).
|
|
|
|
scanner requires -8 flag - Your scanner specification includes recogniz-
|
|
ing 8-bit characters and you did not specify the -8 flag (and your site
|
|
has not installed flex with -8 as the default).
|
|
|
|
fatal flex scanner internal error--end of buffer missed - This can occur
|
|
in an scanner which is reentered after a long-jump has jumped out (or
|
|
over) the scanner's activation frame. Before reentering the scanner,
|
|
use:
|
|
|
|
yyrestart( yyin );
|
|
|
|
too many %t classes! - You managed to put every single character into
|
|
its own %t class. flex requires that at least one of the classes share
|
|
characters.
|
|
|
|
DEFICIENCIES / BUGS
|
|
See flex(1).
|
|
|
|
SEE ALSO
|
|
|
|
flex(1), lex(1), yacc(1), sed(1), awk(1).
|
|
|
|
M. E. Lesk and E. Schmidt, LEX - Lexical Analyzer Generator
|
|
|
|
30 Version 2.3
|
|
|
|
26 May 1990 FLEX(1)
|
|
|
|
AUTHOR
|
|
Vern Paxson, with the help of many ideas and much inspiration from Van
|
|
Jacobson. Original version by Jef Poskanzer. The fast table represen-
|
|
tation is a partial implementation of a design done by Van Jacobson.
|
|
The implementation was done by Kevin Gong and Vern Paxson.
|
|
|
|
Thanks to the many flex beta-testers, feedbackers, and contributors,
|
|
especially Casey Leedom, benson@odi.com, Keith Bostic, Frederic Brehm,
|
|
Nick Christopher, Jason Coughlin, Scott David Daniels, Leo Eskin, Chris
|
|
Faylor, Eric Goldman, Eric Hughes, Jeffrey R. Jones, Kevin B. Kenny,
|
|
Ronald Lamprecht, Greg Lee, Craig Leres, Mohamed el Lozy, Jim Meyering,
|
|
Marc Nozell, Esmond Pitt, Jef Poskanzer, Jim Roskind, Dave Tallman,
|
|
Frank Whaley, Ken Yap, and those whose names have slipped my marginal
|
|
mail-archiving skills but whose contributions are appreciated all the
|
|
same.
|
|
|
|
Thanks to Keith Bostic, John Gilmore, Craig Leres, Bob Mulcahy, Rich
|
|
Salz, and Richard Stallman for help with various distribution headaches.
|
|
|
|
Thanks to Esmond Pitt and Earle Horton for 8-bit character support; to
|
|
Benson Margulies and Fred Burke for C++ support; to Ove Ewerlid for the
|
|
basics of support for NUL's; and to Eric Hughes for the basics of sup-
|
|
port for multiple buffers.
|
|
|
|
Work is being done on extending flex to generate scanners in which the
|
|
state machine is directly represented in C code rather than tables.
|
|
These scanners may well be substantially faster than those generated
|
|
using -f or -F. If you are working in this area and are interested in
|
|
comparing notes and seeing whether redundant work can be avoided, con-
|
|
tact Ove Ewerlid (ewerlid@mizar.DoCS.UU.SE).
|
|
|
|
This work was primarily done when I was at the Real Time Systems Group
|
|
at the Lawrence Berkeley Laboratory in Berkeley, CA. Many thanks to all
|
|
there for the support I received.
|
|
|
|
Send comments to:
|
|
|
|
Vern Paxson
|
|
Computer Science Department
|
|
4126 Upson Hall
|
|
Cornell University
|
|
Ithaca, NY 14853-7501
|
|
|
|
vern@cs.cornell.edu
|
|
decvax!cornell!vern
|
|
|
|
Version 2.3 31
|
|
|
|
99
|