<title=Token Conversion Table>
<keywords=token tokens language conversion table convert #convert lexical>

<link=What is a token?=What is a token?>

The token conversion table is used to convert a token into another. The
basic application is converting identifiers into keywords (eg. when the
scanner finds the 'for' identifier into source program, it converts it
to the 'for_k' keyword).

The conversion contains 4 fields : first is the input
token, the second is the string associated to the 'input' token, the third
is the 'output' token and the fourth is the string associated with the output
token.

When the scanner finds a token, it searches through the conversion table.
If it finds an 'input' token into the table which matches the current token,
it replaces it with the 'output' token from the conversion table.

Example :

  #convert           ;the conversion table starts here

  id for for_k 0     ;input token : id, string associated : 'for'
                     ;ouput token : for_k, no string associated

  id xor xor 0       ;input token : id, string associated : 'xor'
                     ;output token : xor, no string associated

  id int long_k int  ;input token : id, string associated : 'int'
                     ;output token : long_k, string associated : 'int'

  id John const 8678 ;input token : id, string associated : 'John'
                     ;output token : const,string associated : '8678'

  const 086 id Joe   ;input token : const, string associated : '086'
                     ;output token : id, string associated : Joe



<title=What is a token?>
<keywords=token description tokens scanner parser language>

A token is the basic element used in compiling, like a character is the
basic element used in writting. The compiler does not operate with
characters, it operates with tokens.

The scanner transforms the character from the source program into tokens
and send them forward to the compiling process.

Every token has an associated string, which can be empty. This string
contains additional information (like the characters of the identifier,when
the token is an identifier, or the digits when the token is a constant)

See also the <link=Token List=token list>.

<title=Token List>
<keywords=token tokens list language lng ldf>

See also :
  <link=What is a token?=What is a token?>
  <link=Token Conversion Table=Token Conversion Table>
  <link=Keyword List=Keyword List>
  <link=Compiler Directives=Compiler Directives>

   Name                              Description

EndofProgram                     End of program (end of file)
id                               identifier
constdec                         decimal constant
consthex                         hexadecimal constant
constbin                         binar constant
constreal                        real constant
semicolon                        ;
add                              + operator
sub                              - operator
mul                              * operator
div                              /
mod                              %
lp                               (
rp                               )
begin                            start of instruction block
end                              end of instructions block
option                           start of directive #
comma                            ,
point                            .
left_index                       [
right_index                      ]
apostrophe                       '
quote                            "
log_and                          && logical and
log_or                           || logical or
log_xor                          ^^ logical xor
log_not                          !  logical not
log_eq                           == logical equal
and                              & bit and
or                               | bit or
xor                              ^ bit xor
not                              ~ bit not
eq                               =
not_eq                           != <>
shift_left                       <<
shift_right                      >>
lteq                             <=
gteq                             >=
lt                               <
gt                               >
string                           "string"
if_k                             if keyword
while_k                          while keyword
do_k                             do keyword
then_k                           then keyword
else_k                           else keyword
switch_k                         switch keyword
case_k                           case keyword
default_k                        default keyword
continue_k                       continue keyword
break_k                          break keyword
return_k                         return keyword
goto_k                           goto keyword
struct_k                         struct keyword (obsolete)
for_k                            for keyword
cfor_k                           C for keyword
pasfor_k                         Pascal for keyword (Pascal compatibility)
loop_k                           loop keyword
qfor_k                           Quick for keyword
bfor_k                           Basic for keyword
sizeof_k                         sizeof keyword
char_k                           char keyword
byte_k                           byte keyword
int_k                            int keyword
word_k                           word keyword
long_k                           long keyword
dword_k                          dword keyword
function_k                       function keyword  (Pascal compatibility)
procedure_k                      procedure keyword (Pascal compatibility)
type_k                           type keyword (Pascal compatibility)
var_k                            var keyword (Pascal compatibility)
pointerto                        ->
pointer                          *  (obsolete)
pointer2                         ** (obsolete)
void_k                           void keyword
asm_k
colon                            :
point2                           ..
point3                           ...
label_k                          label keyword
include                          include directive
float_k                          float keyword
inc                              ++
dec                              --
addeq                            +=
subeq                            -=
muleq                            *=
diveq                            /=
modeq                            %=
xoreq                            ^=
oreq                             |=
andeq                            &=
shift_left_eq                    <<=
shift_right_eq                   >>=
xchg                             <->
char                             character  ' '
rol                              <<<
ror                              >>>
roleq                            <<<=
roreq                            >>>=
startcode_k                      startcode keyword
endcode_k                        endcode keyword
of_k                             of keyword (Pascal compatibility)
to_k                             to keyword
step_k                           step keyword (Basic compatibility)
link                             link directive
pascal                           obsolete
maxlomem                         maxlomem directive
private_k                        private keyword
external_k                       extern keyword
minlomem                         minlomem directive
maxhimem                         maxhimem directive
minhimem                         minhimem directive
define                           define directive
undef                            undef directive
ifdef                            ifdef directive
ifndef                           ifndef directive
endif                            endif directive
elseif                           else directive
interrupt_k                      interrupt keyword
pascal_k                         pascal keyword - Pascal declaration
irq_k                            irq keyword
cdecl_k                          cdecl keyword - C declaration
push_k                           push keyword
pop_k                            pop keyword
inline_k                         inline keyword
public_k                         public keyword
far_k                            far keyword
linkwith                         linkwith directive
align                            align directive
resolution                       :: resolution operator
with_k                           with keyword