______________________________________________________________________

  2   Lexical conventions                                [lex]

  ______________________________________________________________________

1 A C++ program need not all be translated at the same time.   The  text
  of  the program is kept in units called source files in this standard.
  A source file together with all the headers (_lib.headers_) and source
  files   included   (_cpp.include_)  via  the  preprocessing  directive
  #include, less any source lines skipped  by  any  of  the  conditional
  inclusion  (_cpp.cond_) preprocessing directives, is called a transla­
  tion unit.  Previously translated translation units may  be  preserved
  individually  or  in  libraries.   The separate translation units of a
  program communicate (_basic.link_) by (for example) calls to functions
  whose identifiers have external linkage, manipulation of objects whose
  identifiers have external linkage,  or  manipulation  of  data  files.
  Translation  units  may be separately translated and then later linked
  to produce an executable program.  (_basic.link_).

  2.1  Phases of translation                                [lex.phases]

1 The precedence among the syntax rules of translation is  specified  by
  the following phases.1)

    1 Physical source file characters are mapped to the source character
      set  (introducing  new-line characters for end-of-line indicators)
      if necessary.  Trigraph sequences (_lex.trigraph_) are replaced by
      corresponding single-character internal representations.

    2 Each instance of a new-line character and an immediately preceding
      backslash character is deleted, splicing physical source lines  to
      form  logical source lines.  A source file that is not empty shall
      end in a new-line character, which shall not be  immediately  pre­
      ceded by a backslash character.

    3 The   source   file   is   decomposed  into  preprocessing  tokens
      (_lex.pptoken_) and sequences of white-space characters (including
      comments).  A source file shall not end in a partial preprocessing
      token or partial comment.  Each comment is replaced by  one  space
      character.    New-line  characters  are  retained.   Whether  each
      nonempty sequence of white-space characters other than new-line is
      retained  or  replaced  by  one space character is implementation-
      defined.  The process of dividing a source file's characters  into
      preprocessing  tokens  is context-dependent.  For example, see the
  _________________________
  1)  Implementations must behave as if these separate phases occur, al­
  though in practice different phases may be folded together.

      handling of < within a #include preprocessing directive.

    4 Preprocessing directives are executed and  macro  invocations  are
      expanded.   A  #include  preprocessing  directive causes the named
      header or source file to be processed from phase 1  through  phase
      4, recursively.

    5 Each  source character set member and escape sequence in character
      constants and string literals is converted to a member of the exe­
      cution character set.

    6 Adjacent  character  string  literal  tokens  are concatenated and
      adjacent wide string literal tokens are concatenated.

    7 White-space characters separating tokens are  no  longer  signifi­
      cant.   Each  preprocessing token is converted into a token.  (See
      _lex.digraph_).  The resulting tokens are syntactically and seman­
      tically  analyzed  and  translated.   The  result  of this process
      starting from a single source file is called a translation unit.

    8 The translation units that will form a program are combined.   All
      external object and function references are resolved.

  +-------                 BEGIN BOX 1                -------+
    What about shared libraries?
  +-------                  END BOX 1                 -------+

    Library  components  are  linked  to  satisfy external references to
    functions and objects not defined in the current  translation.   All
    such  translator output is collected into a program image which con­
    tains information needed for execution in its execution environment.

  2.2  Trigraph sequences                                 [lex.trigraph]

1 Before any other processing takes place, each occurrence of one of the
  following  sequences  of  three  characters  (trigraph  sequences)  is
  replaced by the single character indicated in Table 1.

                       Table 1--trigraph sequences

  +-----------------------+------------------------+------------------------+
  |trigraph   replacement | trigraph   replacement | trigraph   replacement |
  +-----------------------+------------------------+------------------------+
  |  ??=           #      |   ??(           [      |   ??<           {      |
  +-----------------------+------------------------+------------------------+
  |  ??/           \      |   ??)           ]      |   ??>           }      |
  +-----------------------+------------------------+------------------------+
  |  ??'           ^      |   ??!           |      |   ??-           ~      |
  +-----------------------+------------------------+------------------------+

2 For example,
          ??=define arraycheck(a,b) a??(b??) ??!??! b??(a??)
  becomes
          #define arraycheck(a,b) a[b] || b[a]

  2.3  Preprocessing tokens                                [lex.pptoken]
          preprocessing-token:
                  header-name
                  identifier
                  pp-number
                  character-constant
                  string-literal
                  operator
                  digraph
                  punctuator
                  each non-white-space character that cannot be one of the above

1 Each  preprocessing  token  that is converted to a token (_lex.token_)
  shall have the lexical form of a keyword, an identifier, a constant, a
  string literal, an operator, a digraph, or a punctuator.

2 A  preprocessing  token is the minimal lexical element of the language
  in translation phases 3 through 6.  The  categories  of  preprocessing
  token are: header names, identifiers, preprocessing numbers, character
  constants, string literals, operators, punctuators, digraphs, and sin­
  gle  non-white-space  characters that do not lexically match the other
  preprocessing token categories.  If a ' or a " character  matches  the
  last category, the behavior is undefined.  Preprocessing tokens can be
  separated by white space; this consists of  comments  (_lex.comment_),
  or  white-space  characters (space, horizontal tab, new-line, vertical
  tab, and form-feed), or both.  As described in Clause _cpp_,  in  cer­
  tain  circumstances  during  translation  phase 4, white space (or the
  absence thereof) serves as more than preprocessing  token  separation.
  White  space may appear within a preprocessing token only as part of a
  header name or between the quotation characters in  a  character  con­
  stant or string literal.

3 If  the input stream has been parsed into preprocessing tokens up to a
  given character, the next preprocessing token is the longest  sequence
  of characters that could constitute a preprocessing token.

4 The  program  fragment  1Ex  is parsed as a preprocessing number token
  (one that is not a valid floating or  integer  constant  token),  even
  though a parse as the pair of preprocessing tokens 1 and Ex might pro­
  duce a valid expression (for example, if Ex were a  macro  defined  as
  +1).  Similarly, the program fragment 1E1 is parsed as a preprocessing
  number (one that is a valid floating constant token), whether or not E
  is a macro name.

5 The program fragment x+++++y is parsed as x ++ ++ + y, which, if x and
  y are of built-in types, violates a constraint on increment operators,
  even though the parse x ++ + ++ y might yield a correct expression.

  2.4  Digraph sequences                                   [lex.digraph]

1 Alternate  representations are provided for the operators and punctua­
  tors whose primary representations use the national characters.  These
  include digraphs and additional reserved words.
          digraph:
                  <%
                  %>
                  <:
                  :>
                  %:

2 In  translation  phase 3 (_lex.phases_) the digraphs are recognized as
  preprocessing tokens.  Then in translation phase 7  the  digraphs  and
  the  additional  identifiers  listed  below  are converted into tokens
  identical to those from the corresponding primary representations,  as
  shown in Table 2.

            Table 2--identifiers that are treated as operators

    +--------------------+---------------------+---------------------+
    |alternate   primary | alternate   primary | alternate   primary |
    +--------------------+---------------------+---------------------+
    |   <%          {    |    and        &&    |  and_eq       &=    |
    +--------------------+---------------------+---------------------+
    |   %>          }    |   bitor        |    |   or_eq       |=    |
    +--------------------+---------------------+---------------------+
    |   <:          [    |    or         ||    |  xor_eq       ^=    |
    +--------------------+---------------------+---------------------+
    |   :>          ]    |    xor         ^    |    not         !    |
    +--------------------+---------------------+---------------------+
    |   %:          #    |   compl        ~    |  not_eq       !=    |
    +--------------------+---------------------+---------------------+
    | bitand        &    |                     |                     |
    +--------------------+---------------------+---------------------+

  2.5  Tokens                                                [lex.token]
          token:
                  identifier
                  keyword
                  literal
                  operator
                  punctuator

1 There are five kinds of tokens: identifiers, keywords, literals (which
  include strings and character and numeric constants),  operators,  and
  other  separators.   Blanks,  horizontal  and vertical tabs, newlines,
  formfeeds, and comments  (collectively,  white  space),  as  described
  below,  are  ignored  except  as  they serve to separate tokens.  Some
  white space is required to separate  otherwise  adjacent  identifiers,
  keywords, and literals.

2 If  the input stream has been parsed into tokens up to a given prepro­
  cessing token, the next token is taken to be  the  longest  string  of
  preprocessing tokens that could possibly constitute a token.

  2.6  Comments                                            [lex.comment]

1 The  characters  /* start a comment, which terminates with the charac­
  ters */.  These comments do not nest.  The characters // start a  com­
  ment, which terminates with the next new-line character. If there is a
  form-feed or a vertical-tab character in such a comment,  only  white-
  space  characters  may  appear between it and the new-line that termi­
  nates the comment; no diagnostic is required.  The comment  characters
  //,  /*,  and  */  have no special meaning within a // comment and are
  treated just like other characters.  Similarly, the comment characters
  // and /* have no special meaning within a /* comment.

  2.7  Identifiers                                            [lex.name]
          identifier:
                  nondigit
                  identifier nondigit
                  identifier digit
          nondigit: one of
                  _ a b c d e f g h i j k l m
                    n o p q r s t u v w x y z
                    A B C D E F G H I J K L M
                    N O P Q R S T U V W X Y Z
          digit: one of
                  0 1 2 3 4 5 6 7 8 9

1 An  identifier  is an arbitrarily long sequence of letters and digits.
  The first character must be a letter; the underscore  _  counts  as  a
  letter.   Upper- and lower-case letters are different.  All characters
  are significant.

  2.8  Keywords                                                [lex.key]

1 The identifiers shown in Table 3 are reserved for use as keywords, and
  may not be used otherwise in phases 7 and 8:

                            Table 3--keywords

  +------------------------------------------------------------------------+
  |asm          delete         if          reinterpret_cast   true         |
  |auto         do             inline      return             try          |
  |bool         double         int         short              typedef      |
  |break        dynamic_cast   long        signed             typeid       |
  |case         else           mutable     sizeof             union        |
  |catch        enum           namespace   static             unsigned     |
  |char         extern         new         static_cast        using        |
  |class        false          operator    struct             virtual      |
  |const        float          private     switch             void         |
  |const_cast   for            protected   template           volatile     |
  |continue     friend         public      this               wchar_t      |
  |default      goto           register    throw              while        |
  +------------------------------------------------------------------------+

2 Furthermore,  the  alternate representations shown in Table 4 for cer­
  tain operators and punctuators (_lex.digraph_) are  reserved  and  may
  not be used otherwise:

                    Table 4--alternate representations

             +-----------------------------------------------+
             |bitand   and     bitor    or    xor      compl |
             |and_eq   or_eq   xor_eq   not   not_eq         |
             +-----------------------------------------------+

3 In addition, identifiers containing a double underscore (__) or begin­
  ning with an underscore and an upper-case letter are reserved for  use
  by C++ implementations and standard libraries and should be avoided by
  users; no diagnostic is required.

4 The ASCII representation of C++ programs  uses  as  operators  or  for
  punctuation the characters shown in Table 5.

              Table 5--operators and punctuation characters

         +------------------------------------------------------+
         |!   %   ^   &   *   (   )   -   +   --  {   }   |   ~ |
         |[   ]   \   ;   '   :   "   <   >   ?   ,   .   /     |
         +------------------------------------------------------+
  Table  6 shows the character combinationations that are used as opera­
  tors.

            Table 6--character combinations used as operators

      +-------------------------------------------------------------+
      |->   ++   --   .*   ->*   <<   >>    <=    >=   ==   !=   && |
      |||   *=   /=   %=   +=    -=   <<=   >>=   &=   ^=   |=   :: |
      +-------------------------------------------------------------+
  Each  is  converted  to  a  single  token  in  translation   phase   7
  (_lex.phases_).

5 Table 7 shows character combinations that are used as alternative rep­
  resentations for certain operators and punctuators (_lex.digraph_).

                            Table 7--digraphs

                         +-----------------------+
                         |<%   %>   <:   :>   %: |
                         +-----------------------+
  Each of these is also recognized as  a  single  token  in  translation
  phases 3 and 7.

6 Table 8 shows additional tokens that are used by the preprocessor.

                      Table 8--preprocessing tokens

                       +---------------------------+
                       |#   ##   %:   %:%:         |
                       +---------------------------+

7 Certain  implementation-dependent  properties,  such  as the type of a
  sizeof  (_expr.sizeof_)  and   the   ranges   of   fundamental   types
  (_basic.fundamental_),  are  defined  in  the  standard  header  files
  (_cpp.include_)
          <float.h>   <limits.h>   <stddef.h>
  These headers are part of the ISO C standard.  In addition the headers
          <new.h>   <stdarg.h>   <stdlib.h>
  define  the  types  of the most basic library functions.  The last two
  headers are part of the ISO C standard; <new.h> is C++ specific.

  2.9  Literals                                            [lex.literal]

1 There are several kinds of literals (often referred to as  constants).
          literal:
                  integer-literal
                  character-literal
                  floating-literal
                  string-literal
                  boolean-literal

  2.9.1  Integer literals                                     [lex.icon]
          integer-literal:
                  decimal-literal integer-suffixopt
                  octal-literal integer-suffixopt
                  hexadecimal-literal integer-suffixopt
          decimal-literal:
                  nonzero-digit
                  decimal-literal digit
          octal-literal:
                  0
                  octal-literal octal-digit
          hexadecimal-literal:
                  0x hexadecimal-digit
                  0X hexadecimal-digit
                  hexadecimal-literal hexadecimal-digit
          nonzero-digit: one of
                  1  2  3  4  5  6  7  8  9
          octal-digit: one of
                  0  1  2  3  4  5  6  7
          hexadecimal-digit: one of
                  0  1  2  3  4  5  6  7  8  9
                  a  b  c  d  e  f
                  A  B  C  D  E  F
          integer-suffix:
                  unsigned-suffix long-suffixopt
                  long-suffix unsigned-suffixopt
          unsigned-suffix: one of
                  u  U
          long-suffix: one of
                  l  L

1 An  integer  literal consisting of a sequence of digits is taken to be
  decimal (base ten) unless it begins with 0 (digit zero).   A  sequence
  of  digits  starting  with  0  is  taken  to be an octal integer (base
  eight).  The digits 8 and 9 are not octal digits.  A sequence of  dig­
  its  preceded  by  0x or 0X is taken to be a hexadecimal integer (base
  sixteen).  The hexadecimal digits include a or A through f or  F  with
  decimal  values  ten  through fifteen.  For example, the number twelve
  can be written 12, 014, or 0XC.

2 The type of an integer literal depends on its form, value, and suffix.
  If it is decimal and has no suffix, it has the first of these types in
  which its value can be represented: int, long int, unsigned long  int.
  If  it  is octal or hexadecimal and has no suffix, it has the first of
  these types in which its value can be represented: int, unsigned  int,
  long int, unsigned long int.  If it is suffixed by u or U, its type is
  the first of these types  in  which  its  value  can  be  represented:
  unsigned  int,  unsigned  long  int.  If it is suffixed by l or L, its
  type is the first of these types in which  its  value  can  be  repre­
  sented: long int, unsigned long int.  If it is suffixed by ul, lu, uL,
  Lu, Ul, lU, UL, or LU, its type is unsigned long int.

3 A program is ill-formed if it contains an integer literal that  cannot
  be represented by any of the allowed types.

  2.9.2  Character literals                                   [lex.ccon]
          character-literal:
                  'c-char-sequence'
                  L'c-char-sequence'
          c-char-sequence:
                  c-char
                  c-char-sequence c-char
          c-char:
                  any member of the source character set except
                          the single-quote ', backslash \, or new-line character
                  escape-sequence
          escape-sequence:
                  simple-escape-sequence
                  octal-escape-sequence
                  hexadecimal-escape-sequence
          simple-escape-sequence: one of
                  \'  \"  \?  \\
                  \a  \b  \f  \n  \r  \t  \v
          octal-escape-sequence:
                  \ octal-digit
                  octal-escape-sequence octal-digit
          hexadecimal-escape-sequence:
                  \x hexadecimal-digit
                  hexadecimal-escape-sequence hexadecimal-digit

1 A  character  literal  is  one  or  more characters enclosed in single
  quotes, as in 'x', optionally preceded by the letter L,  as  in  L'x'.
  Single  character  literals  that  do not begin with L have type char,
  with value equal to the  numerical  value  of  the  character  in  the
  machine's  character  set.   Multicharacter literals that do not begin
  with L have type int and implementation-defined value.

2 A character literal that begins with the letter L, such as L'ab', is a
  wide-character  literal.   Wide-character  literals have type wchar_t.
  They are intended for character sets where a character  does  not  fit
  into  a  single  byte.   Wide-character  literals have implementation-
  defined values, regardless of the number of characters in the literal.

3 Certain nongraphic characters, the single quote ', the double quote ",
  ?, and the backslash \, may be represented according to Table 9.

                        Table 9--escape sequences

                   +----------------------------------+
                   |new-line          NL (LF)   \n    |
                   |horizontal tab    HT        \t    |
                   |vertical tab      VT        \v    |
                   |backspace         BS        \b    |
                   |carriage return   CR        \r    |
                   |form feed         FF        \f    |
                   |alert             BEL       \a    |
                   |backslash         \         \\    |
                   |question mark     ?         \?    |
                   |single quote      '         \'    |
                   |double quote      "         \"    |
                   |octal number      ooo       \ooo  |
                   |hex number        hhh       \xhhh |
                   +----------------------------------+
  If the character following a backslash is not one of those  specified,
  the  behavior  is  undefined.   An  escape sequence specifies a single
  character.

4 The escape \ooo consists of the backslash  followed  by  one  or  more
  octal  digits that are taken to specify the value of the desired char­
  acter.  The escape \xhhh consists of the backslash followed by x  fol­
  lowed  by one or more hexadecimal digits that are taken to specify the
  value of the desired character.  There is no limit to  the  number  of
  digits  in either sequence.  A sequence of octal or hexadecimal digits
  is terminated by the first character that is not an octal digit  or  a
  hexadecimal  digit, respectively.  The value of a character literal is
  implementation dependent if it exceeds that of the largest  char  (for
  ordinary literals) or wchar_t (for wide literals).

  2.9.3  Floating literals                                    [lex.fcon]
          floating-constant:
                  fractional-constant exponent-partopt floating-suffixopt
                  digit-sequence exponent-part floating-suffixopt
          fractional-constant:
                  digit-sequenceopt . digit-sequence
                  digit-sequence .
          exponent-part:
                  e signopt digit-sequence
                  E signopt digit-sequence
          sign: one of
                  +  -
          digit-sequence:
                  digit
                  digit-sequence digit
          floating-suffix: one of
                  f  l  F  L

1 A  floating  literal  consists  of an integer part, a decimal point, a
  fraction part, an e or E, an optionally signed integer  exponent,  and
  an  optional type suffix.  The integer and fraction parts both consist
  of a sequence of decimal (base ten) digits.  Either the  integer  part
  or  the  fraction  part  (not both) may be missing; either the decimal
  point or the letter e (or E) and the exponent (not both) may be  miss­
  ing.  The type of a floating literal is double unless explicitly spec­
  ified by a suffix.  The suffixes f and F specify float, the suffixes l
  and L specify long double.

  2.9.4  String literals                                    [lex.string]
          string-literal:
                  "s-char-sequenceopt"
                  L"s-char-sequenceopt"
          s-char-sequence:
                  s-char
                  s-char-sequence s-char
          s-char:
                  any member of the source character set except
                          the double-quote ", backslash \, or new-line character
                  escape-sequence

1 A   string  literal  is  a  sequence  of  characters  (as  defined  in
  _lex.ccon_) surrounded by double quotes, optionally beginning with the
  letter L, as in "..." or L"...".  A string literal that does not begin
  with  L  has  type  array  of  n  char  and  static  storage  duration
  (_basic.stc_), where n is the size of the string as defined below, and
  is initialized with the given characters.  Whether all string literals
  are distinct (that is, are stored in nonoverlapping objects) is imple­
  mentation dependent.  The effect of attempting to modify a string lit­
  eral is undefined.

2 A  string  literal  that  begins  with  L, such as L"asdf", is a wide-
  character string.  A wide-character string  is  of  type  array  of  n
  wchar_t, where n is the size of the string as defined below.  Concate­
  nation of ordinary and wide-character string literals is undefined.

  +-------                 BEGIN BOX 2                -------+
  Should this render the program  ill-formed?   Or  is  it  deliberately
  undefined to encourage extensions?
  +-------                  END BOX 2                 -------+

3 Adjacent string literals are concatenated.  Characters in concatenated
  strings are kept distinct.  For example,
          "\xA" "B"
  contains the two characters '\xA' and 'B' after concatenation (and not
  the single hexadecimal character '\xAB').

4 After  any  necessary  concatenation '\0' is appended so that programs
  that scan a string can find its end.  The size of a string is the num­
  ber of its characters including this terminator.  Within a string, the
  double quote character " must be preceded by a \.

5 Escape sequences in string literals have the same meaning as in  char­
  acter literals (_lex.ccon_).

  2.9.5  Boolean literals                                     [lex.bool]
          boolean-literal:
                  false
                  true

1 The  Boolean  literals are the keywords false and true.  Such literals
  have type bool and the given values.  They are not lvalues.