Character sets Comments and ignorable characters
12.3.1 Character sets
The CharacterSets component allows for the declaration of names for character sets like letters or digits, and defines the characters that may occur as members of these sets. These names may then be used in the other sections of the scanner specification but not, it should be noted, in the parser specification. CharacterSets = CHARACTERS { NamedCharSet } . NamedCharSet = SetIdent = CharacterSet . . CharacterSet = SimpleSet { + | - SimpleSet } . SimpleSet = SetIdent | string | SingleChar [ .. SingleChar ] | ANY . SingleChar = CHR number . SetIdent = identifier . Simple character sets are denoted by one of SetIdent a previously declared character set with that name String a set consisting of all characters in the string CHRi a set of one character with ordinal value i CHRi .. CHRj a set consisting of all characters whose ordinal values are in the range i ... j. ANY the set of all characters acceptable to the implementation Simple sets may then be combined by the union + and difference operators - . As examples we might have digit = 0123456789 . The set of all digits hexdigit = digit + ABCDEF . The set of all hexadecimal digits eol = CHR10 . Line feed character noDigit = ANY - digit . Any character that is not a digit ctrlChars = CHR1 .. CHR31 . The ASCII control characters InString = ANY - ’’ - eol . Strings may not cross line boundaries12.3.2 Comments and ignorable characters
Usually spaces within the source text of a program are irrelevant, and in scanning for the start of a token, a CocoR generated scanner will simply ignore them. Other separators like tabs, line ends, and form feeds may also be declared irrelevant, and some applications may prefer to ignore the distinction between upper and lower case input. Comments are difficult to specify with the regular expressions used to denote tokens - indeed, nested comments may not be specified at all in this way. Since comments are usually discarded by a parsing process, and may typically appear in arbitrary places in source code, it makes sense to have a special construct to express their structure. Ignorable aspects of the scanning process are defined in Cocol by Comments = COMMENTS FROM TokenExpr TO TokenExpr [ NESTED ] . Ignorable = IGNORE CASE | CharacterSet . where the optional keyword NESTED should have an obvious meaning. A practical restriction is that comment brackets must not be longer than 2 characters. It is possible to declare several kinds of comments within a single grammar, for example, for C ++ : COMMENTS FROM TO COMMENTS FROM TO eol IGNORE CHR9 .. CHR13 The set of ignorable characters in this example is that which includes the standard white space separators in ASCII files. The null character CHR0 should not be included in any ignorable set. It is used internally by CocoR to mark the end of the input file.12.3.3 Tokens
Parts
» compilers and compiler generators 1996
» Objectives compilers and compiler generators 1996
» Systems programs and translators
» The relationship between high-level languages and translators
» T-diagrams Classes of translator
» Phases in translation compilers and compiler generators 1996
» Multi-stage translators compilers and compiler generators 1996
» Interpreters, interpretive compilers, and emulators
» Using a high-level host language Porting a high level translator
» Bootstrapping Self-compiling compilers compilers and compiler generators 1996
» The half bootstrap compilers and compiler generators 1996
» Bootstrapping from a portable interpretive compiler A P-code assembler
» Simple machine architecture compilers and compiler generators 1996
» Addressing modes compilers and compiler generators 1996
» Machine architecture Case study 1 - A single-accumulator machine
» Instruction set Case study 1 - A single-accumulator machine
» A specimen program Case study 1 - A single-accumulator machine
» An emulator for the single-accumulator machine
» A minimal assembler for the machine
» Machine architecture Case study 2 - a stack-oriented computer
» Instruction set Case study 2 - a stack-oriented computer
» Specimen programs Case study 2 - a stack-oriented computer
» An emulator for the stack machine
» Syntax, semantics, and pragmatics
» Languages, symbols, alphabets and strings Regular expressions
» Grammars and productions compilers and compiler generators 1996
» Classic BNF notation for productions
» Simple examples compilers and compiler generators 1996
» Phrase structure and lexical structure -productions
» Semantic overtones The British Standard for EBNF Lexical and phrase structure emphasis
» One- and two-pass assemblers, and symbol tables
» Source handling Towards the construction of an assembler
» Lexical analysis Towards the construction of an assembler
» Syntax analysis Towards the construction of an assembler
» Symbol table structures The first pass - static semantic analysis
» The second pass - code generation
» Symbol table structures The first pass - analysis and code generation
» Error detection compilers and compiler generators 1996
» Simple expressions as addresses
» Improved symbol table handling - hash tables
» Macro processing facilities compilers and compiler generators 1996
» Conditional assembly compilers and compiler generators 1996
» Relocatable code compilers and compiler generators 1996
» Further projects compilers and compiler generators 1996
» Equivalent grammars Case study - equivalent grammars for describing expressions
» Useless productions and reduced grammars -free grammars
» Ambiguous grammars compilers and compiler generators 1996
» Type 1 Grammars Context-sensitive The Chomsky hierarchy
» The relationship between grammar type and language type
» EBNF description of Clang A sample program
» Context sensitivity Deterministic top-down parsing
» Terminal successors, the FOLLOW function, and LL1 conditions for non -free grammars
» Further observations Restrictions on grammars so as to allow LL1 parsing
» Alternative formulations of the LL1 conditions
» The effect of the LL1 conditions on language design
» Construction of simple recursive descent parsers
» Case studies compilers and compiler generators 1996
» Syntax error detection and recovery
» Construction of simple scanners
» LR parsing compilers and compiler generators 1996
» Automated construction of scanners and parsers
» Embedding semantic actions into syntax rules
» Attribute grammars compilers and compiler generators 1996
» Synthesized and inherited attributes
» Classes of attribute grammars
» Case study - a small student database
» Installation Input file preparation
» Execution Output from CocoR Assembling the generated system
» Case study - a simple adding machine
» Character sets Comments and ignorable characters
» Pragmas User names The scanner interface
» Syntax error recovery Parser specification
» Grammar checks Semantic errors
» Interfacing to support modules The parser interface
» Essentials of the driver program Customizing the driver frame file
» Case study - Understanding C declarations
» Case study - Generating one-address code from expressions
» Case study - Generating one-address code from an AST
» Case study - How do parser generators work?
» Project suggestions compilers and compiler generators 1996
» Overall compiler structure compilers and compiler generators 1996
» A hand-crafted source handler
» A hand-crafted scanner Lexical analysis
» A CocoR generated scanner Efficient keyword recognition
» A hand-crafted parser A CocoR generated parser
» Syntax error handling in hand-crafted parsers
» Error reporting The symbol table handler
» Other aspects of symbol table management - further types
» The code generation interface
» Code generation for a simple stack machine
» Machine tyranny Other aspects of code generation
» Abstract syntax trees as intermediate representations
» Simple optimizations - constant folding
» Simple optimizations - removal of redundant code
» Generation of assembler code
» Source handling, lexical analysis and error reporting Syntax
» The scope and extent of identifiers
» Symbol table support for the scope rules
» Parsing declarations and procedure calls
» Hypothetical machine support for the static link model
» Code generation for the static link model
» Hypothetical machine support for the display model
» Code generation for the display model
» Relative merits of the static link and display models
» Syntax and semantics compilers and compiler generators 1996
» Symbol table support for context-sensitive features
» Actual parameters and stack frames
» Hypothetical stack machine support for parameter passing
» Semantic analysis and code generation in a hand-crafted compiler
» Semantic analysis and code generation in a CocoR generated compiler
» Scope rules Language design issues
» Function return mechanisms Language design issues
» Context sensitivity and LL1 conflict resolution Fundamental concepts
» Parallel processes, exclusion and synchronization
Show more