← Back
Editing: grammar.pyo
� { fc @ s� d Z d d l Z d d l Z d d l m Z m Z d e f d � � YZ d � Z d Z i Z xB e j � D]4 Z e rl e j � \ Z Z e e e � e e <ql ql Wd S( s� This module defines the data structures used to represent a grammar. These are a bit arcane because they are derived from the data structures used by Python's 'pgen' parser generator. There's also a table here mapping operators to their names in the token module; the Python tokenize module reports all operators as the fallback token code OP, but the parser needs the actual token code. i����Ni ( t tokent tokenizet Grammarc B sD e Z d Z d � Z d � Z d � Z d � Z d � Z d � Z RS( s� Pgen parsing tables conversion class. Once initialized, this class supplies the grammar tables for the parsing engine implemented by parse.py. The parsing engine accesses the instance variables directly. The class here does not provide initialization of the tables; several subclasses exist to do this (see the conv and pgen modules). The load() method reads the tables from a pickle file, which is much faster than the other ways offered by subclasses. The pickle file is written by calling dump() (after loading the grammar tables using a subclass). The report() method prints a readable representation of the tables to stdout, for debugging. The instance variables are as follows: symbol2number -- a dict mapping symbol names to numbers. Symbol numbers are always 256 or higher, to distinguish them from token numbers, which are between 0 and 255 (inclusive). number2symbol -- a dict mapping numbers to symbol names; these two are each other's inverse. states -- a list of DFAs, where each DFA is a list of states, each state is a list of arcs, and each arc is a (i, j) pair where i is a label and j is a state number. The DFA number is the index into this list. (This name is slightly confusing.) Final states are represented by a special arc of the form (0, j) where j is its own state number. dfas -- a dict mapping symbol numbers to (DFA, first) pairs, where DFA is an item from the states list above, and first is a set of tokens that can begin this grammar rule (represented by a dict whose values are always 1). labels -- a list of (x, y) pairs where x is either a token number or a symbol number, and y is either None or a string; the strings are keywords. The label number is the index in this list; label numbers are used to mark state transitions (arcs) in the DFAs. start -- the number of the grammar's start symbol. keywords -- a dict mapping keyword strings to arc labels. tokens -- a dict mapping token numbers to arc labels. c C sX i | _ i | _ g | _ i | _ d g | _ i | _ i | _ i | _ d | _ d S( Ni t EMPTYi ( i R ( t symbol2numbert number2symbolt statest dfast labelst keywordst tokenst symbol2labelt start( t self( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyt __init__M s c C s> t | d � �) } t | j � } t j | | d � Wd QXd S( s� Dump the grammar tables to a pickle file. dump() recursively changes all dict to OrderedDict, so the pickled file is not exactly the same as what was passed in to dump(). load() uses the pickled file to create the tables, but only changes OrderedDict to dict at the top level; it does not recursively change OrderedDict to dict. So, the loaded tables are different from the original tables that were passed to load() in that some of the OrderedDict (from the pickled file) are not changed back to dict. For parsing, this has no effect on performance because OrderedDict uses dict's __getitem__ with nothing in between. t wbi N( t opent _make_deterministict __dict__t picklet dump( R t filenamet ft d( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyR X s c C s< t | d � } t j | � } | j � | j j | � d S( s+ Load the grammar tables from a pickle file.t rbN( R R t loadt closeR t update( R R R R ( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyR i s c C s | j j t j | � � d S( s3 Load the grammar tables from a pickle bytes object.N( R R R t loads( R t pkl( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyR p s c C sf | j � } x- d D]% } t | | t | | � j � � q W| j | _ | j | _ | j | _ | S( s# Copy the grammar. R R R R R R ( R R R R R R ( t __class__t setattrt getattrt copyR R R ( R t newt dict_attr( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyR! t s # c C sz d d l m } d GH| | j � d GH| | j � d GH| | j � d GH| | j � d GH| | j � d G| j GHd S( s: Dump the grammar tables to standard output, for debugging.i����( t pprintt s2nt n2sR R R R N( R$ R R R R R R ( R R$ ( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyt report� s ( t __name__t __module__t __doc__R R R R R! R' ( ( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyR s 4 c C s� t | t � r2 t j t d � | j � D� � � St | t � r^ g | D] } t | � ^ qH St | t � r� t d � | D� � S| S( Nc s s' | ] \ } } | t | � f Vq d S( N( R ( t .0t kt v( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pys <genexpr>� s c s s | ] } t | � Vq d S( N( R ( R+ t e( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pys <genexpr>� s ( t isinstancet dictt collectionst OrderedDictt sortedt iteritemst listR t tuple( t topR. ( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyR � s s ( LPAR ) RPAR [ LSQB ] RSQB : COLON , COMMA ; SEMI + PLUS - MINUS * STAR / SLASH | VBAR & AMPER < LESS > GREATER = EQUAL . DOT % PERCENT ` BACKQUOTE { LBRACE } RBRACE @ AT @= ATEQUAL == EQEQUAL != NOTEQUAL <> NOTEQUAL <= LESSEQUAL >= GREATEREQUAL ~ TILDE ^ CIRCUMFLEX << LEFTSHIFT >> RIGHTSHIFT ** DOUBLESTAR += PLUSEQUAL -= MINEQUAL *= STAREQUAL /= SLASHEQUAL %= PERCENTEQUAL &= AMPEREQUAL |= VBAREQUAL ^= CIRCUMFLEXEQUAL <<= LEFTSHIFTEQUAL >>= RIGHTSHIFTEQUAL **= DOUBLESTAREQUAL // DOUBLESLASH //= DOUBLESLASHEQUAL -> RARROW ( R* R1 R t R R t objectR R t opmap_rawt opmapt splitlinest linet splitt opt nameR ( ( ( s- /usr/lib64/python2.7/lib2to3/pgen2/grammar.pyt <module> s z =
Save File
Cancel