← Back
Editing: lexer.cpython-38.pyc
U ʗRe} � @ s� d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m Z m Z m Z m Z ddlmZmZmZmZmZmZ ddlmZ dd d d d d ddddddg ZdddddgZedd� �ZG dd � d e�Z G d d� de d �Z G d d � d e �Z G d d � d e �Z G d!d"� d"�Z!e!� Z"G d#d$� d$e#�Z$G d%d&� d&�Z%d'd� Z&G d(d)� d)�Z'e'� Z(d*d� Z)G d+d� d�Z*G d,d� de�Z+G d-d.� d.e �Z,G d/d � d e e,d �Z-G d0d � d �Z.G d1d � d e-�Z/d2d3� Z0G d4d5� d5e,�Z1G d6d7� d7e-e1d �Z2dS )8z� pygments.lexer ~~~~~~~~~~~~~~ Base lexer classes. :copyright: Copyright 2006-2022 by the Pygments team, see AUTHORS. :license: BSD, see LICENSE for details. � N)� apply_filters�Filter)�get_filter_by_name)�Error�Text�Other� _TokenType)� get_bool_opt� get_int_opt� get_list_opt�make_analysator�Future� guess_decode)� regex_opt�Lexer� RegexLexer�ExtendedRegexLexer�DelegatingLexer� LexerContext�include�inherit�bygroups�using�this�default�words)s �utf-8)s �� zutf-32)s ��zutf-32be)s ��zutf-16)s ��zutf-16bec C s dS )N� � ��xr r ��/builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_vendor/pygments/lexer.py�<lambda>! � r" c @ s e Zd ZdZdd� ZdS )� LexerMetaz� This metaclass automagically converts ``analyse_text`` methods into static methods which always return float values. c C s( d|krt |d �|d< t�| |||�S )N� analyse_text)r �type�__new__)�mcs�name�bases�dr r r! r' * s zLexerMeta.__new__N)�__name__� __module__� __qualname__�__doc__r' r r r r! r$ $ s r$ c @ s^ e Zd ZdZdZdZg Zg Zg Zg Z dZ dd� Z dd� Z dd � Z d d � Zdd d�Zdd� ZdS )r a� Lexer for a specific language. Basic options recognized: ``stripnl`` Strip leading and trailing newlines from the input (default: True). ``stripall`` Strip all leading and trailing whitespace from the input (default: False). ``ensurenl`` Make sure that the input ends with a newline (default: True). This is required for some lexers that consume input linewise. .. versionadded:: 1.3 ``tabsize`` If given and greater than 0, expand tabs in the input (default: 0). ``encoding`` If given, must be an encoding name. This encoding will be used to convert the input string to Unicode, if it is not already a Unicode string (default: ``'guess'``, which uses a simple UTF-8 / Locale / Latin1 detection. Can also be ``'chardet'`` to use the chardet library, if it is installed. ``inencoding`` Overrides the ``encoding`` if given. Nr c K s� || _ t|dd�| _t|dd�| _t|dd�| _t|dd�| _|�dd �| _|�d �pZ| j| _g | _ t |d d �D ]}| � |� qpd S ) N�stripnlT�stripallF�ensurenl�tabsizer �encoding�guess� inencoding�filtersr ) �optionsr r0 r1 r2 r r3 �getr4 r7 r � add_filter)�selfr8 �filter_r r r! �__init__a s zLexer.__init__c C s( | j rd| jj| j f S d| jj S d S )Nz <pygments.lexers.%s with %r>z<pygments.lexers.%s>)r8 � __class__r, �r; r r r! �__repr__m s �zLexer.__repr__c K s& t |t�st|f|�}| j�|� dS )z8 Add a new stream filter to this lexer. N)� isinstancer r r7 �append)r; r< r8 r r r! r: t s zLexer.add_filterc C s dS )a~ Has to return a float between ``0`` and ``1`` that indicates if a lexer wants to highlight this text. Used by ``guess_lexer``. If this method returns ``0`` it won't highlight it in any case, if it returns ``1`` highlighting with this lexer is guaranteed. The `LexerMeta` metaclass automatically wraps this function so that it works like a static method (no ``self`` or ``cls`` parameter) and the return value is automatically converted to `float`. If the return value is an object that is boolean `False` it's the same as if the return values was ``0.0``. Nr )�textr r r! r% | s zLexer.analyse_textFc s� t �t��s� jdkr$t��\�}nވ jdkr�zddlm} W n, tk rj } ztd�|�W 5 d}~X Y nX d}tD ].\}}��|�rt�t |�d� � |d�} q�qt|dkr�|� �dd� �} �� | � d �p�d d�}|�n(�� � j����d ��r �t d �d� �n ��d ��r �t d �d� ��� d d ���� dd ��� j�rJ��� �n� j�r\��d ��� jdk�rt��� j��� j�r���d ��s��d 7 �� �fdd�} | � } |�s�t| � j� �} | S )a= Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters. r5 �chardetr )rD zkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/N�replacei r4 r u z � � c 3 s$ � � ��D ]\} }}||fV q d S �N)�get_tokens_unprocessed)�_�t�v�r; rC r r! �streamer� s z"Lexer.get_tokens.<locals>.streamer)rA �strr4 r � pip._vendorrD � ImportError� _encoding_map� startswith�len�decode�detectr9 rE r1 �stripr0 r3 � expandtabsr2 �endswithr r7 ) r; rC � unfilteredrJ rD �e�decoded�bomr4 �encrN �streamr rM r! � get_tokens� sR � � zLexer.get_tokensc C s t �dS )z� Return an iterable of (index, tokentype, value) pairs where "index" is the starting position of the token within the input text. In subclasses, implement this method as a generator to maximize effectiveness. N)�NotImplementedErrorrM r r r! rI � s z Lexer.get_tokens_unprocessed)F)r, r- r. r/ r) �url�aliases� filenames�alias_filenames� mimetypes�priorityr= r@ r: r% r` rI r r r r! r 0 s ;)� metaclassc @ s$ e Zd ZdZefdd�Zdd� ZdS )r a This lexer takes two lexer as arguments. A root lexer and a language lexer. First everything is scanned using the language lexer, afterwards all ``Other`` tokens are lexed using the root lexer. The lexers from the ``template`` lexer package use this base lexer. c K s0 |f |�| _ |f |�| _|| _tj| f|� d S rH )� root_lexer�language_lexer�needler r= )r; � _root_lexer�_language_lexer�_needler8 r r r! r= � s zDelegatingLexer.__init__c C s� d}g }g }| j �|�D ]H\}}}|| jkrP|rF|�t|�|f� g }||7 }q|�|||f� q|rx|�t|�|f� t|| j�|��S )N� )rj rI rk rB rT � do_insertionsri )r; rC �buffered� insertions� lng_buffer�irK rL r r r! rI � s �z&DelegatingLexer.get_tokens_unprocessedN)r, r- r. r/ r r= rI r r r r! r � s c @ s e Zd ZdZdS )r zI Indicates that a state should include rules from another state. N�r, r- r. r/ r r r r! r � s c @ s e Zd ZdZdd� ZdS )�_inheritzC Indicates the a state should inherit from its superclass. c C s dS )Nr r r? r r r! r@ s z_inherit.__repr__N)r, r- r. r/ r@ r r r r! rv � s rv c @ s e Zd ZdZdd� Zdd� ZdS )�combinedz: Indicates a state combined from multiple states. c G s t �| |�S rH )�tupler' )�cls�argsr r r! r' s zcombined.__new__c G s d S rH r )r; rz r r r! r= s zcombined.__init__N)r, r- r. r/ r' r= r r r r! rw s rw c @ sF e Zd ZdZdd� Zddd�Zddd�Zdd d �Zd d � Zd d� Z dS )� _PseudoMatchz: A pseudo match object constructed from a string. c C s || _ || _d S rH )�_text�_start)r; �startrC r r r! r= s z_PseudoMatch.__init__Nc C s | j S rH )r} �r; �argr r r! r~ s z_PseudoMatch.startc C s | j t| j� S rH )r} rT r| r r r r! �end! s z_PseudoMatch.endc C s |r t d��| jS )Nz No such group)� IndexErrorr| r r r r! �group$ s z_PseudoMatch.groupc C s | j fS rH )r| r? r r r! �groups) s z_PseudoMatch.groupsc C s i S rH r r? r r r! � groupdict, s z_PseudoMatch.groupdict)N)N)N) r, r- r. r/ r= r~ r� r� r� r� r r r r! r{ s r{ c s d� fdd� }|S )zL Callback that yields multiple actions for each group in the match. Nc 3 s� t � �D ]�\}}|d kr qqt|�tkrR|�|d �}|r�|�|d �||fV q|�|d �}|d k r|r||�|d �|_|| t|�|d �|�|�D ]}|r�|V q�q|r�|�� |_d S )N� )� enumerater& r r� r~ �posr{ r� )�lexer�match�ctxrt �action�data�item�rz r r! �callback4 s& � zbygroups.<locals>.callback)Nr )rz r� r r� r! r 0 s c @ s e Zd ZdZdS )�_ThiszX Special singleton used for indicating the caller class. Used by ``using``. Nru r r r r! r� J s r� c sj i �d�kr:�� d�}t|ttf�r.|�d< n d|f�d< � tkrTd��fdd� }nd � ��fdd� }|S ) a� Callback that processes the match with a different lexer. The keyword arguments are forwarded to the lexer, except `state` which is handled separately. `state` specifies the state that the new lexer will start in, and can be an enumerable such as ('root', 'inline', 'string') or a simple string which is assumed to be on top of the root state. Note: For that to work, `_other` must not be an `ExtendedRegexLexer`. �state�stack�rootNc 3 sj �r �� | j� | jf ��}n| }|�� }|j|�� f� �D ]\}}}|| ||fV q<|rf|�� |_d S rH )�updater8 r> r~ rI r� r� r� �r� r� r� �lx�srt rK rL )� gt_kwargs�kwargsr r! r� i s zusing.<locals>.callbackc 3 s^ �� | j� � f ��}|�� }|j|�� f��D ]\}}}|| ||fV q0|rZ|�� |_d S rH )r� r8 r~ rI r� r� r� r� ��_otherr� r� r r! r� x s )N)N)�poprA �listrx r )r� r� r� r� r r� r! r S s c @ s e Zd ZdZdd� ZdS )r z� Indicates a state or state action (e.g. #pop) to apply. For example default('#pop') is equivalent to ('', Token, '#pop') Note that state tuples may be used as well. .. versionadded:: 2.0 c C s || _ d S rH )r� )r; r� r r r! r= � s zdefault.__init__N)r, r- r. r/ r= r r r r! r � s c @ s" e Zd ZdZddd�Zdd� ZdS ) r z� Indicates a list of literal words that is transformed into an optimized regex that matches any of the words. .. versionadded:: 2.0 ro c C s || _ || _|| _d S rH )r �prefix�suffix)r; r r� r� r r r! r= � s zwords.__init__c C s t | j| j| jd�S )N�r� r� )r r r� r� r? r r r! r9 � s z words.getN)ro ro )r, r- r. r/ r= r9 r r r r! r � s c @ sJ e Zd ZdZdd� Zdd� Zdd� Zdd � Zdd d �Zd d� Z dd� Z d S )�RegexLexerMetazw Metaclass for RegexLexer, creates the self._tokens attribute from self.tokens on the first instantiation. c C s t |t�r|�� }t�||�jS )zBPreprocess the regular expression component of a token definition.)rA r r9 �re�compiler� )ry �regex�rflagsr� r r r! �_process_regex� s z RegexLexerMeta._process_regexc C s& t |�tks"t|�s"td|f ��|S )z5Preprocess the token component of a token definition.z2token type must be simple type or callable, not %r)r& r �callable�AssertionError)ry �tokenr r r! �_process_token� s �z RegexLexerMeta._process_tokenc C s t |t�rd|dkrdS ||kr$|fS |dkr0|S |dd� dkrRt|dd� � S dsbtd| ��n�t |t�r�d | j }| jd 7 _g }|D ],}||ks�td | ��|�| �|||�� q�|||< |fS t |t��r |D ] }||ks�|d ks�td | ��q�|S d�std| ��dS )z=Preprocess the state transition action of a token definition.�#pop����#pushN� z#pop:Fzunknown new state %rz_tmp_%dr� zcircular state ref %r)r� r� zunknown new state zunknown new state def %r) rA rO �intr� rw �_tmpname�extend�_process_staterx )ry � new_state� unprocessed� processed� tmp_state�itokens�istater r r! �_process_new_state� s> � ��z!RegexLexerMeta._process_new_statec C s� t |�tkstd| ��|d dks0td| ��||kr@|| S g }||< | j}|| D �].}t|t�r�||ks~td| ��|�| �||t|��� qZt|t�r�qZt|t �r�| � |j ||�}|� t �d�jd|f� qZt |�tks�td| ��z| �|d ||�}W n> tk �rD } z td |d || | f �| �W 5 d} ~ X Y nX | �|d �} t|�d k�rhd}n| � |d ||�}|� || |f� qZ|S ) z%Preprocess a single state definition.zwrong state name %rr �#zinvalid state name %rzcircular state reference %rro Nzwrong rule def %rz+uncompilable regex %r in state %r of %r: %sr� � )r& rO r� �flagsrA r r� r� rv r r� r� rB r� r� r� rx r� � Exception� ValueErrorr� rT ) ry r� r� r� �tokensr� �tdefr� �rex�errr� r r r! r� � sJ � �� �z RegexLexerMeta._process_stateNc C s< i }| j |< |p| j| }t|�D ]}| �|||� q$|S )z-Preprocess a dictionary of token definitions.)� _all_tokensr� r� r� )ry r) � tokendefsr� r� r r r! �process_tokendef s z RegexLexerMeta.process_tokendefc C s� i }i }| j D ]�}|j�di �}|�� D ]�\}}|�|�}|dkr||||< z|�t�}W n tk rp Y q(Y nX |||< q(|�|d�}|dkr�q(||||d �< z|�t�} W n tk r� Y q(X || ||< q(q|S )a Merge tokens from superclasses in MRO order, returning a single tokendef dictionary. Any state that is not defined by a subclass will be inherited automatically. States that *are* defined by subclasses will, by default, override that state in the superclass. If a subclass wishes to inherit definitions from a superclass, it can use the special value "inherit", which will cause the superclass' state definition to be included at that point in the state. r� Nr� )�__mro__�__dict__r9 �items�indexr r� r� ) ry r� � inheritable�c�toksr� r� �curitems� inherit_ndx� new_inh_ndxr r r! � get_tokendefs s0 z RegexLexerMeta.get_tokendefsc O sL d| j kr:i | _d| _t| d�r(| jr(n| �d| �� �| _tj | f|�|�S )z:Instantiate cls after preprocessing its token definitions.�_tokensr �token_variantsro ) r� r� r� �hasattrr� r� r� r� r& �__call__)ry rz �kwdsr r r! r� ; s zRegexLexerMeta.__call__)N) r, r- r. r/ r� r� r� r� r� r� r� r r r r! r� � s #, 1r� c @ s$ e Zd ZdZejZi Zddd�ZdS )r z� Base for simple stateful regular expression-based lexers. Simplifies the lexing process so that you need only provide a list of states and regular expressions. �r� c c s� d}| j }t|�}||d }|D �](\}}} |||�} | r"|dk rrt|�tkrb||| �� fV n|| | �E dH | �� }| dk �rHt| t�r�| D ]D} | dkr�t|�dkr�|� � q�| dkr�|� |d � q�|� | � q�nbt| t ��rt | �t|�k�r|dd�= n || d�= n,| dk�r*|� |d � nd�s<t d| ��||d } q q"zP|| d k�r�d g}|d }|td fV |d7 }W q |t|| fV |d7 }W q tk �r� Y �q�Y q X q dS ) z~ Split ``text`` into (tokentype, text) pairs. ``stack`` is the initial stack (default: ``['root']``) r r� Nr� r� r� F�wrong state def: %rrF r� )r� r� r& r r� r� rA rx rT r� rB r� �absr� r r r� ) r; rC r� r� r� � statestack� statetokens�rexmatchr� r� �mr� r r r! rI l sR z!RegexLexer.get_tokens_unprocessedN)r� ) r, r- r. r/ r� � MULTILINEr� r� rI r r r r! r I s c @ s" e Zd ZdZddd�Zdd� ZdS )r z9 A helper object that holds lexer position data. Nc C s* || _ || _|pt|�| _|p"dg| _d S )Nr� )rC r� rT r� r� )r; rC r� r� r� r r r! r= � s zLexerContext.__init__c C s d| j | j| jf S )NzLexerContext(%r, %r, %r))rC r� r� r? r r r! r@ � s �zLexerContext.__repr__)NN)r, r- r. r/ r= r@ r r r r! r � s c @ s e Zd ZdZddd�ZdS )r zE A RegexLexer that uses a context object to store its state. Nc c s< | j }|s t|d�}|d }n|}||jd }|j}|D �]`\}}}|||j|j�} | r:|dk r�t|�tkr�|j|| �� fV | �� |_n$|| | |�E dH |s�||jd }|dk �r�t |t ��r |D ]P} | dkr�t |j�dkr�|j� � q�| dk�r|j� |jd � q�|j� | � q�nlt |t��rZt|�t |j�k�rL|jdd�= n |j|d�= n0|dk�rx|j� |jd � nd�s�td | ��||jd } q6q:zz|j|jk�r�W �q8||j d k�r�dg|_|d }|jtd fV | jd7 _W q6|jt||j fV | jd7 _W q6 tk �r4 Y �q8Y q6X q6dS ) z Split ``text`` into (tokentype, text) pairs. If ``context`` is given, use this lexer context instead. r r� r� Nr� r� r� Fr� rF )r� r r� rC r� r� r& r r� rA rx rT r� rB r� r� r� r r r� ) r; rC �contextr� r� r� r� r� r� r� r� r r r! rI � s` z)ExtendedRegexLexer.get_tokens_unprocessed)NN)r, r- r. r/ rI r r r r! r � s c c s� t | �} zt| �\}}W n tk r8 |E dH Y dS X d}d}|D ]�\}}}|dkr\|}d} |�r|t|� |k�r|| || � } | r�||| fV |t| �7 }|D ]"\} } } || | fV |t| �7 }q�|| } zt| �\}}W q` tk �r d}Y �qY q`X q`| t|�k rF|||| d� fV |t|�| 7 }qF|�r�|�pHd}|D ]$\}}}|||fV |t|�7 }�qNzt| �\}}W n tk �r� d}Y �q�Y nX �q:dS )ag Helper for lexers which must combine the results of several sublexers. ``insertions`` is a list of ``(index, itokens)`` pairs. Each ``itokens`` iterable should be inserted at position ``index`` into the token stream given by the ``tokens`` argument. The result is a combined token stream. TODO: clean up the code here. NTr F)�iter�next� StopIterationrT )rr r� r� r� �realpos�insleftrt rK rL �oldi�tmpval�it_index�it_token�it_value�pr r r! rp sN rp c @ s e Zd ZdZdd� ZdS )�ProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c sL t |t�r t|j|j|jd��n|�t��|��tjf� ���fdd� }|S )Nr� c s` � j d ���fddg�}t�� }��| ||�}t�� }|d d7 < |d || 7 < |S )Nr� r r r� )� _prof_data� setdefault�timer� )rC r� �endpos�info�t0�res�t1�ry �compiledr� r� r r! � match_funcM s z:ProfilingRegexLexerMeta._process_regex.<locals>.match_func) rA r r r� r� r� r� �sys�maxsize)ry r� r� r� r� r r� r! r� E s � z&ProfilingRegexLexerMeta._process_regexN)r, r- r. r/ r� r r r r! r� B s r� c @ s" e Zd ZdZg ZdZddd�ZdS )�ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes.� r� c # s� � j j�i � t�� ||�E d H � j j�� }tdd� |�� D �� fdd�dd�}tdd� |D ��}t � t d� j j t |�|f � t d � t d d � t d � |D ]}t d | � q�t d � d S )Nc s sN | ]F\\}}\}}|t |��d ��dd�dd� |d| d| | fV qdS )zu'z\\�\N�A i� )�reprrW rE )�.0r� �r�nrK r r r! � <genexpr>c s � �z=ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<genexpr>c s | � j S rH )�_prof_sort_indexr r? r r! r" f r# z<ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<lambda>T)�key�reversec s s | ]}|d V qdS )� Nr )r� r r r r! r h s z2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)r� r� zn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) r> r� rB r rI r� �sortedr� �sum�printr, rT )r; rC r� �rawdatar� � sum_totalr+ r r? r! rI ^ s( � �� z*ProfilingRegexLexer.get_tokens_unprocessedN)r� )r, r- r. r/ r� r rI r r r r! r� X s r� )3r/ r� r� r� �pip._vendor.pygments.filterr r � pip._vendor.pygments.filtersr �pip._vendor.pygments.tokenr r r r �pip._vendor.pygments.utilr r r r r r � pip._vendor.pygments.regexoptr �__all__rR � staticmethod�_default_analyser& r$ r r rO r rv r rx rw r{ r r� r r r r r� r r r rp r� r� r r r r! �<module> s` �� !' 2 )aH@
Save File
Cancel