← Back
Editing: refactor.cpython-38.pyc
U i�fk � @ s d Z dZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZ ddlmZ ddlmZmZ dd lmZ d!d d �ZG d d� de�Zdd� Zdd� Zdd� Zdd� Z dd� Z G dd� de�Z G dd � d e �Z G d d � d e�Z!G d d � d e �Z"dS )"z�Refactoring framework. Used as a main program, this can refactor any number of files and/or recursively descend down directories. Imported as a module, this provides infrastructure to write your own refactoring tool. z#Guido van Rossum <guido@python.org>� N)�chain� )�driver�tokenize�token)� find_root)�pytree�pygram)� btm_matcherTc C sT t | g g dg�}g }t�|j�D ].\}}}|�d�r |rD|dd� }|�|� q |S )zEReturn a sorted list of all available fix names in the given package.�*�fix_� N)� __import__�pkgutilZ iter_modules�__path__� startswith�append)Z fixer_pkgZ remove_prefixZpkgZ fix_names�finder�nameZispkg� r �5/opt/alt/python38/lib64/python3.8/lib2to3/refactor.py�get_all_fix_names s r c @ s e Zd ZdS )� _EveryNodeN��__name__� __module__� __qualname__r r r r r + s r c C s� t | tjtjf�r(| jdkr t�| jhS t | tj�rH| jrDt| j�S t�t | tj �r�t � }| jD ] }|D ]}|� t|�� qhq`|S t d| ��dS )zf Accepts a pytree Pattern Node and returns a set of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s) � isinstancer Z NodePatternZ LeafPattern�typer ZNegatedPatternZcontent�_get_head_typesZWildcardPattern�set�update� Exception)Zpat�r�p�xr r r r / s r c C s� t �t�}g }| D ]x}|jrdzt|j�}W n tk rH |�|� Y q�X |D ]}|| �|� qNq|jdk r�||j �|� q|�|� qtt j j � � t j j �D ]}|| �|� q�t|�S )z^ Accepts a list of fixers and returns a dictionary of head node type --> fixer list. N)� collections� defaultdict�list�patternr r r Z _accept_typer r �python_grammarZ symbol2number�values�tokens�extend�dict)Z fixer_listZ head_nodesZevery�fixerZheadsZ node_typer r r �_get_headnode_dictK s$ �r0 c s � fdd�t � d�D �S )zN Return the fully qualified names for fixers in the package pkg_name. c s g | ]}� d | �qS ��.r )�.0�fix_name�Zpkg_namer r � <listcomp>h s �z+get_fixers_from_package.<locals>.<listcomp>F)r r5 r r5 r �get_fixers_from_packaged s �r7 c C s | S �Nr )�objr r r � _identityk s r: c sX d}t �t�| �j�� � fdd�}ttjt jtj h�}t � }z�|� \}}||krTq>q>|tj krl|rf�q6d}q>|tj k�r6|dk�r6|� \}}|tj ks�|dkr��q6|� \}}|tj ks�|dkrq6|� \}}|tj kr�|dkr�|� \}}|tj k�r4|�|� |� \}}|tj k�s.|d k�r"�q4|� \}}q�q>�q6q>W n tk �rN Y nX t|�S ) NFc s t � �} | d | d fS )Nr r )�next)�tok��genr r �advancer s z(_detect_future_features.<locals>.advanceT�fromZ __future__�import�(�,)r �generate_tokens�io�StringIO�readline� frozensetr �NEWLINE�NL�COMMENTr �STRING�NAME�OP�add� StopIteration)�sourceZhave_docstringr? �ignore�features�tp�valuer r= r �_detect_future_featureso sB rV c @ s e Zd ZdZdS )� FixerErrorz A fixer could not be loaded.N)r r r �__doc__r r r r rW � s rW c @ s� e Zd Zddd�ZdZdZd4dd�Zdd � Zd d � Zd d � Z dd� Z dd� Z d5dd�Z d6dd�Z dd� Zd7dd�Zdd� Zd8d d �Zd d � Zd d!� Zd9d"d#�Zd:d$d%�Zd&Zd'Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Z d2d3� Z dS );�RefactoringToolF)�print_function�write_unchanged_filesZFixr Nc C s. || _ |p g | _| j�� | _|dk r0| j�|� | jd rDtj| _ntj | _| j� d�| _ g | _ t �d�| _g | _d| _tj| jtj| jd�| _| �� \| _| _g | _t�� | _ g | _ g | _ t | j| j�D ]F}|j r�| j �!|� q�|| jkr�| j �"|� q�|| jkr�| j �"|� q�t#| j �| _$t#| j �| _%dS )z�Initializer. Args: fixer_names: a list of fixers to import options: a dict with configuration. explicit: a list of fixers to run even if they are explicit. NrZ r[ rY F)�convert�logger)&�fixers�explicit�_default_options�copy�optionsr! r �!python_grammar_no_print_statement�grammarr* �getr[ �errors�loggingZ getLoggerr] � fixer_log�wroter ZDriverr r\ � get_fixers� pre_order� post_order�files�bmZ BottomMatcher�BMZ bmi_pre_orderZbmi_post_orderr Z BM_compatibleZ add_fixerr r0 �bmi_pre_order_heads�bmi_post_order_heads)�selfZ fixer_namesrb r_ r/ r r r �__init__� s> � zRefactoringTool.__init__c C sX g }g }| j D �] }t|i i dg�}|�dd�d }|�| j�rR|t| j�d� }|�d�}| jd�dd � |D �� }zt ||�}W n& t k r� t d ||f �d�Y nX || j | j �} | jr�| jd k r�|| jkr�| �d |� q| �d |� | jdk�r|�| � q| jdk�r |�| � qt d| j ��qt�d�} |j| d� |j| d� ||fS )a Inspects the options to load the requested patterns and handlers. Returns: (pre_order, post_order), where pre_order is the list of fixers that want a pre-order AST traversal, and post_order is the list that want post-order traversal. r r2 r ���N�_� c S s g | ] }|� � �qS r )�title)r3 r$ r r r r6 � s z.RefactoringTool.get_fixers.<locals>.<listcomp>zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sZpreZpostzIllegal fixer order: %rZ run_order��key)r^ r �rsplitr � FILE_PREFIX�len�split� CLASS_PREFIX�join�getattr�AttributeErrorrW rb rh r_ � log_message� log_debug�orderr �operator� attrgetter�sort) rr Zpre_order_fixersZpost_order_fixersZ fix_mod_path�modr4 �parts� class_nameZ fix_classr/ Zkey_funcr r r rj � s: � zRefactoringTool.get_fixersc O s � dS )z Called when an error occurs.Nr )rr �msg�args�kwdsr r r � log_error� s zRefactoringTool.log_errorc G s |r || }| j �|� dS )zHook to log a message.N)r] �info�rr r� r� r r r r� s zRefactoringTool.log_messagec G s |r || }| j �|� d S r8 )r] �debugr� r r r r� s zRefactoringTool.log_debugc C s dS )zTCalled with the old version, new version, and filename of a refactored file.Nr )rr �old_text�new_text�filename�equalr r r � print_output s z RefactoringTool.print_outputc C s8 |D ].}t j�|�r$| �|||� q| �|||� qdS )z)Refactor a list of files and directories.N)�os�path�isdir� refactor_dir� refactor_file)rr �items�write� doctests_onlyZ dir_or_filer r r �refactor s zRefactoringTool.refactorc C s� t jd }t �|�D ]�\}}}| �d|� |�� |�� |D ]>}|�d�s>t j�|�d |kr>t j�||�} | � | ||� q>dd� |D �|dd�< qdS )z�Descends down a directory and refactor every Python file found. Python files are assumed to have a .py extension. Files and subdirectories starting with '.' are skipped. �pyzDescending into %sr2 r c S s g | ]}|� d �s|�qS r1 )r )r3 Zdnr r r r6 . s z0RefactoringTool.refactor_dir.<locals>.<listcomp>N) r� �extsep�walkr� r� r r� �splitextr r� ) rr Zdir_namer� r� Zpy_ext�dirpathZdirnames� filenamesr �fullnamer r r r� s �z RefactoringTool.refactor_dirc C s� zt |d�}W n6 tk rD } z| �d||� W Y �dS d}~X Y nX zt�|j�d }W 5 |�� X tj |d|dd��}|�� |fW 5 Q R � S Q R X dS ) zG Do our best to decode a Python source file correctly. �rbzCan't open %s: %s)NNNr r# rv ��encoding�newline) �open�OSErrorr� �closer �detect_encodingrG rE �read)rr r� �f�errr� r r r �_read_python_source0 s z#RefactoringTool._read_python_sourcec C s� | � |�\}}|dkrdS |d7 }|rn| �d|� | �||�}| jsL||kr`| �|||||� q�| �d|� nH| �||�}| js�|r�|jr�| jt|�dd� |||d� n | �d|� dS )zRefactors a file.N� zRefactoring doctests in %szNo doctest changes in %srt )r� r� zNo changes in %s)r� r� �refactor_docstringr[ �processed_file�refactor_string� was_changed�str)rr r� r� r� �inputr� �output�treer r r r� @ s" �z RefactoringTool.refactor_filec C s� t |�}d|krtj| j_zVz| j�|�}W n@ tk rl } z"| �d||jj |� W Y �W �dS d}~X Y nX W 5 | j| j_X ||_ | � d|� | � ||� |S )aF Refactor a given input string. Args: data: a string holding the code to be refactored. name: a human-readable name for use in error/log messages. Returns: An AST corresponding to the refactored input stream; None if there were errors during the parse. rZ zCan't parse %s: %s: %sNzRefactoring %s) rV r rc r rd Z parse_stringr" r� � __class__r �future_featuresr� � refactor_tree)rr �datar rS r� r� r r r r� W s" � z RefactoringTool.refactor_stringc C s� t j�� }|rN| �d� | �|d�}| js2||krB| �|d|� q�| �d� n:| �|d�}| jsj|r~|jr~| �t |�d|� n | �d� d S )Nz Refactoring doctests in stdinz<stdin>zNo doctest changes in stdinzNo changes in stdin) �sys�stdinr� r� r� r[ r� r� r� r� )rr r� r� r� r� r r r �refactor_stdinr s z RefactoringTool.refactor_stdinc C s� t | j| j�D ]}|�||� q| �| j|�� � | �| j|�� � | j�|� � �}t |� � ��r�| jj D �]D}||krj|| rj|| j tjjdd� |jr�|| j tjjd� t|| �D ]�}||| kr�|| �|� z t|� W n tk �r Y q�Y nX |j�r||jk�rq�|�|�}|r�|�||�}|dk r�|�|� |�� D ] }|j�s^g |_|j�|� �qL| j�|� � �}|D ]*} | |k�r�g || < || � || � �q�q�qjqTt | j| j�D ]}|� ||� �q�|j S )a� Refactors a parse tree (modifying the tree in place). For compatible patterns the bottom matcher module is used. Otherwise the tree is traversed node-to-node for matches. Args: tree: a pytree.Node instance representing the root of the tree to be refactored. name: a human-readable name for this tree. Returns: True if the tree was modified, False otherwise. T)ry �reverserx N) r rk rl Z start_tree� traverse_byrp rq ro �runZleaves�anyr+ r^ r� r ZBaseZdepthZkeep_line_orderZ get_linenor( �remover � ValueErrorZfixers_applied�match� transform�replacer r- Z finish_treer� ) rr r� r r/ Z match_set�node�results�newZ new_matchesZfxrr r r r� � sJ z RefactoringTool.refactor_treec C sV |sdS |D ]D}||j D ]4}|�|�}|r|�||�}|dk r|�|� |}qq dS )a Traverse an AST, applying a set of fixers to each node. This is a helper method for refactor_tree(). Args: fixers: a list of fixer instances. traversal: a generator that yields AST nodes. Returns: None N)r r� r� r� )rr r^ Z traversalr� r/ r� r� r r r r� � s zRefactoringTool.traverse_byc C s� | j �|� |dkr.| �|�d }|dkr.dS ||k}| �||||� |r`| �d|� | js`dS |rv| �||||� n | �d|� dS )zR Called when a file has been refactored and there may be changes. Nr zNo changes to %szNot writing changes to %s)rm r r� r� r� r[ � write_file)rr r� r� r� r� r� r� r r r r� � s z RefactoringTool.processed_filec C s� zt j|d|dd�}W n6 tk rL } z| �d||� W Y �dS d}~X Y nX |�F z|�|� W n0 tk r� } z| �d||� W 5 d}~X Y nX W 5 Q R X | �d|� d| _dS ) z�Writes a string to a file. It first shows a unified diff between the old text and the new text, and then rewrites the file; the latter is only done if the write option is set. �wrv r� zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)rE r� r� r� r� r� ri )rr r� r� r� r� �fpr� r r r r� � s * zRefactoringTool.write_filez>>> z... c C s g }d}d}d}d}|j dd�D ]�}|d7 }|�� �| j�r~|dk rZ|�| �||||�� |}|g}|�| j�} |d| � }q |dk r�|�|| j �s�||| j�� d kr�|� |� q |dk r�|�| �||||�� d}d}|� |� q |dk �r |�| �||||�� d� |�S )a� Refactors a docstring, looking for doctests. This returns a modified version of the input string. It looks for doctests, which start with a ">>>" prompt, and may be continued with "..." prompts, as long as the "..." is indented the same as the ">>>". (Unfortunately we can't use the doctest module's parser, since, like most parsers, it is not geared towards preserving the original source.) Nr T��keependsr r� rv ) � splitlines�lstripr �PS1r- �refactor_doctest�find�PS2�rstripr r ) rr r� r� �result�blockZ block_lineno�indent�lineno�line�ir r r r� sJ � �� � �z"RefactoringTool.refactor_docstringc s. z�� ||� �}W nj tk r| } zL�j�tj�rN|D ]}��d|�d�� q6��d|||j j |� | W Y �S d}~X Y nX �� ||��r*t |�j dd�}|d|d � ||d d� } }| dg|d ks�t| ��|d �d�s�|d d7 < � �j |�d � g}|�r*|� �fd d �|D �7 }|S ) z�Refactors one doctest. A doctest is given as a block of lines, the first of which starts with ">>>" (possibly indented), while the remaining lines start with "..." (identically indented). z Source: %sr� z+Can't parse docstring in %s line %s: %s: %sNTr� r rt r c s g | ]}� �j | �qS r )r� )r3 r� �r� rr r r r6 Z s z4RefactoringTool.refactor_doctest.<locals>.<listcomp>)� parse_blockr" r] Z isEnabledForrg �DEBUGr� r� r� r� r r� r� r� �AssertionError�endswithr� �pop) rr r� r� r� r� r� r� r� r� Zclippedr r� r r� @ s. �"z RefactoringTool.refactor_doctestc C s� | j r d}nd}| js$| �d|� n"| �d|� | jD ]}| �|� q6| jrl| �d� | jD ]}| �|� q\| jr�t| j�dkr�| �d� n| �dt| j�� | jD ] \}}}| j|f|�|� q�d S ) N�werez need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:r zThere was 1 error:zThere were %d errors:)ri rm r� rh rf r| )rr r� �file�messager� r� r� r r r � summarize] s$ zRefactoringTool.summarizec C s" | j �| �|||��}t� |_|S )z�Parses a block into a tree. This is necessary to get correct line number / offset information in the parser diagnostics and embedded into the parse tree. )r Z parse_tokens� wrap_toksrH r� )rr r� r� r� r� r r r r� t s zRefactoringTool.parse_blockc c sd t �| �||�j�}|D ]F\}}\}}\} } } ||d 7 }| |d 7 } ||||f| | f| fV qdS )z;Wraps a tokenize stream to systematically modify start/end.r N)r rD � gen_lines�__next__) rr r� r� r� r, r rU Zline0Zcol0Zline1Zcol1Z line_textr r r r� ~ s zRefactoringTool.wrap_toksc c sx || j }|| j }|}|D ]N}|�|�r>|t|�d� V n(||�� d krVdV ntd||f ��|}q dV qldS )z�Generates lines as expected by tokenize from a list of lines. This strips the first len(indent + self.PS1) characters off each line. Nr� zline=%r, prefix=%rrv )r� r� r r| r� r� )rr r� r� �prefix1Zprefix2�prefixr� r r r r� � s zRefactoringTool.gen_lines)NN)FF)FF)FF)F)NFN)N) r r r r` r~ r{ rs rj r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r r rY � s> � 4( O � + rY c @ s e Zd ZdS )�MultiprocessingUnsupportedNr r r r r r� � s r� c sB e Zd Z� fdd�Zd � fdd� Z� fdd�Z� fd d �Z� ZS ) �MultiprocessRefactoringToolc s" t t| �j||� d | _d | _d S r8 )�superr� rs �queue� output_lock�rr r� �kwargs�r� r r rs � s z$MultiprocessRefactoringTool.__init__Fr c s� |dkr t t���|||�S z dd l� W n tk r@ t�Y nX �jd k rTtd��� �� �_� � � �_ � �fdd�t |�D �}z*|D ] }|�� q�t t���|||� W 5 �j� � t |�D ]}�j� d � q�|D ]}|�� r�|� � q�d �_X d S )Nr r z already doing multiple processesc s g | ]}� j �jd ��qS ))�target)ZProcess�_child)r3 r� ��multiprocessingrr r r r6 � s �z8MultiprocessRefactoringTool.refactor.<locals>.<listcomp>)r� r� r� r� � ImportErrorr� r� � RuntimeErrorZ JoinableQueueZLockr� �ranger �putZis_alive�start)rr r� r� r� Z num_processesZ processesr� r$ r� r� r r� � s< � � � z$MultiprocessRefactoringTool.refactorc sN | j �� }|d k rJ|\}}ztt| �j||� W 5 | j �� X | j �� }q d S r8 )r� re Z task_doner� r� r� )rr Ztaskr� r� r� r r r� � s � z"MultiprocessRefactoringTool._childc s2 | j d k r | j �||f� ntt| �j||�S d S r8 )r� r� r� r� r� r� r� r r r� � s �z)MultiprocessRefactoringTool.refactor_file)FFr )r r r rs r� r� r� � __classcell__r r r� r r� � s � r� )T)#rX � __author__rE r� r r� rg r� r& � itertoolsr Zpgen2r r r Z fixer_utilr rv r r r rn r r" r r r0 r7 r: rV rW �objectrY r� r� r r r r �<module> s8 (
Save File
Cancel