← Back
Editing: collector.pyc
� �Rec @ s d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z m Z d d l m Z d d l m Z m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l! m" Z" d d l# m$ Z$ m% Z% d d l& m' Z' d d l( m) Z) m* Z* d d l+ m, Z, m- Z- e' rd d l. m/ Z/ d d l0 m1 Z1 m2 Z2 m3 Z3 m4 Z4 m5 Z5 m6 Z6 m7 Z7 m8 Z8 d d l9 Z: d d l; m< Z< d d l= m> Z> e: j? j@ jA ZB e4 eC eC f ZD n e jE eF � ZG d � ZH d � ZI d eJ f d � � YZK d � ZL d eJ f d � � YZM d � ZN d � ZO d � ZP d! � ZQ d" � ZR d# � ZS e jT d$ e jU � ZV d% � ZW d& � ZX d' � ZY d( eZ f d) � � YZ[ d* � Z\ e\ d+ � � Z] d, eZ f d- � � YZ^ d d. � Z` ea d/ � Zb d d0 � Zc d1 � Zd ee d2 � Zf d3 eZ f d4 � � YZg d5 eZ f d6 � � YZh d S(7 sM The main purpose of this module is to expose LinkCollector.collect_links(). i����N( t OrderedDict( t html5libt requests( t unescape( t RetryErrort SSLError( t parse( t request( t NetworkConnectionError( t Link( t SearchScope( t raise_for_status( t lru_cache( t ARCHIVE_EXTENSIONS( t pairwiset redact_auth_from_url( t MYPY_CHECK_RUNNING( t path_to_urlt url_to_path( t is_urlt vcs( t Values( t Callablet Iterablet Listt MutableMappingt Optionalt Sequencet Tuplet Union( t Response( t PipSessionc C sG x@ t j D]5 } | j � j | � r | t | � d k r | Sq Wd S( sg Look for VCS schemes in the URL. Returns the matched VCS scheme, or None if there's no match. s +:N( R t schemest lowert startswitht lent None( t urlt scheme( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _match_vcs_scheme8 s +c C s7 t | � j } x! t D] } | j | � r t Sq Wt S( s2 Return whether the URL looks like an archive. ( R t filenameR t endswitht Truet False( R% R( t bad_ext( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _is_url_like_archiveD s t _NotHTMLc B s e Z d � Z RS( c C s/ t t | � j | | � | | _ | | _ d S( N( t superR. t __init__t content_typet request_desc( t selfR1 R2 ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 P s ( t __name__t __module__R0 ( ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR. O s c C sF | j j d d � } | j � j d � sB t | | j j � � n d S( s� Check the Content-Type header to ensure the response contains HTML. Raises `_NotHTML` if the content type is not text/html. s Content-Typet s text/htmlN( t headerst getR! R" R. R t method( t responseR1 ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _ensure_html_headerW s t _NotHTTPc B s e Z RS( ( R4 R5 ( ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR< b s c C si t j | � \ } } } } } | d d h k r< t � � n | j | d t �} t | � t | � d S( s� Send a HEAD request to the URL, and ensure the response contains HTML. Raises `_NotHTTP` if the URL is not available for a HEAD request, or `_NotHTML` if the content type is not text/html. t httpt httpst allow_redirectsN( t urllib_parset urlsplitR< t headR* R R; ( R% t sessionR& t netloct patht queryt fragmentt resp( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _ensure_html_responsef s c C sp t | � r t | d | �n t j d t | � � | j | d i d d 6d d 6�} t | � t | � | S( s Access an HTML page with GET, and return the response. This consists of three parts: 1. If the URL looks suspiciously like an archive, send a HEAD first to check the Content-Type is HTML, to avoid downloading a large file. Raise `_NotHTTP` if the content type cannot be determined, or `_NotHTML` if it is not HTML. 2. Actually perform the request. Raise HTTP exceptions on network failures. 3. Check the Content-Type header to make sure we got HTML, and raise `_NotHTML` otherwise. RC s Getting page %sR7 s text/htmlt Accepts max-age=0s Cache-Control( R- RI t loggert debugR R8 R R; ( R% RC RH ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _get_html_responsew s c C sF | rB d | k rB t j | d � \ } } d | k rB | d Sn d S( sB Determine if we have any encoding information in our headers. s Content-Typet charsetN( t cgit parse_headerR$ ( R7 R1 t params( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _get_encoding_from_headers� s c C s= x6 | j d � D]% } | j d � } | d k r | Sq W| S( s� Determine the HTML document's base URL. This looks for a ``<base>`` tag in the HTML document. If present, its href attribute denotes the base URL of anchor tags in the document. If there is no such tag (or if it does not have a valid href attribute), the HTML file's URL is used as the base URL. :param document: An HTML document representation. The current implementation expects the result of ``html5lib.parse()``. :param page_url: The URL of the HTML document. s .//baset hrefN( t findallR8 R$ ( t documentt page_urlt baseRS ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _determine_base_url� s c C s t j t j | � � S( sP Clean a "part" of a URL path (i.e. after splitting on "@" characters). ( R@ t quotet unquote( t part( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _clean_url_path_part� s c C s t j t j | � � S( s� Clean the first part of a URL path that corresponds to a local filesystem path (i.e. the first part after splitting on "@" characters). ( t urllib_requestt pathname2urlt url2pathname( R[ ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _clean_file_url_path� s s (@|%2F)c C s� | r t } n t } t j | � } g } xO t t j | d g � � D]2 \ } } | j | | � � | j | j � � qF Wd j | � S( s* Clean the path portion of a URL. R6 ( R` R\ t _reserved_chars_ret splitR t itertoolst chaint appendt uppert join( RE t is_local_patht clean_funct partst cleaned_partst to_cleant reserved( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _clean_url_path� s (c C sG t j | � } | j } t | j d | �} t j | j d | � � S( s� Make sure a link is fully quoted. For example, if ' ' occurs in the URL, it will be replaced with "%20", and without double-quoting other characters. Rh RE ( R@ t urlparseRD Rn RE t urlunparset _replace( R% t resultRh RE ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _clean_link� s c C s� | j d � } | s d St t j | | � � } | j d � } | rR t | � n d } | j d � } | r| t | � } n t | d | d | d | �} | S( sJ Convert an anchor element in a simple repository page to a Link. RS s data-requires-pythons data-yankedt comes_fromt requires_pythont yanked_reasonN( R8 R$ Rs R@ t urljoinR R ( t anchorRV t base_urlRS R% t pyrequireRv t link( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _create_link_from_element s t CacheablePageContentc B s# e Z d � Z d � Z d � Z RS( c C s | j s t � | | _ d S( N( t cache_link_parsingt AssertionErrort page( R3 R� ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 , s c C s+ t | t | � � o* | j j | j j k S( N( t isinstancet typeR� R% ( R3 t other( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt __eq__1 s c C s t | j j � S( N( t hashR� R% ( R3 ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt __hash__6 s ( R4 R5 R0 R� R� ( ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR} + s c sC t d d � � f d � � � t j � � � � f d � � } | S( s� Given a function that parses an Iterable[Link] from an HTMLPage, cache the function's result (keyed by CacheablePageContent), unless the HTMLPage `page` has `page.cache_link_parsing == False`. t maxsizec s t � | j � � S( N( t listR� ( t cacheable_page( t fn( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt wrapperE s c s) | j r � t | � � St � | � � S( N( R~ R} R� ( R� ( R� R� ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt wrapper_wrapperJ s N( R R$ t functoolst wraps( R� R� ( ( R� R� s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt with_cached_html_pages; s !c c s� t j | j d | j d t �} | j } t | | � } xF | j d � D]5 } t | d | d | �} | d k ry qI n | VqI Wd S( sP Parse an HTML document, and yield its anchor elements as Link objects. t transport_encodingt namespaceHTMLElementss .//aRV Ry N( R R t contentt encodingR+ R% RX RT R| R$ ( R� RU R% Ry Rx R{ ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt parse_linksT s t HTMLPagec B s# e Z d Z e d � Z d � Z RS( s' Represents one page, along with its URLc C s( | | _ | | _ | | _ | | _ d S( sm :param encoding: the encoding to decode the given content. :param url: the URL from which the HTML was downloaded. :param cache_link_parsing: whether links parsed from this page's url should be cached. PyPI index urls should have this set to False, for example. N( R� R� R% R~ ( R3 R� R� R% R~ ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 p s c C s t | j � S( N( R R% ( R3 ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt __str__� s ( R4 R5 t __doc__R* R0 R� ( ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR� m s c C s, | d k r t j } n | d | | � d S( Ns% Could not fetch URL %s: %s - skipping( R$ RK RL ( R{ t reasont meth( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _handle_get_page_fail� s c C s1 t | j � } t | j d | d | j d | �S( NR� R% R~ ( RR R7 R� R� R% ( R: R~ R� ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _make_html_page� s c C s, | d k r t d � � n | j j d d � d } t | � } | r] t j d | | � d St j | � \ } } } } } } | d k r� t j j t j | � � r� | j d � s� | d 7} n t j | d � } t j d | � n y t | d | �} Wnt k r t j d | � n t k rM} t j d | | j | j � n� t k rl} t | | � n� t k r�} t | | � n� t k r�} d } | t | � 7} t | | d t j �n_ t j k r�} t | d j | � � n4 t j k rt | d � n Xt | d | j! �Sd S( Ns? _get_html_page() missing 1 required keyword argument: 'session't #i i sI Cannot look at %s URL %s because it does not support lookup as web pages.t filet /s index.htmls# file: URL is directory, getting %sRC s` Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.si Skipping page %s because the %s request got Content-Type: %s.The only supported Content-Type is text/htmls4 There was a problem confirming the ssl certificate: R� s connection error: {}s timed outR~ (" R$ t TypeErrorR% Rb R' RK t warningR@ Ro t osRE t isdirR] R_ R) Rw RL RM R< R. R2 R1 R R� R R t strt infoR t ConnectionErrort formatt TimeoutR� R~ ( R{ RC R% t vcs_schemeR& t _RE RH t excR� ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _get_html_page� sP !' c C s t t j | � � S( sQ Return a list of links, with duplicates removed and ordering preserved. ( R� R t fromkeys( t links( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt _remove_duplicate_links� s c sh g � g � � � f d � } x=| D]5} t j j | � } | j d � } | sX | r.| rg | } n t | � } t j j | � r� | r� t j j | � } xY t j | � D] } | t j j | | � � q� Wq+| r� � j | � q+t j d | � qZt j j | � r| | � qZt j d | � q% t | � rJ� j | � q% t j d | � q% W� � f S( s� Divide a list of locations into two groups: "files" (archives) and "urls." :return: A pair of lists (files, urls). c sL t | � } t j | d t �d d k r; � j | � n � j | � d S( Nt stricti s text/html( R t mimetypest guess_typeR+ Re ( RE R% ( t filest urls( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt sort_path� s s file:s( Path '%s' is ignored: it is a directory.s: Url '%s' is ignored: it is neither a file nor a directory.sQ Url '%s' is ignored. It is either a non-existing path or lacks a specific scheme.( R� RE t existsR" R R� t realpatht listdirRg Re RK R� t isfileR ( t locationst expand_dirR� R% Rh t is_file_urlRE t item( ( R� R� s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt group_locations� s< t CollectedLinksc B s e Z d Z d � Z RS( s� Encapsulates the return value of a call to LinkCollector.collect_links(). The return value includes both URLs to project pages containing package links, as well as individual package Link objects collected from other sources. This info is stored separately as: (1) links from the configured file locations, (2) links from the configured find_links, and (3) urls to HTML project pages, as described by the PEP 503 simple repository API. c C s | | _ | | _ | | _ d S( s� :param files: Links from file locations. :param find_links: Links from find_links. :param project_urls: URLs to HTML project pages, as described by the PEP 503 simple repository API. N( R� t find_linkst project_urls( R3 R� R� R� ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 , s ( R4 R5 R� R0 ( ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR� s t LinkCollectorc B sJ e Z d Z d � Z e e d � � Z e d � � Z d � Z d � Z RS( s� Responsible for collecting Link objects from all configured locations, making network requests as needed. The class's main method is its collect_links() method. c C s | | _ | | _ d S( N( t search_scopeRC ( R3 RC R� ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR0 G s c C s� | j g | j } | j rO | rO t j d d j d � | D� � � g } n | j p[ g } t j d | d | � } t d | d | � } | S( s� :param session: The Session to use to make requests. :param suppress_no_index: Whether to ignore the --no-index option when constructing the SearchScope object. s Ignoring indexes: %st ,c s s | ] } t | � Vq d S( N( R ( t .0R% ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pys <genexpr>\ s R� t index_urlsRC R� ( t index_urlt extra_index_urlst no_indexRK RL Rg R� R t createR� ( t clsRC t optionst suppress_no_indexR� R� R� t link_collector( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR� P s c C s | j j S( N( R� R� ( R3 ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR� k s c C s t | d | j �S( s> Fetch an HTML page containing package links. RC ( R� RC ( R3 t location( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt fetch_pagep s c C s] | j } | j | � } t | � \ } } t | j d t �\ } } g t j | | � D] } t | � ^ qX } g | j D] } t | d � ^ qz } g t j d � | D� d � | D� � D] } | j j | � r� | ^ q� } t | � } d j t | � | � g } x$ | D] } | j d j | � � qWt j d j | � � t d | d | d | � S( s� Find all available links for the given project name. :return: All the Link objects (unfiltered), as a CollectedLinks object. R� s -fc s s! | ] } t | d t �Vq d S( R~ N( R R+ ( R� R% ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pys <genexpr>� s c s s | ] } t | � Vq d S( N( R ( R� R% ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pys <genexpr>� s s, {} location(s) to search for versions of {}:s * {}s R� R� R� ( R� t get_index_urls_locationsR� R� R* Rc Rd R RC t is_secure_originR� R� R# Re RK RL Rg R� ( R3 t project_nameR� t index_locationst index_file_loct index_url_loct fl_file_loct fl_url_locR% t file_linkst find_link_linksR{ t url_locationst lines( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt collect_linksw s* +% ( R4 R5 R� R0 t classmethodR+ R� t propertyR� R� R� ( ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyR� > s (i R� RO R� Rc t loggingR� R� t ret collectionsR t pip._vendorR R t pip._vendor.distlib.compatR t pip._vendor.requests.exceptionsR R t pip._vendor.six.moves.urllibR R@ R R] t pip._internal.exceptionsR t pip._internal.models.linkR t! pip._internal.models.search_scopeR t pip._internal.network.utilsR t pip._internal.utils.compatR t pip._internal.utils.filetypesR t pip._internal.utils.miscR R t pip._internal.utils.typingR t pip._internal.utils.urlsR R t pip._internal.vcsR R t optparseR t typingR R R R R R R R t xml.etree.ElementTreet xmlt pip._vendor.requestsR t pip._internal.network.sessionR t etreet ElementTreet Elementt HTMLElementR� t ResponseHeaderst getLoggerR4 RK R' R- t ExceptionR. R; R< RI RM RR RX R\ R` t compilet IGNORECASERa Rn Rs R| t objectR} R� R� R� R$ R� R* R� R� R� R+ R� R� R� ( ( ( s� /builddir/build/BUILDROOT/alt-python27-pip-20.2.4-5.el8.x86_64/opt/alt/python27/lib/python2.7/site-packages/pip/_internal/index/collector.pyt <module> st : 3 9 ;#
Save File
Cancel