← Back
Editing: robotparser.pyo
� z fc @ s} d Z d d l Z d d l Z d g Z d d d � � YZ d d d � � YZ d d d � � YZ d e j f d � � YZ d S( s& robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt i����Nt RobotFileParserc B sb e Z d Z d d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z d � Z RS( ss This class provides a set of methods to read, parse and answer questions about a single robots.txt file. t c C s> g | _ d | _ t | _ t | _ | j | � d | _ d S( Ni ( t entriest Nonet default_entryt Falset disallow_allt allow_allt set_urlt last_checked( t selft url( ( s# /usr/lib64/python2.7/robotparser.pyt __init__ s c C s | j S( s� Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. ( R ( R ( ( s# /usr/lib64/python2.7/robotparser.pyt mtime! s c C s d d l } | j � | _ d S( sY Sets the time the robots.txt file was last fetched to the current time. i����N( t timeR ( R R ( ( s# /usr/lib64/python2.7/robotparser.pyt modified* s c C s/ | | _ t j | � d d !\ | _ | _ d S( s, Sets the URL referring to a robots.txt file.i i N( R t urlparset hostt path( R R ( ( s# /usr/lib64/python2.7/robotparser.pyR 2 s c C s� t � } | j | j � } g | D] } | j � ^ q" } | j � | j | _ | j d k rk t | _ nO | j d k r� | j d k r� t | _ n% | j d k r� | r� | j | � n d S( s4 Reads the robots.txt URL and feeds it to the parser.i� i� i� i� i� N( i� i� ( t URLopenert openR t stript closet errcodet TrueR R t parse( R t openert ft linet lines( ( s# /usr/lib64/python2.7/robotparser.pyt read7 s c C sA d | j k r- | j d k r= | | _ q= n | j j | � d S( Nt *( t useragentsR R R t append( R t entry( ( s# /usr/lib64/python2.7/robotparser.pyt _add_entryE s c C s& d } d } t � } | j � x�| D]�} | d 7} | s� | d k rZ t � } d } q� | d k r� | j | � t � } d } q� n | j d � } | d k r� | | } n | j � } | s� q& n | j d d � } t | � d k r& | d j � j � | d <t j | d j � � | d <| d d k ru| d k rX| j | � t � } n | j j | d � d } q| d d k r�| d k r�| j j t | d t � � d } q�q| d d k r| d k r�| j j t | d t � � d } q�qq& q& W| d k r"| j | � n d S( s� parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines.i i i t #t :s user-agentt disallowt allowN( t EntryR R# t findR t splitt lent lowert urllibt unquoteR R! t rulelinest RuleLineR R ( R R t statet linenumberR" R t i( ( s# /usr/lib64/python2.7/robotparser.pyR N sP c C s� | j r t S| j r t S| j s' t St j t j | � � } t j d d | j | j | j | j f � } t j | � } | s� d } n x- | j D]" } | j | � r� | j | � Sq� W| j r� | j j | � St S( s= using the parsed robots.txt decide if useragent can fetch urlR t /( R R R R R R R- R. t urlunparseR t paramst queryt fragmentt quoteR t applies_tot allowanceR ( R t useragentR t parsed_urlR" ( ( s# /usr/lib64/python2.7/robotparser.pyt can_fetch� s$ c C sE | j } | j d k r+ | | j g } n d j t t | � � d S( Ns ( R R R t joint mapt str( R R ( ( s# /usr/lib64/python2.7/robotparser.pyt __str__� s ( t __name__t __module__t __doc__R R R R R R# R R> RB ( ( ( s# /usr/lib64/python2.7/robotparser.pyR s 4 R0 c B s) e Z d Z d � Z d � Z d � Z RS( so A rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.c C sS | d k r | r t } n t j t j | � � } t j | � | _ | | _ d S( NR ( R R R5 R- R9 R R; ( R R R; ( ( s# /usr/lib64/python2.7/robotparser.pyR � s c C s | j d k p | j | j � S( NR ( R t startswith( R t filename( ( s# /usr/lib64/python2.7/robotparser.pyR: � s c C s | j r d p d d | j S( Nt Allowt Disallows : ( R; R ( R ( ( s# /usr/lib64/python2.7/robotparser.pyRB � s ( RC RD RE R R: RB ( ( ( s# /usr/lib64/python2.7/robotparser.pyR0 � s R( c B s2 e Z d Z d � Z d � Z d � Z d � Z RS( s? An entry has one or more user-agents and zero or more rulelinesc C s g | _ g | _ d S( N( R R/ ( R ( ( s# /usr/lib64/python2.7/robotparser.pyR � s c C sj g } x' | j D] } | j d | d g � q Wx* | j D] } | j t | � d g � q: Wd j | � S( Ns User-agent: s R ( R t extendR/ RA R? ( R t rett agentR ( ( s# /usr/lib64/python2.7/robotparser.pyRB � s c C s] | j d � d j � } x= | j D]2 } | d k r9 t S| j � } | | k r# t Sq# Wt S( s2 check if this entry applies to the specified agentR4 i R ( R* R, R R R ( R R< RL ( ( s# /usr/lib64/python2.7/robotparser.pyR: � s c C s. x' | j D] } | j | � r | j Sq Wt S( sZ Preconditions: - our agent applies to this entry - filename is URL decoded( R/ R: R; R ( R RG R ( ( s# /usr/lib64/python2.7/robotparser.pyR; � s ( RC RD RE R RB R: R; ( ( ( s# /usr/lib64/python2.7/robotparser.pyR( � s R c B s# e Z d � Z d � Z d � Z RS( c G s t j j | | � d | _ d S( Ni� ( R- t FancyURLopenerR R ( R t args( ( s# /usr/lib64/python2.7/robotparser.pyR � s c C s d S( N( NN( R ( R R t realm( ( s# /usr/lib64/python2.7/robotparser.pyt prompt_user_passwd� s c C s( | | _ t j j | | | | | | � S( N( R R- RM t http_error_default( R R t fpR t errmsgt headers( ( s# /usr/lib64/python2.7/robotparser.pyRQ � s ( RC RD R RP RQ ( ( ( s# /usr/lib64/python2.7/robotparser.pyR � s ( ( ( ( RE R R- t __all__R R0 R( RM R ( ( ( s# /usr/lib64/python2.7/robotparser.pyt <module> s �$
Save File
Cancel