← Back
Editing: pool.cpython-35.pyc
��Yf�d � @ s� d d g Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z m Z d Z d Z d Z e j � Z d d � Z d d � Z Gd d � d e � Z Gd d � d � Z d d � Z Gd d � d e � Z d f d d d d � Z d d � Z Gd d � d e � Z Gd d � d e � Z e Z Gd d � d e � Z Gd d � d e � Z Gd d! � d! e � Z Gd" d � d e � Z! d S)#�Pool� ThreadPool� N� )�util)� get_context� TimeoutError� c C s t t | � � S)N)�list�map)�args� r �9/opt/alt/python35/lib64/python3.5/multiprocessing/pool.py�mapstar+ s r c C s t t j | d | d � � S)Nr r )r � itertools�starmap)r r r r � starmapstar. s r c @ s( e Z d Z d d � Z d d � Z d S)�RemoteTracebackc C s | | _ d S)N)�tb)�selfr r r r �__init__6 s zRemoteTraceback.__init__c C s | j S)N)r )r r r r �__str__8 s zRemoteTraceback.__str__N)�__name__� __module__� __qualname__r r r r r r r 5 s r c @ s( e Z d Z d d � Z d d � Z d S)�ExceptionWithTracebackc C sD t j t | � | | � } d j | � } | | _ d | | _ d S)N� z """ %s""")� traceback�format_exception�type�join�excr )r r r r r r r < s z ExceptionWithTraceback.__init__c C s t | j | j f f S)N)� rebuild_excr r )r r r r � __reduce__A s z!ExceptionWithTraceback.__reduce__N)r r r r r" r r r r r ; s r c C s t | � | _ | S)N)r � __cause__)r r r r r r! D s r! c s@ e Z d Z d Z � f d d � Z d d � Z d d � Z � S)�MaybeEncodingErrorzVWraps possible unpickleable errors, so they can be safely sent through the socket.c sA t | � | _ t | � | _ t t | � j | j | j � d S)N)�reprr �value�superr$ r )r r r& )� __class__r r r P s zMaybeEncodingError.__init__c C s d | j | j f S)Nz(Error sending result: '%s'. Reason: '%s')r& r )r r r r r U s zMaybeEncodingError.__str__c C s d | j j | f S)Nz<%s: %s>)r( r )r r r r �__repr__Y s zMaybeEncodingError.__repr__)r r r �__doc__r r r) r r )r( r r$ L s r$ Fc ' C s7 | d k s0 t | � t k r* | d k s0 t � | j } | j } t | d � rk | j j � | j j � | d k r� | | � d } x�| d k s� | r!| | k r!y | � } Wn& t t f k r� t j d � PYn X| d k r� t j d � P| \ } } } } } y d | | | � f } WnU t k r}} z5 | r_| t k r_t | | j � } d | f } WYd d } ~ Xn Xy | | | | f � Wnb t k r�} zB t | | d � } t j d | � | | | d | f f � WYd d } ~ Xn Xd } } } } } } | d 7} q� Wt j d | � d S) Nr �_writerz)worker got EOFError or OSError -- exitingz worker got sentinel -- exitingTFr z0Possible encoding error while sending result: %sz worker exiting after %d tasks)r �int�AssertionError�put�get�hasattrr+ �close�_reader�EOFError�OSErrorr �debug� Exception�_helper_reraises_exceptionr � __traceback__r$ )�inqueue�outqueue� initializer�initargsZmaxtasksZwrap_exceptionr. r/ Z completed�task�job�i�funcr �kwds�result�e�wrappedr r r �worker] sF 0 ! ,rE c C s | � d S)z@Pickle-able helper function for use by _guarded_task_generation.Nr )Zexr r r r7 � s r7 c @ s� e Z d Z d Z d Z d d � Z d d f d d d d � Z d d � Z d d � Z d d � Z d d � Z f i d d � Z d d d � Z d d d � Z d d d d d � Z d d � Z d d d � Z d d d � Z f i d d d d � Z d d d d! d" � Z d d d d# d$ � Z e d% d& � � Z e d' d( � � Z e d) d* � � Z e d+ d, � � Z d- d. � Z d/ d0 � Z d1 d2 � Z d3 d4 � Z e d5 d6 � � Z e d7 d8 � � Z d9 d: � Z! d; d< � Z" d S)=r zS Class which supports an async version of applying functions to arguments. Tc O s | j j | | � S)N)�_ctx�Process)r r rA r r r rG � s z Pool.ProcessNc C s# | p t � | _ | j � t j � | _ i | _ t | _ | | _ | | _ | | _ | d k rv t j � ps d } | d k r� t d � � | d k r� t | � r� t d � � | | _ g | _ | j � t j d t j d | f � | _ d | j _ t | j _ | j j � t j d t j d | j | j | j | j | j f � | _ d | j _ t | j _ | j j � t j d t j d | j | j | j f � | _! d | j! _ t | j! _ | j! j � t" j# | | j$ d | j | j% | j | j | j | j | j! | j f d d �| _& d S) Nr z&Number of processes must be at least 1z initializer must be a callable�targetr TZ exitpriority� )'r rF � _setup_queues�queue�Queue� _taskqueue�_cache�RUN�_state�_maxtasksperchild� _initializer� _initargs�os� cpu_count� ValueError�callable� TypeError� _processes�_pool�_repopulate_pool� threadingZThreadr �_handle_workers�_worker_handler�daemon�start� _handle_tasks� _quick_put� _outqueue� _task_handler�_handle_results� _quick_get�_result_handlerr ZFinalize�_terminate_pool�_inqueue� _terminate)r � processesr; r< Zmaxtasksperchild�contextr r r r � sT z Pool.__init__c C sw d } xj t t t | j � � � D]M } | j | } | j d k r" t j d | � | j � d } | j | =q" W| S)z�Cleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. FNzcleaning up worker %dT)�reversed�range�lenrZ �exitcoder r5 r )r Zcleanedr? rE r r r �_join_exited_workers� s " zPool._join_exited_workersc C s� x� t | j t | j � � D]� } | j d t d | j | j | j | j | j | j f � } | j j | � | j j d d � | _ d | _ | j � t j d � q Wd S)z�Bring the number of pool processes up to the specified number, for use after reaping workers which have exited. rH r rG Z PoolWorkerTz added workerN)rn rY ro rZ rG rE ri rc rR rS rQ �_wrap_exception�append�name�replacer_ r` r r5 )r r? �wr r r r[ � s # zPool._repopulate_poolc C s | j � r | j � d S)zEClean up any exited workers and start replacements for them. N)rq r[ )r r r r �_maintain_pool� s zPool._maintain_poolc C sL | j j � | _ | j j � | _ | j j j | _ | j j j | _ d S)N) rF Z SimpleQueueri rc r+ �sendrb r2 �recvrf )r r r r rJ � s zPool._setup_queuesc C s. | j t k s t � | j | | | � j � S)z6 Equivalent of `func(*args, **kwds)`. )rP rO r- � apply_asyncr/ )r r@ r rA r r r �apply� s z Pool.applyc C s | j | | t | � j � S)zx Apply `func` to each element in `iterable`, collecting the results in a list that is returned. )� _map_asyncr r/ )r r@ �iterable� chunksizer r r r s zPool.mapc C s | j | | t | � j � S)z� Like `map()` method but the elements of the `iterable` are expected to be iterables as well and will be unpacked as arguments. Hence `func` and (a, b) becomes func(a, b). )r| r r/ )r r@ r} r~ r r r r s z Pool.starmapc C s | j | | t | | | � S)z= Asynchronous version of `starmap()` method. )r| r )r r@ r} r~ �callback�error_callbackr r r � starmap_async s zPool.starmap_asyncc c s� y> d } x1 t | � D]# \ } } | | | | f i f Vq WWn@ t k r� } z | | d t | f i f VWYd d } ~ Xn Xd S)z�Provides a generator of tasks for imap and imap_unordered with appropriate handling for iterables which throw exceptions during iteration.r N���)� enumerater6 r7 )r Z result_jobr@ r} r? �xrC r r r �_guarded_task_generation s z Pool._guarded_task_generationr c C s� | j t k r t d � � | d k re t | j � } | j j | j | j | | � | j f � | S| d k sw t � t j | | | � } t | j � } | j j | j | j t | � | j f � d d � | D� Sd S)zP Equivalent of `map()` -- can be MUCH slower than `Pool.map()`. zPool not runningr c s s" | ] } | D] } | Vq q d S)Nr )�.0�chunk�itemr r r � <genexpr>@ s z Pool.imap.<locals>.<genexpr>N)rP rO rV � IMapIteratorrN rM r. r� �_job� _set_lengthr- r � _get_tasksr )r r@ r} r~ rB � task_batchesr r r �imap' s" z Pool.imapc C s� | j t k r t d � � | d k re t | j � } | j j | j | j | | � | j f � | S| d k sw t � t j | | | � } t | j � } | j j | j | j t | � | j f � d d � | D� Sd S)zL Like `imap()` method but ordering of results is arbitrary. zPool not runningr c s s" | ] } | D] } | Vq q d S)Nr )r� r� r� r r r r� [ s z&Pool.imap_unordered.<locals>.<genexpr>N)rP rO rV �IMapUnorderedIteratorrN rM r. r� r� r� r- r r� r )r r@ r} r~ rB r� r r r �imap_unorderedB s" zPool.imap_unorderedc C s_ | j t k r t d � � t | j | | � } | j j | j d | | | f g d f � | S)z; Asynchronous version of `apply()` method. zPool not runningr N)rP rO rV � ApplyResultrN rM r. r� )r r@ r rA r r� rB r r r rz ] s +zPool.apply_asyncc C s | j | | t | | | � S)z9 Asynchronous version of `map()` method. )r| r )r r@ r} r~ r r� r r r � map_asynch s zPool.map_asyncc C s� | j t k r t d � � t | d � s6 t | � } | d k rz t t | � t | j � d � \ } } | rz | d 7} t | � d k r� d } t j | | | � } t | j | t | � | d | �} | j j | j | j | | � d f � | S)zY Helper function to implement map, starmap and their async counterparts. zPool not running�__len__N� r r r� )rP rO rV r0 r �divmodro rZ r r� � MapResultrN rM r. r� r� ) r r@ r} Zmapperr~ r r� Zextrar� rB r r r r| p s& ( zPool._map_asyncc C sr t j � } xB | j t k s6 | j rP | j t k rP | j � t j d � q W| j j d � t j d � d S)Ng�������?zworker handler exiting) r\ �current_threadrP rO rN � TERMINATErw �time�sleeprM r. r r5 )�pool�threadr r r r] � s * zPool._handle_workersc C s� t j � } x+t | j d � D] \ } } d } z� x� | D]� } | j rX t j d � Py | | � Wq; t k r� } zN | d d � \ } } y | | j | d | f � Wn t k r� Yn XWYd d } ~ Xq; Xq; W| rt j d � | r� | d n d } | | d � w PWd d } } } Xq Wt j d � yF t j d � | j d � t j d � x | D] } | d � qkWWn t k r�t j d � Yn Xt j d � d S) Nz'task handler found thread._state != RUNr Fzdoing set_length()r ztask handler got sentinelz/task handler sending sentinel to result handlerz(task handler sending sentinel to workersz/task handler got OSError when sending sentinelsztask handler exitingr� ) r\ r� �iterr/ rP r r5 r6 �_set�KeyErrorr. r4 ) � taskqueuer. r: r� �cacher� ZtaskseqZ set_lengthr= rC r> �idx�pr r r ra � sB zPool._handle_tasksc C s t j � } x� y | � } Wn) t t f k rG t j d � d SYn X| j rt | j t k sf t � t j d � P| d k r� t j d � P| \ } } } y | | j | | � Wn t k r� Yn Xd } } } q Wx� | r�| j t k r�y | � } Wn) t t f k r,t j d � d SYn X| d k rIt j d � q� | \ } } } y | | j | | � Wn t k r�Yn Xd } } } q� Wt | d � r t j d � y2 x+ t d � D] } | j j � s�P| � q�WWn t t f k r�Yn Xt j d t | � | j � d S) Nz.result handler got EOFError/OSError -- exitingz,result handler found thread._state=TERMINATEzresult handler got sentinelz&result handler ignoring extra sentinelr2 z"ensuring that outqueue is not full� z7result handler exiting: len(cache)=%s, thread._state=%s)r\ r� r4 r3 r r5 rP r� r- r� r� r0 rn r2 �pollro )r: r/ r� r� r= r> r? �objr r r re � s\ zPool._handle_resultsc c sD t | � } x1 t t j | | � � } | s1 d S| | f Vq Wd S)N)r� �tupler �islice)r@ �it�sizer� r r r r� s zPool._get_tasksc C s t d � � d S)Nz:pool objects cannot be passed between processes or pickled)�NotImplementedError)r r r r r" s zPool.__reduce__c C s5 t j d � | j t k r1 t | _ t | j _ d S)Nz closing pool)r r5 rP rO �CLOSEr^ )r r r r r1 s z Pool.closec C s0 t j d � t | _ t | j _ | j � d S)Nzterminating pool)r r5 r� rP r^ rj )r r r r � terminate s zPool.terminatec C sq t j d � | j t t f k s( t � | j j � | j j � | j j � x | j D] } | j � qY Wd S)Nz joining pool) r r5 rP r� r� r- r^ r rd rg rZ )r r� r r r r s z Pool.joinc C sZ t j d � | j j � x9 | j � rU | j j � rU | j j � t j d � q Wd S)Nz7removing tasks from inqueue until task handler finishedr ) r r5 Z_rlock�acquire�is_aliver2 r� ry r� r� )r9 � task_handlerr� r r r �_help_stuff_finish( s zPool._help_stuff_finishc C s� t j d � t | _ t | _ t j d � | j | | t | � � | j � si t | � d k si t � t | _ | j d � t j d � t j � | k r� | j � | r� t | d d � r� t j d � x' | D] } | j d k r� | j � q� Wt j d � t j � | k r!| j � t j d � t j � | k rJ| j � | r�t | d d � r�t j d � x8 | D]0 } | j � rwt j d | j � | j � qwWd S) Nzfinalizing poolz&helping task handler/workers to finishr zjoining worker handlerr� zterminating workerszjoining task handlerzjoining result handlerzjoining pool workerszcleaning up worker %d)r r5 r� rP r� ro r� r- r. r\ r� r r0 rp r� �pid) �clsr� r9 r: r� Zworker_handlerr� Zresult_handlerr� r� r r r rh 1 s8 $ zPool._terminate_poolc C s | S)Nr )r r r r � __enter___ s zPool.__enter__c C s | j � d S)N)r� )r �exc_typeZexc_valZexc_tbr r r �__exit__b s z Pool.__exit__)#r r r r* rr rG r rq r[ rw rJ r{ r r r� r� r� r� rz r� r| � staticmethodr] ra re r� r" r1 r� r r� � classmethodrh r� r� r r r r r � sF 8 .< . c @ s^ e Z d Z d d � Z d d � Z d d � Z d d d � Z d d d � Z d d � Z d S)r� c C sJ t j � | _ t t � | _ | | _ | | _ | | _ | | | j <d S)N) r\ ZEvent�_event�next� job_counterr� rN � _callback�_error_callback)r r� r r� r r r r k s zApplyResult.__init__c C s | j j � S)N)r� Zis_set)r r r r �readys s zApplyResult.readyc C s | j � s t � | j S)N)r� r- �_success)r r r r � successfulv s zApplyResult.successfulNc C s | j j | � d S)N)r� �wait)r �timeoutr r r r� z s zApplyResult.waitc C s< | j | � | j � s t � | j r/ | j S| j � d S)N)r� r� r r� �_value)r r� r r r r/ } s zApplyResult.getc C su | \ | _ | _ | j r4 | j r4 | j | j � | j rW | j rW | j | j � | j j � | j | j =d S)N)r� r� r� r� r� �setrN r� )r r? r� r r r r� � s zApplyResult._set) r r r r r� r� r� r/ r� r r r r r� i s r� c @ s( e Z d Z d d � Z d d � Z d S)r� c C s� t j | | | d | �d | _ d g | | _ | | _ | d k rj d | _ | j j � | | j =n | | t | | � | _ d S)Nr� Tr ) r� r r� r� � _chunksize� _number_leftr� r� r� �bool)r r� r~ �lengthr r� r r r r � s zMapResult.__init__c C s� | \ } } | r� | | j | | j | d | j � <| j d 8_ | j d k r� | j rn | j | j � | j | j =| j j � nE d | _ | | _ | j r� | j | j � | j | j =| j j � d S)Nr r F) r� r� r� r� rN r� r� r� r� r� )r r? Zsuccess_result�successrB r r r r� � s % zMapResult._setN)r r r r r� r r r r r� � s r� c @ sU e Z d Z d d � Z d d � Z d d d � Z e Z d d � Z d d � Z d S) r� c C sk t j t j � � | _ t t � | _ | | _ t j � | _ d | _ d | _ i | _ | | | j <d S)Nr )r\ Z ConditionZLock�_condr� r� r� rN � collections�deque�_items�_index�_length� _unsorted)r r� r r r r � s zIMapIterator.__init__c C s | S)Nr )r r r r �__iter__� s zIMapIterator.__iter__Nc C s� | j �� y | j j � } Wn� t k r� | j | j k rE t � | j j | � y | j j � } Wn0 t k r� | j | j k r� t � t � Yn XYn XWd QRX| \ } } | r� | S| � d S)N) r� r� �popleft� IndexErrorr� r� � StopIterationr� r )r r� r� r� r& r r r r� � s" zIMapIterator.nextc C s� | j �� | j | k r� | j j | � | j d 7_ xJ | j | j k r� | j j | j � } | j j | � | j d 7_ q; W| j j � n | | j | <| j | j k r� | j | j =Wd QRXd S)Nr ) r� r� r� rs r� �pop�notifyr� rN r� )r r? r� r r r r� � s zIMapIterator._setc C sJ | j �: | | _ | j | j k r? | j j � | j | j =Wd QRXd S)N)r� r� r� r� rN r� )r r� r r r r� � s zIMapIterator._set_length) r r r r r� r� �__next__r� r� r r r r r� � s r� c @ s e Z d Z d d � Z d S)r� c C s` | j �P | j j | � | j d 7_ | j j � | j | j k rU | j | j =Wd QRXd S)Nr )r� r� rs r� r� r� rN r� )r r? r� r r r r� � s zIMapUnorderedIterator._setN)r r r r� r r r r r� � s r� c @ s[ e Z d Z d Z e d d � � Z d d f d d � Z d d � Z e d d � � Z d S) r Fc O s d d l m } | | | � S)Nr )rG )ZdummyrG )r rA rG r r r rG s zThreadPool.ProcessNc C s t j | | | | � d S)N)r r )r rk r; r< r r r r s zThreadPool.__init__c C s@ t j � | _ t j � | _ | j j | _ | j j | _ d S)N)rK rL ri rc r. rb r/ rf )r r r r rJ s zThreadPool._setup_queuesc C sF | j �6 | j j � | j j d g | � | j j � Wd QRXd S)N)Z not_emptyrK �clear�extendZ notify_all)r9 r� r� r r r r� s z ThreadPool._help_stuff_finish) r r r rr r� rG r rJ r� r r r r r s )"�__all__r\ rK r r� rT r� r r r r r rO r� r� �countr� r r r6 r r r! r$ rE r7 �objectr r� Z AsyncResultr� r� r� r r r r r �<module> s< * � �&%@
Save File
Cancel