D7net
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
usr
/
lib
/
python2.7
/
site-packages
/
urlgrabber
/
Filename :
byterange.pyc
back
Copy
� y]Rc @ s� d d l Z d d l Z d d l Z d d l Z d d l Z d Z y d d l m Z Wn# e k r{ Z d d l m Z n Xd e f d � � YZ d e j f d � � YZ d e f d � � YZ d d d � � YZ d e j f d � � YZ d d l m Z m Z m Z m Z m Z m Z m Z d d l Z d d l Z d d l Z d d l Z d d l Z d e j f d � � YZ d e j! f d � � YZ! d a" d � Z# d � Z$ d � Z% d S( i����N( t StringIOt RangeErrorc B s e Z d Z RS( s6 Error raised when an unsatisfiable range is requested.( t __name__t __module__t __doc__( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR " s t HTTPRangeHandlerc B s e Z d Z d � Z d � Z RS( s� Handler that enables HTTP Range headers. This was extremely simple. The Range header is a HTTP feature to begin with so all this class does is tell urllib2 that the "206 Partial Content" response from the HTTP server is what we expected. Example: import urllib2 import byterange range_handler = range.HTTPRangeHandler() opener = urllib2.build_opener(range_handler) # install it urllib2.install_opener(opener) # create Request and set Range header req = urllib2.Request('http://www.python.org/') req.header['Range'] = 'bytes=30-50' f = urllib2.urlopen(req) c C s1 t j | | | j � � } | | _ | | _ | S( N( t urllibt addinfourlt get_full_urlt codet msg( t selft reqt fpR R t hdrst r( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt http_error_206> s c C s t d d � � d S( Ni s Requested Range Not Satisfiable( R ( R R R R R R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt http_error_416E s ( R R R R R ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR & s t HTTPSRangeHandlerc B s e Z d Z d � Z d � Z RS( s! Range Header support for HTTPS. c C s | j | | | | | � S( N( R ( R R R R R R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt https_error_206L s c C s | j | | | | | � d S( N( t https_error_416( R R R R R R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR O s ( R R R R R ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR I s t RangeableFileObjectc B sh e Z d Z d � Z d � Z d � Z d d � Z d d � Z d d � Z d � Z d � Z d � Z RS( s" File object wrapper to enable raw range handling. This was implemented primarilary for handling range specifications for file:// urls. This object effectively makes a file object look like it consists only of a range of bytes in the stream. Examples: # expose 10 bytes, starting at byte position 20, from # /etc/aliases. >>> fo = RangeableFileObject(file('/etc/passwd', 'r'), (20,30)) # seek seeks within the range (to position 23 in this case) >>> fo.seek(3) # tell tells where your at _within the range_ (position 3 in # this case) >>> fo.tell() # read EOFs if an attempt is made to read past the last # byte in the range. the following will return only 7 bytes. >>> fo.read(30) c C s> | | _ t | � \ | _ | _ d | _ | j | j � d S( s Create a RangeableFileObject. fo -- a file like object. only the read() method need be supported but supporting an optimized seek() is preferable. rangetup -- a (firstbyte,lastbyte) tuple specifying the range to work over. The file object provided is assumed to be at byte offset 0. i N( t fot range_tuple_normalizet firstbytet lastbytet realpost _do_seek( R R t rangetup( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt __init__g s c C s/ t | j | � r" t | j | � St | � d S( s� This effectively allows us to wrap at the instance level. Any attribute not found in _this_ object will be searched for in self.fo. This includes methods.N( t hasattrR t getattrt AttributeError( R t name( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt __getattr__u s c C s | j | j S( s5 Return the position within the range. This is different from fo.seek in that position 0 is the first byte position of the range tuple. For example, if this object was created with a range tuple of (500,899), tell() will return 0 when at byte position 500 of the file. ( R R ( R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt tell} s i c C s� | d k s t � | d k r. | j | } n7 | d k rJ | j | } n | d k re t d � � n | j r� | | j k r� | j } n | j | | j � d S( se Seek within the byte range. Positioning is identical to that described under tell(). i i i s$ seek from end of file not supported.N( i i i ( t AssertionErrorR R t IOErrorR R ( R t offsett whencet realoffset( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt seek� s i����c C s: | j | � } | j j | � } | j t | � 7_ | S( s` Read within the range. This method will limit the size read based on the range. ( t _calc_read_sizeR t readR t len( R t sizet rslt( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR+ � s c C s: | j | � } | j j | � } | j t | � 7_ | S( sf Read lines within the range. This method will limit the size read based on the range. ( R* R t readlineR R, ( R R- R. ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR/ � s c C sX | j rT | d k rA | j | | j k rQ | j | j } qQ qT | j | j } n | S( sS Handles calculating the amount of data to read based on the range. i����( R R ( R R- ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR* � s c C s^ | d k s t � t | j d � s4 | j | � n | j j | j | � | j | 7_ d S( s� Seek based on whether wrapped object supports seek(). offset is relative to the current position (self.realpos). i R) N( R$ R R t _poor_mans_seekR) R ( R R& ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR � s c C s� d } d } xm | | k r{ | | | k r8 | | } n | j j | � } t | � | k rn t d d � � n | | 7} q Wd S( s� Seek by calling the wrapped file objects read() method. This is used for file like objects that do not have native seek support. The wrapped objects read() method is called to manually seek to the desired position. offset -- read this number of bytes from the wrapped file object. raise RangeError if we encounter EOF before reaching the specified offset. i i i s Requested Range Not SatisfiableN( R R+ R, R ( R R& t post bufsizet buf( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR0 � s ( R R R R R" R# R) R+ R/ R* R R0 ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR R s t FileRangeHandlerc B s e Z d Z d � Z RS( s~ FileHandler subclass that adds Range support. This class handles Range headers exactly like an HTTP server would. c C s� d d l } d d l } | j � } | j � } t j | � } t j | � } | t j } t j | t j � } | j | � d } | r� t j | � \ } } | s� t j | � | j � k r� t j d � � q� n t | d � } | j j d d � } t | � } | d k st � | r�| \ } } | d k r>| } n | d k sb| | k sb| | k rtt d d � � n | | } t | | | f � } n | j t d | p�d | | f � � } t j | | d | � S( Ni����i s file not on local hostt rbt Ranget i s Requested Range Not Satisfiables6 Content-Type: %s Content-Length: %d Last-modified: %s s text/plains file:( ( t mimetypest mimetoolst get_hostt get_selectorR t url2pathnamet ost statt ST_SIZEt rfc822t formatdatet ST_MTIMEt guess_typet splitportt sockett gethostbynamet get_namest urllib2t URLErrort opent headerst gett Nonet range_header_to_tupleR$ R R t MessageR R ( R R R8 R9 t hostt filet localfilet statsR- t modifiedt mtypet portR t branget fbt lbRK ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt open_local_file� s: ! $ ( R R R RZ ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR4 � s ( RD t splitusert splitpasswdt splitattrt unquotet addclosehookR t FTPRangeHandlerc B s e Z d � Z d � Z RS( c C s� | j � } | s t d � n t | � \ } } | d k rH t j } n t | � } t | � \ } } | r� t | � \ } } n d } t | � } t | p� d � } t | p� d � } y t j | � } Wn% t j k r� } t j | � � n Xt | j � � \ } } | j d � } t t | � } | d | d } } | r[| d r[| d } n y$| j | | | | | � } | r�d p�d } xM | D]E } t | � \ } } | j � d k r�| d k r�| j � } q�q�Wd } t | j j d d � � } | d k st � | r8| \ } } | d k r8| } q8n | j | | | � \ } } | r�| \ } } | d k r�| d k s�| d k r�t d d � � n | } | | } | d k r�t d d � � q�q�| | } t | d | f � } n d } t j | j � � d } | r$| d | 7} n | d k rM| d k rM| d | 7} n t | � } t! j" | � } t# | | | j � � SWn2 t j$ k r�} t d | f t% j& � d � n Xd S( Ns ftp errors no host givenR7 t /i����i i t It Dt typet at At it dR6 i s@ Requested Range Not Satisfiable due to unobtainable file length.s Requested Range Not Satisfiables Content-Type: %s s Content-Length: %d i ( s ftp errors no host given( Re Rf Rg Rb Rh Rc ( (' R: R% RD RM t ftplibt FTP_PORTt intR[ R\ R^ RE RF t errorRH RI R] R; t splitt mapt connect_ftpt lowert upperRN RK RL R$ t retrfileR R R8 RC R R R9 RO R t all_errorst syst exc_info( R R RP RV t usert passwdR t patht attrst dirsRQ t fwRd t attrt valuet restt range_tupRX RY R t retrlenRK RU t sf( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt ftp_open s| c C s t | | | | | � } | S( N( t ftpwrapper( R Rv Rw RP RV Rz R{ ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRo X s ( R R R� Ro ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR` s NR� c B s e Z d d � Z RS( c C s | j � | d k r% d } d } n d | } d } y | j j | � Wn. t j k ry | j � | j j | � n Xd } | r�| r�y | j j | � Wn2 t j k r� } t d | f t j � d � n X| j j | � y# d | } | j j | | � } Wq�t j k r�} t | � d d k rk| j | | � \ } } t | | d f � } | | f St | � d d k r�t d | f t j � d � q�q�Xn | s�| j j d � | r�d | } n d } | j j | � } n d | _ t | d j d � | j � | d f S( NRh Rc s TYPE Ai s TYPE i s ftp errori s RETR i t 501R7 t 550s LIST t LISTR5 ( Rh Rc ( t endtransfert ftpt voidcmdRi Rs t initRM t nlstt error_permR% Rt Ru t ntransfercmdt strRr R t busyR_ t makefile( R RQ Rd R~ t cmdt isdirt connt reasonR R� ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRr a sJ & N( R R RM Rr ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR� \ s c C s� | d k r d St d k r: d d l } | j d � a n t j | � } | r� t | j d d � � } | r� | d r� | d | d d f } n | Sd S( s� Get a (firstbyte,lastbyte) tuple from a Range header value. Range headers have the form "bytes=<firstbyte>-<lastbyte>". This function pulls the firstbyte and lastbyte values and returns a (firstbyte,lastbyte) tuple. If lastbyte is not specified in the header value, it is returned as an empty string in the tuple. Return None if range_header is None Return () if range_header does not conform to the range spec pattern. i����Ns ^bytes=(\d{1,})-(\d*)i i i ( ( RM t _rangeret ret compilet matchR t group( t range_headerR� R� t tup( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRN � s c C sS | d k r d St | � } | rO | d rG | d | d d f } n d | Sd S( s� Convert a range tuple to a Range header value. Return a string of the form "bytes=<firstbyte>-<lastbyte>" or None if no range is needed. i i s bytes=%s-%sN( RM R ( R ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt range_tuple_to_header� s c C s� | d k r d S| d } | d k r/ d } n t | � } y | d } Wn t k rb d } n1 X| d k rx d } n | d k r� t | � } n | | f d k r� d S| | k r� t d d | | f � � n | | f S( s7 Normalize a (first_byte,last_byte) range tuple. Return a tuple whose first element is guaranteed to be an int and whose second element will be '' (meaning: the last byte) or an int. Finally, return None if the normalized tuple == (0,'') as that is equivalent to retrieving the entire file. i R7 i i s Invalid byte range: %s-%sN( NR7 ( i R7 ( RM Rk t IndexErrorR ( R RX RY ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR � s& ( (& R= R> R RH R@ RM t DEBUGt cStringIOR t ImportErrorR R% R t BaseHandlerR R R t FileHandlerR4 RD R[ R\ R] R^ R_ R Ri RE Rt R8 R9 t FTPHandlerR` R� R� RN R� R ( ( ( s8 /usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt <module> s4 # �+4 S4