D7net
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
usr
/
lib64
/
python2.7
/
site-packages
/
lxml
/
html
/
Filename :
clean.pyo
back
Copy
� �gc @ s3 d Z d d l Z d d l Z y d d l m Z Wn! e k rU d d l m Z n Xd d l m Z d d l m Z d d l m Z m Z m Z d d l m Z m Z y e Wn e k r� e Z n Xy e Wn e k r� e Z n Xy e Wn e k r e Z n Xy e Wn e k r5e e f Z n Xd d d d d d d g Z e j d e j e j B� Z e j d e j � Z e j d e j � Z e j d � j Z e j d e j e j B� Z e j! d � Z" e j! d d i e d 6�Z# d e$ f d � � YZ% e% � Z& e& j' Z' e j d e j � e j d e j � g Z( d d d d d d g Z) e j d! e j � e j d" e j � e j d# � g Z* d$ g Z+ e( e) e* e+ d% � Z, d& � Z- d' � Z. e, j e. _ d d d g Z/ d( g Z0 d) e/ e0 e d* � d+ � Z1 d, � Z2 d- � Z3 e j d. e j � Z4 d/ � Z5 d S(0 sc A cleanup tool for HTML. Removes unwanted tags and content. See the `Cleaner` class for details. i����N( t urlsplit( t etree( t defs( t fromstringt tostringt XHTML_NAMESPACE( t xhtml_to_htmlt _transform_resultt clean_htmlt cleant Cleanert autolinkt autolink_htmlt word_breakt word_break_htmls expression\s*\(.*?\)s @\s*imports? \s*(?:javascript|jscript|livescript|vbscript|data|about|mocha):s \s+s \[if[\s\n\r]+.*?][\s\n\r]*>s descendant-or-self::*[@style]s� descendant-or-self::a [normalize-space(@href) and substring(normalize-space(@href),1,1) != '#'] |descendant-or-self::x:a[normalize-space(@href) and substring(normalize-space(@href),1,1) != '#']t namespacest xc B sI e Z d Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z d Z d Z d Z e Z e Z e j Z e Z d Z e d d g � Z d � Z e d d d d d d d g d d d d d d d d � Z d � Z d � Z! d � Z" d � Z# d � Z$ d d � Z% d � Z& e' j( d e' j) � j* Z+ d � Z, d � Z- RS( s� Instances cleans the document of each of the possible offending elements. The cleaning is controlled by attributes; you can override attributes in a subclass, or set them in the constructor. ``scripts``: Removes any ``<script>`` tags. ``javascript``: Removes any Javascript, like an ``onclick`` attribute. Also removes stylesheets as they could contain Javascript. ``comments``: Removes any comments. ``style``: Removes any style tags or attributes. ``links``: Removes any ``<link>`` tags ``meta``: Removes any ``<meta>`` tags ``page_structure``: Structural parts of a page: ``<head>``, ``<html>``, ``<title>``. ``processing_instructions``: Removes any processing instructions. ``embedded``: Removes any embedded objects (flash, iframes) ``frames``: Removes any frame-related tags ``forms``: Removes any form tags ``annoying_tags``: Tags that aren't *wrong*, but are annoying. ``<blink>`` and ``<marquee>`` ``remove_tags``: A list of tags to remove. Only the tags will be removed, their content will get pulled up into the parent tag. ``kill_tags``: A list of tags to kill. Killing also removes the tag's content, i.e. the whole subtree, not just the tag itself. ``allow_tags``: A list of tags to include (default include all). ``remove_unknown_tags``: Remove any tags that aren't standard parts of HTML. ``safe_attrs_only``: If true, only include 'safe' attributes (specifically the list from the feedparser HTML sanitisation web site). ``safe_attrs``: A set of attribute names to override the default list of attributes considered 'safe' (when safe_attrs_only=True). ``add_nofollow``: If true, then any <a> tags will have ``rel="nofollow"`` added to them. ``host_whitelist``: A list or set of hosts that you can use for embedded content (for content like ``<object>``, ``<link rel="stylesheet">``, etc). You can also implement/override the method ``allow_embedded_url(el, url)`` or ``allow_element(el)`` to implement more complex rules for what can be embedded. Anything that passes this test will be shown, regardless of the value of (for instance) ``embedded``. Note that this parameter might not work as intended if you do not make the links absolute before doing the cleaning. Note that you may also need to set ``whitelist_tags``. ``whitelist_tags``: A set of tags that can be included with ``host_whitelist``. The default is ``iframe`` and ``embed``; you may wish to include other tags like ``script``, or you may want to implement ``allow_embedded_url`` for more control. Set to None to include all tags. This modifies the document *in place*. t iframet embedc K sY xR | j � D]D \ } } t | | � sA t d | | f � � n t | | | � q Wd S( Ns Unknown parameter: %s=%r( t itemst hasattrt TypeErrort setattr( t selft kwt namet value( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyt __init__� s t scriptt srct linkt hreft applett codet objectt layert ac C s� t | d � r | j � } n t | � x | j d � D] } d | _ q8 W| j sd | j | � n t | j ps d( � } t | j p� d) � } t | j p� d* � } | j r� | j d � n | j r$t | j � } xM | j � D]< } | j } x* | j � D] } | | k r� | | =q� q� Wq� Wn | j r�| j oE| j t j k s�xP | j � D]? } | j } x- | j � D] } | j d � rq| | =qqqqWqUWn | j | j d t �| j s�x� t | � D]t } | j d � } t j d | � } t j d | � } | j | � r | j d =q�| | k r�| j d | � q�q�Wx� t | j d � � D]� } | j d d � j � j � d k r�| j! � qUn | j" p�d } t j d | � } t j d | � } | j | � r�d | _" qU| | k rU| | _" qUqUWq�n | j s| j# r$| j t$ j% � n | j# r@| j t$ j& � n | j ri| j d � t$ j' | d � n | j( r�| j d � nr | j s�| j r�x] t | j d � � D]C } d | j d d � j � k r�| j) | � s�| j! � q�q�q�Wn | j* r | j d � n | j+ r&| j, d+ � n | j- r�xv t | j d � � D]_ } t } | j. � } x+ | d' k r�| j d, k r�| j. � } q`W| d' k rE| j! � qEqEW| j, d- � | j, d. � n | j0 r�| j, t j1 � n | j2 r| j d � | j, d/ � n | j3 r | j, d0 � n g } g } x| | j � D]n } | j | k rs| j) | � rcq9n | j4 | � q9| j | k r9| j) | � r�q9n | j4 | � q9q9W| r�| d | k r�| j5 d � } d! | _ | j j6 � nM | r6| d | k r6| j5 d � } | j d k r)d! | _ n | j6 � n | j7 � x | D] } | j! � qGWx | D] } | j8 � qbW| j } | j9 r�| r�t: d" � � n t t j; � } n | rPg } x3 | j � D]% } | j | k r�| j4 | � q�q�W| rP| d | k r/| j5 d � } d! | _ | j j6 � n x | D] } | j8 � q6WqPn | j< r�x� t= | � D]r } | j> | � sf| j d � } | r�d# | k r�d$ d% | k r�qfn d& | } n d# } | j d | � qfqfWn d' S(1 s& Cleans the document. t getroott imaget imgR t ont resolve_base_hreft stylet t types text/javascripts /* deleted */R t stylesheett relt metat headt htmlt titlet paramR R" R R R# t formt buttont inputt selectt textareat blinkt marqueei t divsI It does not make sense to pass in both allow_tags and remove_unknown_tagst nofollows nofollow s %s s %s nofollowN( ( ( ( s heads htmls title( s applets object( s applet( s iframes embeds layers objectR3 ( R5 s inputR7 R8 ( R9 R: (? R R% R t itert tagt commentst kill_conditional_commentst sett kill_tagst remove_tagst allow_tagst scriptst addt safe_attrs_onlyt safe_attrst attribt keyst javascriptR t startswitht rewrite_linkst _remove_javascript_linkt FalseR* t _find_styled_elementst gett _css_javascript_ret subt _css_import_ret _has_sneaky_javascriptt listt lowert stript drop_treet textt processing_instructionsR t Commentt ProcessingInstructiont strip_attributest linkst allow_elementR/ t page_structuret updatet embeddedt getparentt Nonet framest frame_tagst formst annoying_tagst appendt popt cleart reverset drop_tagt remove_unknown_tagst ValueErrort tagst add_nofollowt _find_external_linkst allow_follow( R t doct elRB RC RD RH RI t anamet oldt newt found_parentt parentt _removet _killt badR. ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyt __call__� s $ c C s t S( sF Override to suppress rel="nofollow" on some anchors. ( RO ( R t anchor( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyRt � s c C s� | j | j k r t S| j | j } t | t t f � r x= | D]5 } | j | � } | sa t S| j | | � sB t SqB Wt S| j | � } | s� t S| j | | � Sd S( N( R> t _tag_link_attrsRO t isinstanceRV t tupleRQ t allow_embedded_urlt True( R Rv t attrt one_attrt url( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR` � s c C s� | j d k r% | j | j k r% t St | � \ } } } } } | j � j d d � d } | d k rl t S| | j k r t St S( Nt :i i t httpt https( R� R� ( t whitelist_tagsRe R> RO R RW t splitt host_whitelistR� ( R Rv R� t schemet netloct patht queryt fragment( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR� � s c C s# g } | j | d � t j � d S( s� IE conditional comments basically embed HTML that the parser doesn't normally see. We can't allow anything like that, so we'll kill any comments that could be conditional. c S s t j | j � S( N( t _conditional_comment_ret searchRZ ( Rv ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyt <lambda>� s N( t _kill_elementsR R\ ( R Ru R~ ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR@ � s c C s[ g } x3 | j | � D]" } | | � r | j | � q q Wx | D] } | j � qC Wd S( N( R= Rj RY ( R Ru t conditiont iterateR~ Rv ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR� � s c C s& t d | � } t j | � r" d S| S( NR+ ( t _substitute_whitespacet _javascript_scheme_reR� ( R R Ry ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyRN � s s /\*.*?\*/c C s� | j d | � } | j d d � } t d | � } | j � } d | k rO t Sd | k r_ t Sd | k ro t Sd | k r t St S( s� Depending on the browser, stuff like ``e x p r e s s i o n(...)`` can get interpreted, or ``expre/* stuff */ssion(...)``. This checks for attempt to do stuff like this. Typically the response will be to kill the entire style; if you have just a bit of Javascript in the style another rule will catch that and remove only the Javascript from the style; this catches more sneaky attempts. R+ s \s javascript:s expression(s </noscripts @import( t _substitute_commentst replaceR� RW R� RO ( R R* ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyRU � s c C sP t | � } t | t � r* t | � } n t j | � } | | � t | | � S( N( R, R� t basestringR t copyt deepcopyR ( R R1 t result_typeRu ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR � s N( (. t __name__t __module__t __doc__R� RE RK R? RO R* R_ R/ Ra R[ Rc Rf Rh Ri Re RC RD RB Ro RG R RH Rr R� RA R� R t dictR� R Rt R` R� R@ R� RN t ret compilet SRS R� RU R ( ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR Z sR Z � sb (?P<body>https?://(?P<host>[a-z0-9._-]+)(?:/[/\-_.,a-z0-9%&?;=~]*)?(?:\([/\-_.,a-z0-9%&?;=~]*\))?)s9 mailto:(?P<body>[a-z0-9._-]+@(?P<host>[a-z0-9_._]+[a-z]))R8 t preR! R0 R7 R$ s ^localhosts \bexample\.(?:com|org|net)$s ^127\.0\.0\.1$t nolinkc C sG | j | k r d S| j d � } | rX | j � } x! | D] } | | k r; d Sq; Wn x� t | � D]� } t | d | d | d | d | �| j re t | j | | d | j �\ } } | r� | | _ | j | � } | | | d | d +q� qe qe W| j rCt | j | | d | j �\ } } | rC| | _ | | d *qCn d S( s Turn any URLs into links. It will search for links identified by the given regular expressions (by default mailto and http(s) links). It won't link text in an element in avoid_elements, or an element with a class in avoid_classes. It won't link to anything with a host that matches one of the regular expressions in avoid_hosts (default localhost and 127.0.0.1). If you pass in an element, the element's tail will not be substituted, only the contents of the element. Nt classt link_regexest avoid_elementst avoid_hostst avoid_classest factoryi i ( R> RQ R� RV R t tailt _link_textt makeelementt indexRZ ( Rv R� R� R� R� t class_namet match_classt childRZ t tail_childrenR� t pre_children( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR s4 ! ! c C s" d } g } d } xd \ } } x� | D]� } | } xh | j | d | �} | d k r\ Pn | j d � } x. | D]% } | j | � rr | j � } Pqr qr WPq7 | d k r� q( n | d k s� | j � | k r( | } | j � } q( q( W| d k r| r | | d _ n | } Pn | j d � } | j � } | j d � sM| j d � rd| d 8} | d } n | | j � } | r�| | d _ n | } | d � } | j d | � | j d � } | s�| } n | j d � s�| j d � r�| d } n | | _ | j | � | | } q | | f S( NR+ i t post hosti����t .t ,i R$ R t body( NN( Re R� t groupt endt startR� t endswithRA RZ Rj ( RZ R� R� R� t leading_textR_ t last_post best_matcht best_post regext regex_post matchR� t host_regexR R� t prev_textR� R� ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR� D s^ c O sV t | � } t | t � r* t | � } n t j | � } t | | | � t | | � S( N( R, R� R� R R� R� R R ( R1 t argsR R� Ru ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR ~ s t nobreaki( i c C s� | j t k r d S| j d � } | rn t } | j � } x$ | D] } | | k rA t } PqA qA W| rn d Sn | j r� t | j | | � | _ n xT | D]L } t | d | d | d | d | �| j r� t | j | | � | _ q� q� Wd S( s� Breaks any long words found in the body of the text (not attributes). Doesn't effect any of the tags in avoid_elements, by default ``<textarea>`` and ``<pre>`` Breaks words by inserting ​, which is a unicode character for Zero Width Space character. This generally takes up no space in rendering, but does copy as a space, and in monospace contexts usually takes up space. See http://www.cs.tut.fi/~jkorpela/html/nobr.html for a discussion NR� t max_widthR� R� t break_character( R> t _avoid_word_break_elementsRQ RO R� R� RZ t _break_textR R� ( Rv R� R� R� R� R� t dont_breakt avoidR� ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR � s* c O s5 t | � } t | � } t | | | � t | | � S( N( R, R R R ( R1 R� R R� Ru ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR � s c C sZ | j � } xG | D]? } t | � | k r t | | | � } | j | | � } q q W| S( N( R� t lent _insert_breakR� ( RZ R� R� t wordst wordt replacement( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR� � s s [^a-z]c C s� | } d } x� t | � | k r� | | } t t j | � � } | r| | d } | j � | d k r| | | j � } q| n | | | 7} | t | � } q W| | 7} | S( NR+ i����i ( R� RV t _break_prefer_ret finditerR� ( R� t widthR� t orig_wordt resultR� t breakst last_break( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyR� � s (6 R� R� R� t urlparseR t ImportErrort urllib.parset lxmlR t lxml.htmlR R R R R R t unichrt NameErrort chrt unicodet strt bytesR� t __all__R� R� t IRR RT R� RS R� R� t XPathRP Rs R" R R R t _link_regexest _avoid_elementst _avoid_hostst _avoid_classesR R� R R� t _avoid_word_break_classesR R R� R� R� ( ( ( s5 /usr/lib64/python2.7/site-packages/lxml/html/clean.pyt <module> s� � � * : &