D7net
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
lib
/
python2.7
/
site-packages
/
kitchen
/
text
/
Filename :
misc.pyo
back
Copy
� i�:Oc @ s5 d Z d d l Z d d l Z d d l Z y d d l Z Wn e k rS d Z n Xd d l Z d d l m Z d d l m Z e j � d Z e e d d � d d g e d d � � Z e e j e e � � Z e j d � Z e d � Z d d d � Z d d � Z d � Z d d � Z d d � Z d Z d S( s� --------------------------------------------- Miscellaneous functions for manipulating text --------------------------------------------- Collection of text functions that don't fit in another category. i����N( t sets( t ControlCharErrorg333333�?i i i i i i s (?s)<[^>]*>|&#?\w+;c C s� t | t � s' t t j d � � � n d } y t | | d � Wn t k rZ d } n X| r� t r� | r� t j | � } | d t k r� | d } q� n | s� d } n | S( s# Try to guess the encoding of a byte :class:`str` :arg byte_string: byte :class:`str` to guess the encoding of :kwarg disable_chardet: If this is True, we never attempt to use :mod:`chardet` to guess the encoding. This is useful if you need to have reproducibility whether :mod:`chardet` is installed or not. Default: :data:`False`. :raises TypeError: if :attr:`byte_string` is not a byte :class:`str` type :returns: string containing a guess at the encoding of :attr:`byte_string`. This is appropriate to pass as the encoding argument when encoding and decoding unicode strings. We start by attempting to decode the byte :class:`str` as :term:`UTF-8`. If this succeeds we tell the world it's :term:`UTF-8` text. If it doesn't and :mod:`chardet` is installed on the system and :attr:`disable_chardet` is False this function will use it to try detecting the encoding of :attr:`byte_string`. If it is not installed or :mod:`chardet` cannot determine the encoding with a high enough confidence then we rather arbitrarily claim that it is ``latin-1``. Since ``latin-1`` will encode to every byte, decoding from ``latin-1`` to :class:`unicode` will not cause :exc:`UnicodeErrors` although the output might be mangled. s' byte_string must be a byte string (str)s utf-8t strictt confidencet encodings latin-1N( t isinstancet strt TypeErrort kt b_t unicodet UnicodeDecodeErrort Nonet chardett detectt _CHARDET_THRESHHOLD( t byte_stringt disable_chardett input_encodingt detection_info( ( s5 /usr/lib/python2.7/site-packages/kitchen/text/misc.pyt guess_encoding; s s utf-8t replacec C sz y | | k o | | k SWn t k r/ n Xt | t � rT | j | | � } n | j | | � } | | k rv t St S( s� Compare two stringsi, converting to byte :class:`str` if one is :class:`unicode` :arg str1: First string to compare :arg str2: Second string to compare :kwarg encoding: If we need to convert one string into a byte :class:`str` to compare, the encoding to use. Default is :term:`utf-8`. :kwarg errors: What to do if we encounter errors when encoding the string. See the :func:`kitchen.text.converters.to_bytes` documentation for possible values. The default is ``replace``. This function prevents :exc:`UnicodeError` (python-2.4 or less) and :exc:`UnicodeWarning` (python 2.5 and higher) when we compare a :class:`unicode` string to a byte :class:`str`. The errors normally arise because the conversion is done to :term:`ASCII`. This function lets you convert to :term:`utf-8` or another encoding instead. .. note:: When we need to convert one of the strings from :class:`unicode` in order to compare them we convert the :class:`unicode` string into a byte :class:`str`. That means that strings can compare differently if you use different encodings for each. Note that ``str1 == str2`` is faster than this function if you can accept the following limitations: * Limited to python-2.5+ (otherwise a :exc:`UnicodeDecodeError` may be thrown) * Will generate a :exc:`UnicodeWarning` if non-:term:`ASCII` byte :class:`str` is compared to :class:`unicode` string. ( t UnicodeErrorR R t encodet Truet False( t str1t str2R t errors( ( s5 /usr/lib/python2.7/site-packages/kitchen/text/misc.pyt str_eqd s ! c C s t | t � s' t t j d � � � n | d k rX t t t d g t t � � � } n� | d k r� t t t d g t t � � � } ns | d k r� d } t | � } g t D] } | | k r� | ^ q� r� t t j d � � � q� n t t j d � � � | r| j | � } n | S( s� Look for and transform :term:`control characters` in a string :arg string: string to search for and transform :term:`control characters` within :kwarg strategy: XML does not allow :term:`ASCII` :term:`control characters`. When we encounter those we need to know what to do. Valid options are: :replace: (default) Replace the :term:`control characters` with ``"?"`` :ignore: Remove the characters altogether from the output :strict: Raise a :exc:`~kitchen.text.exceptions.ControlCharError` when we encounter a control character :raises TypeError: if :attr:`string` is not a unicode string. :raises ValueError: if the strategy is not one of replace, ignore, or strict. :raises kitchen.text.exceptions.ControlCharError: if the strategy is ``strict`` and a :term:`control character` is present in the :attr:`string` :returns: :class:`unicode` string with no :term:`control characters` in it. sD process_control_char must have a unicode type as the first argument.t ignoreR u ?R s* ASCII control code present in string inputsX The strategy argument to process_control_chars must be one of ignore, replace, or strictN( R R R R R t dictt zipt _CONTROL_CODESR t lent frozensett _CONTROL_CHARSR t ValueErrort translate( t stringt strategyt control_tablet datat c( ( s5 /usr/lib/python2.7/site-packages/kitchen/text/misc.pyt process_control_chars� s % % %c C sC d � } t | t � s0 t t j d � � � n t j t | | � S( s/ Substitute unicode characters for HTML entities :arg string: :class:`unicode` string to substitute out html entities :raises TypeError: if something other than a :class:`unicode` string is given :rtype: :class:`unicode` string :returns: The plain text without html entities c S s | j d � } | d d k r# d S| d d k r� yE | d d k r` t t | d d !d � � St t | d d !� � SWq t k r� q Xn� | d d k r t j j | d d !j d � � } | r | d d k r y t t | d d !� � SWqt k rqXqt | d � Sq n | S( Ni i u <t i u &#i u &#xi����i u &s utf-8s &#s iso-8859-1( t groupt unichrt intR% t htmlentitydefst entitydefst getR R ( t matchR' t entity( ( s5 /usr/lib/python2.7/site-packages/kitchen/text/misc.pyt fixup� s( " sF html_entities_unescape must have a unicode type for its first argument( R R R R R t ret subt _ENTITY_RE( R' R6 ( ( s5 /usr/lib/python2.7/site-packages/kitchen/text/misc.pyt html_entities_unescape� s c C s^ t | t � s t Sy t | | � } Wn t k r: t SXt | � } | j t � rZ t St S( s� Check that a byte :class:`str` would be valid in xml :arg byte_string: Byte :class:`str` to check :arg encoding: Encoding of the xml file. Default: :term:`UTF-8` :returns: :data:`True` if the string is valid. :data:`False` if it would be invalid in the xml file In some cases you'll have a whole bunch of byte strings and rather than transforming them to :class:`unicode` and back to byte :class:`str` for output to xml, you will just want to make sure they work with the xml file you're constructing. This function will help you do that. Example:: ARRAY_OF_MOSTLY_UTF8_STRINGS = [...] processed_array = [] for string in ARRAY_OF_MOSTLY_UTF8_STRINGS: if byte_string_valid_xml(string, 'utf-8'): processed_array.append(string) else: processed_array.append(guess_bytes_to_xml(string, encoding='utf-8')) output_xml(processed_array) ( R R R R R R# t intersectionR$ R ( R R t u_stringR* ( ( s5 /usr/lib/python2.7/site-packages/kitchen/text/misc.pyt byte_string_valid_xml� s c C s* y t | | � Wn t k r% t SXt S( s� Detect if a byte :class:`str` is valid in a specific encoding :arg byte_string: Byte :class:`str` to test for bytes not valid in this encoding :kwarg encoding: encoding to test against. Defaults to :term:`UTF-8`. :returns: :data:`True` if there are no invalid :term:`UTF-8` characters. :data:`False` if an invalid character is detected. .. note:: This function checks whether the byte :class:`str` is valid in the specified encoding. It **does not** detect whether the byte :class:`str` actually was encoded in that encoding. If you want that sort of functionality, you probably want to use :func:`~kitchen.text.misc.guess_encoding` instead. ( R R R R ( R R ( ( s5 /usr/lib/python2.7/site-packages/kitchen/text/misc.pyt byte_string_valid_encoding s R> R= R R: R, R ( s byte_string_valid_encodings byte_string_valid_xmls guess_encodings html_entities_unescapes process_control_charss str_eq( t __doc__R1 t itertoolsR7 R t ImportErrorR t kitchenR t kitchen.pycompat24R t kitchen.text.exceptionsR t add_builtin_setR R# t rangeR! t imapR/ R$ t compileR9 R R R R, R: R= R> t __all__( ( ( s5 /usr/lib/python2.7/site-packages/kitchen/text/misc.pyt <module> s0 , )/ 8 * (