3
]9Y @ s0 d dl Z d dlZddlmZ G dd deZdS ) N ) ProbingStatec @ sn e Zd ZdZdddZdd Zedd Zd d
Zed d Z d
d Z
e dd Z e dd Z
e dd ZdS )
CharSetProbergffffff?Nc C s d | _ || _tjt| _d S )N)_state lang_filterloggingZ getLogger__name__Zlogger)selfr r
#/usr/lib/python3.6/charsetprober.py__init__' s zCharSetProber.__init__c C s t j| _d S )N)r Z DETECTINGr )r r
r
r reset, s zCharSetProber.resetc C s d S )Nr
)r r
r
r charset_name/ s zCharSetProber.charset_namec C s d S )Nr
)r bufr
r
r feed3 s zCharSetProber.feedc C s | j S )N)r )r r
r
r state6 s zCharSetProber.statec C s dS )Ng r
)r r
r
r get_confidence: s z CharSetProber.get_confidencec C s t jdd| } | S )Ns ([ -])+ )resub)r r
r
r filter_high_byte_only= s z#CharSetProber.filter_high_byte_onlyc C sb t }tjd| }xJ|D ]B}|j|dd |dd }|j rP|dk rPd}|j| qW |S )u9
We define three types of bytes:
alphabet: english alphabets [a-zA-Z]
international: international characters [-ÿ]
marker: everything else [^a-zA-Z-ÿ]
The input buffer can be thought to contain a series of words delimited
by markers. This function works to filter all words that contain at
least one international character. All contiguous sequences of markers
are replaced by a single space ascii character.
This filter applies to all scripts which do not use English characters.
s% [a-zA-Z]*[-]+[a-zA-Z]*[^a-zA-Z-]?Nr r r ) bytearrayr findallextendisalpha)r filteredZwordsZwordZ last_charr
r
r filter_international_wordsB s
z(CharSetProber.filter_international_wordsc C s t }d}d}xtt| D ]r}| ||d }|dkr>d}n |dkrJd}|dk r |j r ||kr| r|j| || |jd |d }q W |s|j| |d |S )
a
Returns a copy of ``buf`` that retains only the sequences of English
alphabet and high byte characters that are not between <> characters.
Also retains English alphabet and high byte characters immediately
before occurrences of >.
This filter can be applied to all scripts which contain both English
characters and extended ASCII characters, but is currently only used by
``Latin1Prober``.
Fr r >