File: //usr/local/CyberCP/lib/python3.10/site-packages/charset_normalizer/__pycache__/md.cpython-310.pyc
o
�h�N � @ sV d dl mZ d dlmZ d dlmZ ddlmZmZm Z ddl
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z G dd� d�Z G d d
� d
e �Z G d d � d e �Z G d
d� de �Z G dd� de �Z!G dd� de �Z"G dd� de �Z#G dd� de �Z$G dd� de �Z%G dd� de �Z&edd �d/d"d#��Z'ed$d � &d0d1d,d-��Z(d.S )2� )� annotations)� lru_cache)� getLogger� )� COMMON_SAFE_ASCII_CHARACTERS�TRACE� UNICODE_SECONDARY_RANGE_KEYWORD)�is_accentuated� is_arabic�is_arabic_isolated_form�is_case_variable�is_cjk� is_emoticon� is_hangul� is_hiragana� is_katakana�is_latin�is_punctuation� is_separator� is_symbol�is_thai�is_unprintable�
remove_accent�
unicode_range�is_cjk_uncommonc @ s< e Zd ZdZddd�Zdd d
�Zdd d �Zeddd��ZdS )�MessDetectorPluginzy
Base abstract class used for mess detection plugins.
All detectors MUST extend and implement given methods.
� character�str�return�boolc C � t �)z@
Determine if given character should be fed in.
��NotImplementedError��selfr � r% �H/usr/local/CyberCP/lib/python3.10/site-packages/charset_normalizer/md.py�eligible' � zMessDetectorPlugin.eligible�Nonec C r )z�
The main routine to be executed upon character.
Insert the logic in witch the text would be considered chaotic.
r! r# r% r% r& �feed- s zMessDetectorPlugin.feedc C r )zB
Permit to reset the plugin to the initial state.
r! �r$ r% r% r& �reset4 r( zMessDetectorPlugin.reset�floatc C r )z�
Compute the chaos ratio based on what your feed() has seen.
Must NOT be lower than 0.; No restriction gt 0.
r! r+ r% r% r& �ratio: s zMessDetectorPlugin.ratioN�r r r r �r r r r) �r r) �r r- ) �__name__�
__module__� __qualname__�__doc__r' r* r, �propertyr. r% r% r% r& r ! s
r c @ �B e Zd Zddd�Zddd �Zdd
d �Zdd d
�Zeddd��ZdS )� TooManySymbolOrPunctuationPluginr r) c C s" d| _ d| _d| _d | _d| _d S )Nr F)�_punctuation_count�
_symbol_count�_character_count�_last_printable_char�_frenzy_symbol_in_wordr+ r% r% r& �__init__D s
z)TooManySymbolOrPunctuationPlugin.__init__r r r c C � |� � S �N�� isprintabler# r% r% r& r' L � z)TooManySymbolOrPunctuationPlugin.eligiblec C sp | j d7 _ || jkr3|tvr3t|�r | jd7 _n|�� du r3t|�r3t|�du r3| jd7 _|| _d S )Nr F� ) r< r= r r r: �isdigitr r r; r# r% r% r& r* O s
�
z%TooManySymbolOrPunctuationPlugin.feedc C s d| _ d| _d| _d S �Nr )r: r< r; r+ r% r% r& r, a �
z&TooManySymbolOrPunctuationPlugin.resetr- c C s0 | j dkrdS | j| j | j }|dkr|S dS )Nr � �333333�?)r< r: r; )r$ �ratio_of_punctuationr% r% r& r. f s
�z&TooManySymbolOrPunctuationPlugin.ratioNr1 r/ r0 r2 � r3 r4 r5 r? r' r* r, r7 r. r% r% r% r& r9 C s
r9 c @ r8 )�TooManyAccentuatedPluginr r) c C � d| _ d| _d S rG �r< �_accentuated_countr+ r% r% r& r? s �
z!TooManyAccentuatedPlugin.__init__r r r c C r@ rA )�isalphar# r% r% r& r' w rD z!TooManyAccentuatedPlugin.eligiblec C �, | j d7 _ t|�r| jd7 _d S d S �Nr )r< r rP r# r% r% r& r* z � �z TooManyAccentuatedPlugin.feedc C rN rG rO r+ r% r% r& r, � rQ z TooManyAccentuatedPlugin.resetr- c C s* | j dk rdS | j| j }|dkr|S dS )N� rI gffffff�?rO )r$ �ratio_of_accentuationr% r% r& r. � s
z TooManyAccentuatedPlugin.ratioNr1 r/ r0 r2 rL r% r% r% r& rM r s
rM c @ r8 )�UnprintablePluginr r) c C rN rG )�_unprintable_countr< r+ r% r% r& r? � rQ zUnprintablePlugin.__init__r r r c C � dS �NTr% r# r% r% r& r' � � zUnprintablePlugin.eligiblec C s( t |�r | jd7 _| jd7 _d S rT )r rY r< r# r% r% r& r* � s zUnprintablePlugin.feedc C s
d| _ d S rG )rY r+ r% r% r& r, � s
zUnprintablePlugin.resetr- c C � | j dkrdS | jd | j S )Nr rI rV )r< rY r+ r% r% r& r. � �
zUnprintablePlugin.ratioNr1 r/ r0 r2 rL r% r% r% r& rX � s
rX c @ r8 )� SuspiciousDuplicateAccentPluginr r) c C � d| _ d| _d | _d S rG ��_successive_countr< �_last_latin_characterr+ r% r% r& r? � s
z(SuspiciousDuplicateAccentPlugin.__init__r r r c C s |� � ot|�S rA )rR r r# r% r% r& r' � s z(SuspiciousDuplicateAccentPlugin.eligiblec C st | j d7 _ | jd ur5t|�r5t| j�r5|�� r%| j�� r%| jd7 _t|�t| j�kr5| jd7 _|| _d S rT )r< rc r �isupperrb r r# r% r% r& r* � s
��
z$SuspiciousDuplicateAccentPlugin.feedc C r` rG ra r+ r% r% r& r, � rH z%SuspiciousDuplicateAccentPlugin.resetr- c C r] )Nr rI rE )r< rb r+ r% r% r& r. � r^ z%SuspiciousDuplicateAccentPlugin.ratioNr1 r/ r0 r2 rL r% r% r% r& r_ � s
r_ c @ r8 )�SuspiciousRanger r) c C r` rG )�"_suspicious_successive_range_countr< �_last_printable_seenr+ r% r% r& r? � rH zSuspiciousRange.__init__r r r c C r@ rA rB r# r% r% r& r' � rD zSuspiciousRange.eligiblec C sx | j d7 _ |�� st|�s|tv rd | _d S | jd u r"|| _d S t| j�}t|�}t||�r7| jd7 _|| _d S rT )r< �isspacer r rg r � is_suspiciously_successive_rangerf )r$ r �unicode_range_a�unicode_range_br% r% r& r* � s ��
zSuspiciousRange.feedc C r` rG )r< rf rg r+ r% r% r& r, � rH zSuspiciousRange.resetr- c C s" | j dkrdS | jd | j }|S )N�
rI rE )r< rf )r$ � ratio_of_suspicious_range_usager% r% r& r. � s
�zSuspiciousRange.ratioNr1 r/ r0 r2 rL r% r% r% r& re � s
re c @ r8 )�SuperWeirdWordPluginr r) c C s@ d| _ d| _d| _d| _d| _d| _d| _d| _d| _d| _ d S )Nr F� )
� _word_count�_bad_word_count�_foreign_long_count�_is_current_word_bad�_foreign_long_watchr< �_bad_character_count�_buffer�_buffer_accent_count�_buffer_glyph_countr+ r% r% r& r? � s
z SuperWeirdWordPlugin.__init__r r r c C rZ r[ r% r# r% r% r& r' r\ z SuperWeirdWordPlugin.eligiblec C s� |� � rc| j|7 _t|�r| jd7 _| jdu rFt|�du s%t|�rFt|�du rFt|�du rFt|�du rFt |�du rFt
|�du rFd| _t|�sZt|�sZt|�sZt |�sZt
|�ra| j d7 _ d S | jshd S |� � sut
|�sut|��r$| j�r$| jd7 _t| j�}| j|7 _|dkr�| j| dkr�d| _n4t| jd �r�| jd �� r�tdd� | jD ��du r�| jd7 _d| _n| j dkr�d| _| jd7 _|d kr�| jr�d
d � t| jtd |��D �}d}|r�t|�| d
kr�d}|s�| jd7 _d| _| j�r| jd7 _| jt| j�7 _d| _d| _d| _d | _d | _ d S |dv�rA|�� du �rCt|��rEd| _| j|7 _d S d S d S d S )Nr FT� � �?���c s s � | ]}|� � V qd S rA �rd )�.0�_r% r% r& � <genexpr>8 s � z,SuperWeirdWordPlugin.feed.<locals>.<genexpr>� c S s g | ]
\}}|� � r|�qS r% r| )r} �c�ir% r% r& �
<listcomp>@ s ��z-SuperWeirdWordPlugin.feed.<locals>.<listcomp>r rJ ro > r~ �-�<�=�>�|�~) rR rv r rw rt r r
r r r r rx rh r r rp �lenr< rs rd �allrr �zip�rangerq ru rF r )r$ r �
buffer_length�camel_case_dst�probable_camel_casedr% r% r& r* s�
���������
� �
�
��zSuperWeirdWordPlugin.feedc C s4 d| _ d| _d| _d| _d| _d| _d| _d| _d S )Nro Fr )rv rs rt rq rp r< ru rr r+ r% r% r& r, _ s
zSuperWeirdWordPlugin.resetr- c C s$ | j dkr | jdkr dS | j| j S )N�
r rI )rp rr ru r< r+ r% r% r& r. i s zSuperWeirdWordPlugin.ratioNr1 r/ r0 r2 rL r% r% r% r& rn � s
Q
rn c @ sF e Zd ZdZddd�Zdd d
�Zdd d �Zdd
d�Zeddd��Z dS )�CjkUncommonPluginz<
Detect messy CJK text that probably means nothing.
r r) c C rN rG �r< �_uncommon_countr+ r% r% r& r? v rQ zCjkUncommonPlugin.__init__r r r c C � t |�S rA )r
r# r% r% r& r' z rD zCjkUncommonPlugin.eligiblec C rS rT )r< r r� r# r% r% r& r* } s
�zCjkUncommonPlugin.feedc C rN rG r� r+ r% r% r& r, � rQ zCjkUncommonPlugin.resetr- c C s. | j dk rdS | j| j }|dkr|d S dS )NrV rI rz r� r� )r$ �uncommon_form_usager% r% r& r. � s
zCjkUncommonPlugin.ratioNr1 r/ r0 r2 )
r3 r4 r5 r6 r? r' r* r, r7 r. r% r% r% r& r� q s
r� c @ r8 )�ArchaicUpperLowerPluginr r) c C s. d| _ d| _d| _d| _d| _d | _d| _d S )NFr T)�_buf� _character_count_since_last_sep� _successive_upper_lower_count�#_successive_upper_lower_count_finalr< �_last_alpha_seen�_current_ascii_onlyr+ r% r% r& r? � s
z ArchaicUpperLowerPlugin.__init__r r r c C rZ r[ r% r# r% r% r& r' � r\ z ArchaicUpperLowerPlugin.eligiblec C s$ |� � ot|�}|du }|rC| jdkrC| jdkr+|�� du r+| jdu r+| j| j7 _d| _d| _d | _d| _| j d7 _ d| _d S | jdu rQ|�
� du rQd| _| jd ur|� � r_| j� � sh|� � r|| j� � r|| jdu rx| jd7 _d| _nd| _nd| _| j d7 _ | jd7 _|| _d S )NFr �@ r TrE )
rR r r� rF r� r� r� r� r� r< �isasciird �islower)r$ r � is_concerned� chunk_sepr% r% r& r* � s@
�
��
z ArchaicUpperLowerPlugin.feedc C s. d| _ d| _d| _d| _d | _d| _d| _d S )Nr FT)r< r� r� r� r� r� r� r+ r% r% r& r, � s
z ArchaicUpperLowerPlugin.resetr- c C s | j dkrdS | j| j S )Nr rI )r< r� r+ r% r% r& r. � s
z ArchaicUpperLowerPlugin.ratioNr1 r/ r0 r2 rL r% r% r% r& r� � s
* r� c @ sB e Zd Zddd�Zddd�Zdd
d �Zdd d
�Zeddd��ZdS )�ArabicIsolatedFormPluginr r) c C rN rG �r< �_isolated_form_countr+ r% r% r& r? � rQ z!ArabicIsolatedFormPlugin.__init__c C rN rG r� r+ r% r% r& r, � rQ z ArabicIsolatedFormPlugin.resetr r r c C r� rA )r
r# r% r% r& r' � rD z!ArabicIsolatedFormPlugin.eligiblec C rS rT )r< r r� r# r% r% r& r* � rU z ArabicIsolatedFormPlugin.feedr- c C s | j dk rdS | j| j }|S )NrV rI r� )r$ �isolated_form_usager% r% r& r. � s
z ArabicIsolatedFormPlugin.ratioNr1 r/ r0 r2 ) r3 r4 r5 r? r, r' r* r7 r. r% r% r% r& r� � s
r� � )�maxsizerj �
str | Nonerk r r c C sv | du s|du r
dS | |krdS d| v rd|v rdS d| v s"d|v r$dS d| v s,d|v r6d| v s4d|v r6dS | � d�|� d�}}|D ]}|tv rJqC||v rQ dS qC| dv |dv }}|s_|rid | v sgd |v ridS |ro|rodS d
| v swd
|v r�d | v sd |v r�dS | d ks�|d kr�dS d | v s�d |v s�| d v r�|d v r�d
| v s�d
|v r�dS d| v s�d|v r�dS | d ks�|d kr�dS dS )za
Determine if two Unicode range seen next to each other can be considered as suspicious.
NTF�Latin� Emoticons� Combining� )�Hiragana�Katakana�CJK�Hangulz Basic Latin)r� r� � Punctuation�Forms)�splitr )rj rk �keywords_range_a�keywords_range_b�el�range_a_jp_chars�range_b_jp_charsr% r% r& ri � sZ ����ri i 皙�����?F�decoded_sequencer �maximum_thresholdr- �debugc C sR dd� t �� D �}t| �d }d}|dk rd}n |dkr d}nd }t| d
t|��D ]2\}}|D ] } | �|�r<| �|� q0|d krG|| d ksM||d kr\td d
� |D ��}||kr\ nq*|r�td�}
|
� t
d|� d|� d|� �� t| �dkr�|
� t
d| dd� � �� |
� t
d| dd� � �� |D ]} |
� t
| j � d| j � �� q�t
|d�S )zw
Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier.
c S s g | ]}|� �qS r% r% )r} �md_classr% r% r& r� N s �z mess_ratio.<locals>.<listcomp>r rI i � r� r� � �
r c s s � | ]}|j V qd S rA )r. )r} �dtr% r% r& r e s � z mess_ratio.<locals>.<genexpr>�charset_normalizerzIMess-detector extended-analysis start. intermediary_mean_mess_ratio_calc=z mean_mess_ratio=z maximum_threshold=� zStarting with: Nz
Ending with: i�z: � )r �__subclasses__r� r� r� r' r* �sumr �logr � __class__r. �round) r� r� r� � detectors�length�mean_mess_ratio�!intermediary_mean_mess_ratio_calcr �index�detector�loggerr� r% r% r& �
mess_ratioF sN �
� �����
r� N)rj r� rk r� r r )r� F)r� r r� r- r� r r r- ))�
__future__r � functoolsr �loggingr �constantr r r �utilsr r
r r r
r r r r r r r r r r r r r r r9 rM rX r_ re rn r� r� r� ri r� r% r% r% r& �<module> s( P"/%1v#L I�