HEX

File: //usr/local/lib/python3.10/dist-packages/pip/_internal/index/__pycache__/collector.cpython-310.pyc
o

��h9?�@sNdZddlmZddlZddlZddlZddlZddlZddl	Z	ddl
Z
ddlZddl
ZddlmZmZmZddlmZddlmZddlmZddlmZmZmZdd	lmZdd
lmZddl m!Z!m"Z"ddl#m$Z$dd
l%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5m6Z6e	�7e8�Z9ee:e:fZ;dSdd�Z<Gdd�de=�Z>dTd!d"�Z?Gd#d$�d$e=�Z@dUd'd(�ZAdVd)d*�ZBdWd-d.�ZCGd/d0�d0�ZDGd1d2�d2e�ZEdXd4d5�ZFeFdYd9d:��ZGed;d<�Gd=d7�d7��ZHGd>d?�d?e�ZI	dZd[dFdG�ZJ	;d\d]dJdK�ZKd^dMdN�ZLGdOdP�dPe�ZMGdQdR�dR�ZNdS)_zO
The main purpose of this module is to expose LinkCollector.collect_sources().
�)�annotationsN)�Iterable�MutableMapping�Sequence)�	dataclass)�
HTMLParser)�Values)�Callable�
NamedTuple�Protocol)�requests)�Response)�
RetryError�SSLError)�NetworkConnectionError)�Link)�SearchScope)�
PipSession)�raise_for_status)�is_archive_file��redact_auth_from_url)�vcs�)�CandidatesFromPage�
LinkSource�build_source�url�str�return�
str | NonecCs6tjD]}|���|�r|t|�dvr|SqdS)zgLook for VCS schemes in the URL.

    Returns the matched VCS scheme, or None if there's no match.
    z+:N)r�schemes�lower�
startswith�len)r�scheme�r&�H/usr/local/lib/python3.10/dist-packages/pip/_internal/index/collector.py�_match_vcs_scheme.s

�r(cseZdZd�fdd�Z�ZS)	�_NotAPIContent�content_typer�request_descr�Nonecst��||�||_||_dS�N)�super�__init__r*r+)�selfr*r+��	__class__r&r'r/:s
z_NotAPIContent.__init__)r*rr+rrr,)�__name__�
__module__�__qualname__r/�
__classcell__r&r&r1r'r)9sr)�responser
r,cCs2|j�dd�}|��}|�d�rdSt||jj��)z�
    Check the Content-Type header to ensure the response contains a Simple
    API Response.

    Raises `_NotAPIContent` if the content type is not a valid content-type.
    �Content-Type�Unknown)z	text/htmlz#application/vnd.pypi.simple.v1+html�#application/vnd.pypi.simple.v1+jsonN)�headers�getr"r#r)�request�method)r7r*�content_type_lr&r&r'�_ensure_api_header@s�r@c@seZdZdS)�_NotHTTPN)r3r4r5r&r&r&r'rAVsrA�sessionrcCsFtj�|�\}}}}}|dvrt��|j|dd�}t|�t|�dS)z�
    Send a HEAD request to the URL, and ensure the response contains a simple
    API Response.

    Raises `_NotHTTP` if the URL is not available for a HEAD request, or
    `_NotAPIContent` if the content type is not a valid content type.
    >�http�httpsT)�allow_redirectsN)�urllib�parse�urlsplitrA�headrr@)rrBr%�netloc�path�query�fragment�respr&r&r'�_ensure_api_responseZsrOcCsxtt|�j�r
t||d�t�dt|��|j|d�gd��dd�d�}t	|�t
|�t�dt|�|j�d	d
��|S)aYAccess an Simple API response with GET, and return the response.

    This consists of three parts:

    1. If the URL looks suspiciously like an archive, send a HEAD first to
       check the Content-Type is HTML or Simple API, to avoid downloading a
       large file. Raise `_NotHTTP` if the content type cannot be determined, or
       `_NotAPIContent` if it is not HTML or a Simple API.
    2. Actually perform the request. Raise HTTP exceptions on network failures.
    3. Check the Content-Type header to make sure we got a Simple API response,
       and raise `_NotAPIContent` otherwise.
    �rBzGetting page %sz, )r:z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z	max-age=0)�Acceptz
Cache-Control)r;zFetched page %s as %sr8r9)rr�filenamerO�logger�debugrr<�joinrr@r;)rrBrNr&r&r'�_get_simple_responsels&
����rVr;�ResponseHeaderscCs<|rd|vrtj��}|d|d<|�d�}|rt|�SdS)z=Determine if we have any encoding information in our headers.r8zcontent-type�charsetN)�email�message�Message�	get_paramr)r;�mrXr&r&r'�_get_encoding_from_headers�s

r^c@s*eZdZddd�Zdd
d�Zdd
d�ZdS)�CacheablePageContent�page�IndexContentrr,cCs|jsJ�||_dSr-)�cache_link_parsingr`�r0r`r&r&r'r/�s

zCacheablePageContent.__init__�other�object�boolcCst|t|��o|jj|jjkSr-)�
isinstance�typer`r)r0rdr&r&r'�__eq__�szCacheablePageContent.__eq__�intcCst|jj�Sr-)�hashr`r�r0r&r&r'�__hash__�szCacheablePageContent.__hash__N)r`rarr,)rdrerrf)rrj)r3r4r5r/rirmr&r&r&r'r_�s

r_c@seZdZddd�ZdS)	�
ParseLinksr`rar�Iterable[Link]cCsdSr-r&rcr&r&r'�__call__�szParseLinks.__call__N�r`rarro)r3r4r5rpr&r&r&r'rn�srn�fncs2tjd�fdd���t���d
��fd	d
��}|S)z�
    Given a function that parses an Iterable[Link] from an IndexContent, cache the
    function's result (keyed by CacheablePageContent), unless the IndexContent
    `page` has `page.cache_link_parsing == False`.
    �cacheable_pager_r�
list[Link]cst�|j��Sr-)�listr`)rs)rrr&r'�wrapper�sz*with_cached_index_content.<locals>.wrapperr`racs|jr	�t|��St�|��Sr-)rbr_ru)r`�rrrvr&r'�wrapper_wrapper�sz2with_cached_index_content.<locals>.wrapper_wrapperN)rsr_rrt)r`rarrt)�	functools�cache�wraps)rrrxr&rwr'�with_cached_index_content�s
r|r`raroc
cs��|j��}|�d�r+t�|j�}|�dg�D]}t�||j	�}|dur%q|VqdSt
|j	�}|jp4d}|�|j�
|��|j	}|jpE|}|jD]}	tj|	||d�}|durXqI|VqIdS)z\
    Parse a Simple API's Index Content, and yield its anchor elements as Link objects.
    r:�filesNzutf-8)�page_url�base_url)r*r"r#�json�loads�contentr<r�	from_jsonr�HTMLLinkParser�encoding�feed�decoder�anchors�from_element)
r`r?�data�file�link�parserr�rr�anchorr&r&r'�parse_links�s*�





�r�T)�frozenc@sHeZdZUdZded<ded<ded<ded<d	Zd
ed<dd
d�ZdS)raa�Represents one response (or page), along with its URL.

    :param encoding: the encoding to decode the given content.
    :param url: the URL from which the HTML was downloaded.
    :param cache_link_parsing: whether links parsed from this page's url
                               should be cached. PyPI index urls should
                               have this set to False, for example.
    �bytesr�rr*r r�rTrfrbrcCs
t|j�Sr-)rrrlr&r&r'�__str__s
zIndexContent.__str__N)rr)r3r4r5�__doc__�__annotations__rbr�r&r&r&r'ra�s
	cs6eZdZdZd�fdd�Zddd�Zddd�Z�ZS)r�zf
    HTMLParser that keeps the first base HREF and a list of all anchor
    elements' attributes.
    rrrr,cs$t�jdd�||_d|_g|_dS)NT)�convert_charrefs)r.r/rrr�)r0rr1r&r'r/s
zHTMLLinkParser.__init__�tag�attrs�list[tuple[str, str | None]]cCsR|dkr|jdur|�|�}|dur||_dSdS|dkr'|j�t|��dSdS)N�base�a)r�get_hrefr��append�dict)r0r�r��hrefr&r&r'�handle_starttags

��zHTMLLinkParser.handle_starttagr cCs"|D]\}}|dkr|SqdS)Nr�r&)r0r��name�valuer&r&r'r�!s
�zHTMLLinkParser.get_href)rrrr,)r�rr�r�rr,)r�r�rr )r3r4r5r�r/r�r�r6r&r&r1r'r�s

r�r�r�reason�str | Exception�meth�Callable[..., None] | NonecCs|durtj}|d||�dS)Nz%Could not fetch URL %s: %s - skipping)rSrT)r�r�r�r&r&r'�_handle_get_simple_fail(sr�rbrfcCs&t|j�}t|j|jd||j|d�S)Nr8)r�rrb)r^r;rar�r)r7rbr�r&r&r'�_make_index_content2s
�r��IndexContent | Nonec

Cs�|j�dd�d}t|�}|rt�d||�dStj�|�\}}}}}}|dkrHtj	�
tj�|��rH|�
d�s;|d7}tj�|d�}t�d|�zt||d	�}Wn�ty`t�d
|�YdSty|}zt�d||j|j�WYd}~dSd}~wty�}zt||�WYd}~dSd}~wty�}zt||�WYd}~dSd}~wty�}zd}	|	t|�7}	t||	tjd
�WYd}~dSd}~wtjy�}zt|d|���WYd}~dSd}~wtjy�t|d�YdSwt||jd�S)N�#rrzICannot look at %s URL %s because it does not support lookup as web pages.r��/z
index.htmlz# file: URL is directory, getting %srPz`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.z�Skipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r�zconnection error: z	timed out)rb) r�splitr(rS�warningrFrG�urlparse�osrK�isdirr=�url2pathname�endswith�urljoinrTrVrAr)r+r*rr�rrr�infor�ConnectionError�Timeoutr�rb)
r�rBr�
vcs_schemer%�_rKrN�excr�r&r&r'�_get_index_content?sn�
�����
���������r�c@seZdZUded<ded<dS)�CollectedSourceszSequence[LinkSource | None]�
find_links�
index_urlsN)r3r4r5r�r&r&r&r'r�|s
r�c@sNeZdZdZd!dd	�Ze	
d"d#dd��Zed$dd��Zd%dd�Z	d&dd�Z
d S)'�
LinkCollectorz�
    Responsible for collecting Link objects from all configured locations,
    making network requests as needed.

    The class's main method is its collect_sources() method.
    rBr�search_scoperrr,cCs||_||_dSr-)r�rB)r0rBr�r&r&r'r/�s
zLinkCollector.__init__F�optionsr�suppress_no_indexrfcCsd|jg|j}|jr|st�dd�dd�|D���g}|jp g}tj|||jd�}t	||d�}|S)z�
        :param session: The Session to use to make requests.
        :param suppress_no_index: Whether to ignore the --no-index option
            when constructing the SearchScope object.
        zIgnoring indexes: %s�,css�|]}t|�VqdSr-r)�.0rr&r&r'�	<genexpr>�s�z'LinkCollector.create.<locals>.<genexpr>)r�r��no_index)rBr�)
�	index_url�extra_index_urlsr�rSrTrUr�r�creater�)�clsrBr�r�r�r�r��link_collectorr&r&r'r��s$
�
��zLinkCollector.create�	list[str]cCs|jjSr-)r�r�rlr&r&r'r��szLinkCollector.find_links�locationrr�cCst||jd�S)z>
        Fetch an HTML page containing package links.
        rP)r�rB)r0r�r&r&r'�fetch_response�szLinkCollector.fetch_response�project_namer�candidates_from_pagerr�cs�t����fdd��j���D����}t����fdd��jD����}t�tj	�rKdd�t
�||�D�}t|��d��d�g|}t�
d�|��tt|�t|�d	�S)
Nc	3�(�|]}t|��jjdd�d�VqdS)F�r��page_validator�
expand_dirrbr�N�rrB�is_secure_origin�r��loc�r�r�r0r&r'r����	��
�z0LinkCollector.collect_sources.<locals>.<genexpr>c	3r�)Tr�Nr�r�r�r&r'r��r�cSs*g|]}|dur|jdurd|j���qS)Nz* )r�)r��sr&r&r'�
<listcomp>�s

�z1LinkCollector.collect_sources.<locals>.<listcomp>z' location(s) to search for versions of �:�
)r�r�)�collections�OrderedDictr��get_index_urls_locations�valuesr�rS�isEnabledFor�logging�DEBUG�	itertools�chainr$rTrUr�ru)r0r�r��index_url_sources�find_links_sources�linesr&r�r'�collect_sources�s2
	�
�	�
�
�
����zLinkCollector.collect_sourcesN)rBrr�rrr,)F)rBrr�rr�rfrr�)rr�)r�rrr�)r�rr�rrr�)r3r4r5r�r/�classmethodr��propertyr�r�r�r&r&r&r'r��s
�!
r�)rrrr )r7r
rr,)rrrBrrr,)rrrBrrr
)r;rWrr )rrrnrrnrqr-)r�rr�r�r�r�rr,)T)r7r
rbrfrra)r�rrBrrr�)Or��
__future__rr��
email.messagerYryr�r�r�r��urllib.parserF�urllib.request�collections.abcrrr�dataclassesr�html.parserr�optparser�typingr	r
r�pip._vendorr�pip._vendor.requestsr
�pip._vendor.requests.exceptionsrr�pip._internal.exceptionsr�pip._internal.models.linkr�!pip._internal.models.search_scoper�pip._internal.network.sessionr�pip._internal.network.utilsr�pip._internal.utils.filetypesr�pip._internal.utils.miscr�pip._internal.vcsr�sourcesrrr�	getLoggerr3rSrrWr(�	Exceptionr)r@rArOrVr^r_rnr|r�rar�r�r�r�r�r�r&r&r&r'�<module>sh





?
��

=