*#h=)ddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z m Z ddlZddlmZmZddlmZddlmZmZmZmZmZmZmZmZmZmZdd lm Z  d d e!fd Z" dd e!fd Z#y)N)contextmanager)partial)Path)urljoinurlparse)DownloadConfigconfig)ExtractManager) !_raise_if_offline_mode_is_enabledftp_getftp_head"get_authentication_headers_for_urlhash_url_to_filenamehttp_get http_head is_local_path is_remote_urllogger)FileLockreturnc | tdi|}|jxstj}t |t r t |}t |t r t |}t|r|t|||j|j|j|j|j|j|j|j |j"|j$ }nKt&j(j+|r|}n)t-|rt/d|dt1d|d||S|j2r1t5|jj7||j8}|S) a Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and return the path to the cached file. If it's already a local path, make sure the file exists and then return the path. Return: Local path (string) Raises: FileNotFoundError: in case of non-recoverable file (non-existent or no cache on disk) ConnectionError: in case of unreachable url and no cache on disk ValueError: if it couldn't parse the url or filename correctly requests.exceptions.ConnectionError: in case of internet connection issue ) cache_dirforce_downloadproxiesresume_download user_agentlocal_files_onlyuse_etag max_retriesuse_auth_tokenignore_url_params download_descz Local file z doesn't existzunable to parse z as a URL or as a local path)r) force_extract)rrr DOWNLOADED_DATASETS_PATH isinstancerstrrget_from_cacherrrrrrrr r!r"ospathexistsrFileNotFoundError ValueErrorextract_compressed_filer extractr#)url_or_filenamedownload_configdownload_kwargsr output_paths Q/var/www/html/sandstorm/venv/lib/python3.12/site-packages/datasets/utils/cache.py cached_pathr5s\,(;?;))LV-L-LI)T" N /4(o._%$ *99#+++;;&11,==$--'33*99-??)77    (%  '+o->n MNN+O+<j@f$r }|}Yd}~d}~wwxYw#1swYxYw#1swYRxYw#1swY|SxYw)4a Given a URL, look for the corresponding file in the local cache. If it's not there, download it. Then return the path to the cached file. Return: Local path (string) Raises: FileNotFoundError: in case of non-recoverable file (non-existent or no cache on disk) ConnectionError: in case of unreachable url and no cache on disk NT)exist_okF)etag)r z user-agentzftp://)allow_redirectsrtimeoutrheadersETagdownload_warningzdrive.google.comz &confirm=zconfirm=z &confirm=tizfirebasestorage.googleapis.comiiz7^https?://github.com/.*?/.*?/releases/download/.*?/.*?$z#^https://.*?s3.*?amazonaws.com/.*?$zndownloader.figstatic.comz"Couldn't get ETag version for url izUnauthorized for URL z^. Please use the parameter `use_auth_token=True` after logging in with `huggingface-cli login`z6Cannot find the requested files in the cached path at zi and outgoing traffic has been disabled. To enable file online look-ups, set 'local_files_only' to False.izCouldn't find file at zTried to reach zCouldn't reach z ()z (error z.lockz .incompletec3XKtd5}|dddy#1swYyxYww)Nza+b)open)fincomplete_paths r4_resumable_file_managerz/get_from_cache.._resumable_file_managers,/51QGs * *'*r)dirdeletezB not found in cache or force_download set to True, downloading to )r resume_sizer<cookiesrdesczstoring z in cache at zcreating metadata file for )urlr9z.jsonwzutf-8)encoding)3r HF_DATASETS_CACHEr&rr'r)makedirsrrr*rjoinr+r startswithr r status_coder<getrIitemsrematchrKrinfo HF_ENDPOINTConnectionErrorOSErrorrequests exceptionsTimeoutr,r reprrrstatst_sizertempfileNamedTemporaryFilenamer rshutilmoverBjsondump)"rKrrr etag_timeoutrrrrrr r!r" cached_url connectedresponserIr9 head_errorfilename cache_pathr<kve lock_pathrEtemp_file_managerrH temp_filemeta meta_path meta_filerDs" @r4r(r(bsO8,, )T" N KK D)S(3-"4"45  IHG DJ$JT:Hi2J ww~~j!.1^TG *   >>( #  I*   $$' H##s*7?x''++F3T$,,2243DAq||$67>* %n  #H U^^  !h&:&:c&A#&>  '(C(C[` a K  I KK3%abkbpbpaqr s~~h'Y'# +## +&  $  hse= => INNJ/ 1*>?D)( )S7 3 'y IIdI & 'Y-'^ e,,445 J  n  0 ' 'Y-'^ se:A0S+S0C1S,A?T(+ATA3T(4T T(#T TT T T(T% !T((T2)N) NFNdFNFTrNFN)$rfr)rUrdra contextlibr functoolsrpathlibr urllib.parserrr[datasetsrr datasets.utils.extractr datasets.utils.file_utilsr r r rrrrrrrdatasets.utils.filelockrr'r5r(r$r6r4rs %*+1   - A AL  y yr6