"h VrddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZmZddlmZmZmZmZej.dZGd d ZGd d eZ dd ZdZdZ ddZddZ dZ!dZ"dZ#dZ$ ddZ%dZ&GddejNZ(y)N) has_magic) BaseCache BlockCache BytesCache MMapCacheReadAheadCachecaches)compr) filesystemget_filesystem_class)_unstrip_protocolbuild_name_functioninfer_compressionstringify_pathfsspeccVeZdZdZ d dZdZdZdZdZe dZ d Z d Z y) OpenFileaF File-like object to be used in a context Can layer (buffered) text-mode and compression over any file-system, which are typically binary-only. These instances are safe to serialize, as the low-level file object is not created until invoked using ``with``. Parameters ---------- fs: FileSystem The file system to use for opening the file. Should be a subclass or duck-type with ``fsspec.spec.AbstractFileSystem`` path: str Location to open mode: str like 'rb', optional Mode of the opened file compression: str or None, optional Compression to apply encoding: str or None, optional The encoding to use if opened in text mode. errors: str or None, optional How to handle encoding errors if opened in text mode. newline: None or str Passed to TextIOWrapper in text mode, how to handle line endings. autoopen: bool If True, calls open() immediately. Mostly used by pickle pos: int If given and autoopen is True, seek to this location immediately Nc||_||_||_t|||_||_||_||_g|_yN) fspathmodeget_compression compressionencodingerrorsnewlinefobjects)selfrrrrrrrs H/var/www/html/sandstorm/venv/lib/python3.12/site-packages/fsspec/core.py__init__zOpenFile.__init__=sD  *4=     ct|j|j|j|j|j |j |jffSr)rrrrrrrrr s r! __reduce__zOpenFile.__reduce__PsG           r#c"d|jdS)Nz )rr%s r!__repr__zOpenFile.__repr__^sTYYKr**r#c|jjddjdddz}|jj|j|}|g|_|j ;t|j }|||d}|j j|d|jvrHt||j|j|j}|j j||j dS)Ntb)rr)rrr) rreplaceropenrrrr appendPickleableTextIOWrapperrrr)r rfcompresss r! __enter__zOpenFile.__enter__asyy  b)11#r:S@ GGLLL .    'T--.Ha)A MM  # dii 'DMM$++t||A MM  #}}R  r#c$|jyr)close)r argss r!__exit__zOpenFile.__exit__vs  r#cBt|j|jSr)rrrr%s r! full_namezOpenFile.full_nameys DGG44r#c"|jS)aMaterialise this as a real open file without context The OpenFile object should be explicitly closed to avoid enclosed file instances persisting. You must, therefore, keep a reference to the OpenFile during the life of the file-like it generates. )r4r%s r!r/z OpenFile.open}s~~r#ct|jD]<}d|jvr|js|j |j >|jj y)z#Close all encapsulated file objectsrN)reversedrrclosedflushr6clear)r r2s r!r6zOpenFile.closesP$--( A$))#AHH GGI  r#)rbNNNN) __name__ __module__ __qualname____doc__r"r&r(r4r8propertyr:r/r6r#r!rrsPH&  +!*55 r#rcFeZdZdZdddfd ZdZdZfdZd ZxZ S) OpenFilesaList of OpenFile instances Can be used in a single context, which opens and closes all of the contained files. Normal list access to get the elements works as normal. A special case is made for caching filesystems - the files will be down/uploaded together at the start or end of the context, and this may happen concurrently, if the target filesystem supports it. rBNrrcH||_||_g|_t||yr)rrfilessuperr")r rrr7 __class__s r!r"zOpenFiles.__init__s%   $r#c:|j td|j} t|dr"|j||_|jSt|dr|j |j}nnU|Dcgc]}|j c}Scc}w)NzContext has already been used open_manyr)r ValueErrorhasattrrQrMr4)r rss r!r4zOpenFiles.__enter__s 77?<= = WWr;'\\$/ zz!r4 RUU%6UU(,,! ,,,s>Bc|j}|Dcgc]}|j|c}d|jvrP t|dr|j |j yt|dr|j |j}nyOycc}w)Nr=rQr)rr8rrS commit_manyrM)r r7rrTs r!r8zOpenFiles.__exit__s~ WW$()qT ) dii 2{+NN4::.2t$):B  *sB ct||}t|tr"t ||j |j S|S)NrK)rN __getitem__ isinstanceslicerJrr)r itemoutrOs r!rXzOpenFiles.__getitem__s8g!$' dE "StyyTWW= = r#c dt|dS)Nz )lenr%s r!r(zOpenFiles.__repr__s3t9+%9::r#) rCrDrErFr"r4r8rXr( __classcell__rOs@r!rJrJs( $(D -  ;r#rJc  \t||||| || \} } }| jdk(r| | _nEd|vrA| r?|Dchc]}| j|}}|Dcgc]}| j |dc}t |Dcgc]}t | ||||||c}|| Scc}wcc}wcc}w)av Given a path or paths, return a list of ``OpenFile`` objects. For writing, a str path must contain the "*" character, which will be filled in by increasing numbers, e.g., "part*" -> "part1", "part2" if num=2. For either reading or writing, can instead provide explicit list of paths. Parameters ---------- urlpath: string or list Absolute or relative filepath(s). Prefix with a protocol like ``s3://`` to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol. mode: 'rb', 'wt', etc. compression: string or None If given, open file using compression codec. Can either be a compression name (a key in ``fsspec.compression.compr``) or "infer" to guess the compression from the filename suffix. encoding: str For text mode only errors: None or str Passed to TextIOWrapper in text mode name_function: function or None if opening a set of files for writing, those files do not yet exist, so we need to generate their names by formatting the urlpath for each sequence number num: int [1] if writing mode, number of files we expect to create (passed to name+function) protocol: str or None If given, overrides the protocol found in the URL. newline: bytes or None Used for line terminator in text mode. If None, uses system default; if blank, uses no translation. auto_mkdir: bool (True) If in write mode, this will ensure the target directory exists before writing, by calling ``fs.mkdirs(exist_ok=True)``. expand: bool **kwargs: dict Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc. Examples -------- >>> files = open_files('2015-*-*.csv') # doctest: +SKIP >>> files = open_files( ... 's3://bucket/2015-*-*.csv.gz', compression='gzip' ... ) # doctest: +SKIP Returns ------- An ``OpenFiles`` instance, which is a list of ``OpenFile`` objects that can be used as a single context Notes ----- For a full list of the available protocols and the implementations that they map across to see the latest online documentation: - For implementations built into ``fsspec`` see https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations - For implementations in separate packages see https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations )num name_functionstorage_optionsprotocolexpandfiler=T)exist_ok)rrrrrrK)get_fs_token_pathsre auto_mkdir_parentmakedirsrJr)urlpathrrrrrcrbrerrjrfkwargsrfs_tokenpathsrparentsparents r! open_filesrss^- #B% {{f" DZ0562::d#66:ABVd +B   '!    7B sBB$<B)c~tjd}d|vr9|jdDcgc]}d|vs|j|r|n|dz c}n|g}g}d}|j }t |D]}|j ddxst|dxsd}t|} | j|} |j |i} ||dur| j|td i| | } | j|}|dvrd| vr|}|j||| f|}tt |}|Scc}w) Nz .*[^a-z]+.*z::://rerrg> filecache blockcache simplecachetarget_protocolrH)recompilesplitmatchcopyr>popsplit_protocolr _get_kwargs_from_urlsupdatedict_strip_protocolr0list) rrnxpbitsr\ previous_bitbitrecls extra_kwargskwskws r! _un_chainr8sQ =!A 4<@Dzz$?OP!eqjAGGAJAI 5PV CL [[]F~::j$/S>#3Fq3ISV"8,005 jj2& $q'> JJv   (L (C (!!#& B B!+C C2&'  x} C J3 Qs#D:c jhd}|jDcic] \}}||vs ||}}}t||}i}tt|D]D\}}|\} } } |t |dz k(rt di| |}+t di| ||d<| |d<| |d<F|d\} } } t | fi|}|| fScc}}w)a Turn fully-qualified and potentially chained URL into filesystem instance Parameters ---------- url : str The fsspec-compatible URL **kwargs: dict Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc. Returns ------- filesystem : FileSystem The new filesystem discovered from ``url`` and created with ``**kwargs``. urlpath : str The file-systems-specific URL for ``url``. >rbrrrfrrrrcrtarget_optionsryforrH)itemsr enumerater>r^rr )urlrn known_kwargskvchaininkwargsichurlsrerrm_rs r! url_to_fsrWs, L &||~ Gtq!,1Fad GF G c6 "EH8E?+2h E Q -b-H-H %)%;B%;(%;!"&."#!8GXq H ) )B w;Hs B/B/c Ptd|g||||||dd|}|s t||dS)aGiven a path or paths, return one ``OpenFile`` object. Parameters ---------- urlpath: string or list Absolute or relative filepath. Prefix with a protocol like ``s3://`` to read from alternative filesystems. Should not include glob character(s). mode: 'rb', 'wt', etc. compression: string or None If given, open file using compression codec. Can either be a compression name (a key in ``fsspec.compression.compr``) or "infer" to guess the compression from the filename suffix. encoding: str For text mode only errors: None or str Passed to TextIOWrapper in text mode protocol: str or None If given, overrides the protocol found in the URL. newline: bytes or None Used for line terminator in text mode. If None, uses system default; if blank, uses no translation. **kwargs: dict Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc. Examples -------- >>> openfile = open('2015-01-01.csv') # doctest: +SKIP >>> openfile = open( ... 's3://bucket/2015-01-01.csv.gz', compression='gzip' ... ) # doctest: +SKIP >>> with openfile as f: ... df = pd.read_csv(f) # doctest: +SKIP ... Returns ------- ``OpenFile`` object. Notes ----- For a full list of the available protocols and the implementations that they map across to see the latest online documentation: - For implementations built into ``fsspec`` see https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations - For implementations in separate packages see https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations F)rmrrrrrerrfrrH)rsFileNotFoundError) rmrrrrrerrnr\s r!r/r/sQx      C (( q6Mr#c 4d|vr tdt|fd|i|}t|djdds td|5}|Dcgc]}|j}}dddt |t rt|sdSScc}w#1swY0xYw) aKOpen file(s) which can be resolved to local For files which either are local, or get downloaded upon open (e.g., by file caching) Parameters ---------- url: str or list(str) mode: str Must be read mode storage_options: passed on to FS for or used by open_files (e.g., compression) r=z(Can only ensure local files when readingrr local_fileFzOopen_local can only be used on a filesystem which has attribute local_file=TrueN)rRrsgetattrrnamerYstrr)rrrdofrMr2rps r! open_localrs $CDD C 6d 6o 6B 2a588\5 1 -   (u!&'A''(#sIcNQx L(((sB B B BBcX|dk(r t|}||tvrtd|d|S)NinferzCompression type z not supported)rr rR)rmrs r!rrs<g'0 ;e#;,[MHII r#cvt|}d|vr'|jdd\}}t|dkDr||fSd|fS)zReturn protocol, path pairrurN)rr|r^)rmrers r!rrsHW%G  ua0$ x=1 T> ! =r#cVt|\}}t|}|j|S)zCReturn only path part of full URL, according to appropriate backend)rr r)rmrerrs r!strip_protocolrs+ )KHa x (C   w ''r#cg}t|}d|vrt|Dcgc] }d|vsd c}dkDr tdt|t |}|D]4}d|vr|j t |||$|j|6t ||kDr|d|}|S|D]?}t|r!|j |j|/|j|A|Scc}w)a(Expand paths if they have a ``*`` in them (write mode) or any of ``*?[]`` in them (read mode). :param paths: list of paths mode: str Mode in which to open files. num: int If opening in writing mode, number of files we expect to create. fs: filesystem object name_function: callable If opening in writing mode, this callable is used to generate path names. Names are generated for each partition by ``urlpath.replace('*', name_function(partition_index))``. :return: list of paths w*rz;When writing data, only one filename mask can be specified.N) rsumrRmaxr^extend _expand_pathsr0rglob)rprrbrrcexpanded_pathsr curr_paths r!expand_paths_if_neededr s N KE d{ 5-aC1H- . 2M #s5z" 1Ii%%mI}c&RS%%i0  1 ~  $+DS1N  1I#%%bggi&89%%i0  1 1.s C'C'cJt|tttfr%|s t dt t|d}n t |}|xsi}|r||d<t ||xsi}i} tt|D]D\} } | \} } }| t|dz k(rtd i|| } +td i|| | d<| | d<| | d<F|d\}}}t|fi| }t|tttfrg|Dcgc]}t t ||xsid }}t|Dchc]}|d c}dkDr t d||Dcgc]}|d }}n|j|}t|tttfrt|||||}nid |vr|rt|||}nUd |vr|rt|||}nAd |vr:t|j!|Dcgc]}|j#|r|}}n|g}||j$|fScc}wcc}wcc}wcc}w) aFilesystem, deterministic token, and paths from a urlpath and options. Parameters ---------- urlpath: string or iterable Absolute or relative filepath, URL (may include protocols like ``s3://``), or globstring pointing to data. mode: str, optional Mode in which to open files. num: int, optional If opening in writing mode, number of files we expect to create. name_function: callable, optional If opening in writing mode, this callable is used to generate path names. Names are generated for each partition by ``urlpath.replace('*', name_function(partition_index))``. storage_options: dict, optional Additional keywords to pass to the filesystem class. protocol: str or None To override the protocol specifier in the URL expand: bool Expand string paths for writing, assuming the path is a directory zempty urlpath sequencerrerrryrz$Protocol mismatch getting fs from %srrrrH)rYrtuplesetrRrrrr>r^rr rrrsortedrisdir _fs_token)rmrrbrcrdrerfurlpath0rrrrrnested_protocolrrprrupchainspcr2s r!riri:sI>'D%-.56 6!$w-"23!'*%+O&. # h 52 6EH8E?+2$&!or E Q -b-H-H %)%;B%;(%;!"&5"#qE8Q H ) )B'D%-.LS GHInQ')>B ? B   ("1( )A -CWM M!()2A))""5)%$s+,&udC]K $;6!%CCjIc=#34II F5M ! NN*  L D5$- (4yCT  L  7  Js1C5c4eZdZdZ dfd ZdZxZS)r1zTextIOWrapper cannot be pickled. This solves it. Requires that ``buffer`` be pickleable, which all instances of AbstractBufferedFile are. cL||||||f|_t||jyr)r7rNr")r bufferrrrline_buffering write_throughrOs r!r"z PickleableTextIOWrapper.__init__s*Hfg~}T  $))$r#c&t|jfSr)r1r7r%s r!r&z"PickleableTextIOWrapper.__reduce__s& 11r#)NNNFF)rCrDrErFr"r&r_r`s@r!r1r1s# %2r#r1) rBNutf8NNrNNTT)rBNrNNN)rB)rBrNNNT))iologgingrrzrrcachingrrrrr r rr registryr r utilsrrrr getLoggerrrrrJrsrrr/rrrrrrir TextIOWrapperr1rHr#r!rs  6   8 $ppf7;7;x      l^>.f     IX:(,b    J#Z<2b..2r#