"h!UddlmZddlZddlZddlZddlmZddlmZ ddl Z er$ddlm Z m Z mZmZddlmZddlmZe ee fZd ed <Gd d Zy#e $r esddl Z Y>wxYw) ) annotationsN) TYPE_CHECKING) atomic_write)AnyDictIteratorLiteral) TypeAlias)CachingFileSystemr Detailc~eZdZdZddZddZddZ d ddZ ddZddZ ddZ dd Z dd Z dd Z dd Zy ) CacheMetadataa\Cache metadata. All reading and writing of cache metadata is performed by this class, accessing the cached files and blocks is not. Metadata is stored in a single file per storage directory in JSON format. For backward compatibility, also reads metadata stored in pickle format which is converted to JSON when next saved. cJ|s td||_ig|_d|_y)z Parameters ---------- storage: list[str] Directories containing cached files, must be at least one. Metadata is stored in the last of these directories by convention. z3CacheMetadata expects at least one storage locationFN) ValueError_storage cached_files_force_save_pickle)selfstorages b/var/www/html/sandstorm/venv/lib/python3.12/site-packages/fsspec/implementations/cache_metadata.py__init__zCacheMetadata.__init__%s.RS S +-$#(c t|d5}tj|cdddS#1swYyxYw#t$r;t|d5}t j|cdddcYS#1swYYyxYwwxYw)z6Low-level function to load metadata from specific filerNrb)openjsonloadrpickle)rfnfs r_loadzCacheMetadata._load8sk &b# $!yy| $ $ $ &b$ &1{{1~ & & & &s; 9- 9699A=A/# A=/A9 4A=9A=c|jr+t|5}tj||dddyt|d5}t j||dddy#1swYyxYw#1swYyxYw)z4Low-level function to save metadata to specific fileNw)mode)rrr dumpr)rmetadata_to_saver!r"s r_savezCacheMetadata._saveAsu  " "b! 1Q ,a0 1 1bs+ /q *A. / / 1 1 / /sA%A1%A.1A:c#Kt|j}t|jD]7\}}||dz k(}|r|stjj |d||f9yw)aYield locations (filenames) where metadata is stored, and whether writable or not. Parameters ---------- writable: bool Set to True to only yield writable locations. Returns ------- Yields (str, str, bool) r cacheN)lenr enumerateospathjoin)r writable_onlynirwritables r_scan_locationszCacheMetadata._scan_locationsJse  #DMM2 DJAwAEzHX'',,w0'8C C  DsA&A(ct|j|jD]\\}}}}||vr||j}|b|jr"|d|j j |k7rR|jr(tj|dz |jkDrtjj||d}tjj|s||fcSy)zIf path is in cache return its details, otherwise return ``False``. If the optional CachingFileSystem is specified then it is used to perform extra checks to reject possible matches, such as if they are too old. uidtimer!F) zipr5rcopy check_filesfsukeyexpiryr8r.r/r0exists)rr/cfsr!base_r+details r check_filezCacheMetadata.check_file`s%((<(<(>@Q@Q$R " MRq55 4[%%'F??ve} D8I'I::$))+v">"KdF4L1Bww~~b!rz! " rcg}|jdjjD]\}}tj|dz |kDs$|j dd}|st d|t jj|jd|}|j||jdj||jdrLt jj|jdd}|j|jd||jd }||fS)zRemove expired metadata from the cache. Returns names of files corresponding to expired metadata and a boolean flag indicating whether the writable cache is empty. Caller is responsible for deleting the expired files. r8r!z)Cache metadata does not contain 'fn' for r+) rr:itemsr8get RuntimeErrorr.r/r0rappendpopr))r expiry_time expired_filesr/rCr! cache_pathwritable_cache_emptys r clear_expiredzCacheMetadata.clear_expired{s&  --b1668>>@ 0LD&yy{VF^+k9ZZb)&CD6JWW\\$--"3R8$$R(!!"%))$/ 0   R dmmB&7AJ JJt((,j 9#'#4#4R#88222rclg}|jD]\}}}tjj|r]|j |}|j D]'}t |dtst|d|d<)|j||ji|xsig|_ y)z>Load all metadata from disk and store in ``self.cached_files``blocksN) r5r.r/r?r#values isinstancelistsetrKr)rrr!rBloaded_cached_filescs rrzCacheMetadata.loads ,,. (HB1ww~~b!&*jjn#,3357A!!H+t4&)!H+&6( 7##$78##B' ()0RDrc|jd|}|ddur/t|d|jz|jk\rd|d<yyy)zPerform side-effect actions on closing a cached file. The actual closing of the file is the responsibility of the caller. rFrSTN)rr, blocksizesize)rr"r/rYs ron_close_cached_filez"CacheMetadata.on_close_cached_filesT   b !$ ' X;d "s1X;'7!++'E'OAhK(P "rc|j|d}|sy|\}}|j|jdr0|jdj ||j |St d)zRemove metadata of cached file. If path is in the cache, return the filename of the cached file, otherwise return ``None``. Caller is responsible for deleting the cached file. NrFzr}sf" % 33+)S#XFI&N-N- sA A$#A$